[PATCH v4 00/24] FPGA Device Feature List (DFL) Device Drivers

STALE3052d

Revision v4 of 4 in this series.

93 messages, 4 authors, 2018-04-06 · page 1 of 2 · open the first message on its own page

[PATCH v4 00/24] FPGA Device Feature List (DFL) Device Drivers

From: Wu Hao <hidden>
Date: 2018-02-13 09:56:11

Hi All,

Here is v4 patch-series adding drivers for FPGA DFL devices.
(This is the new version of [PATCH v3 00/21] Intel FPGA Device Drivers[1])

This patch series provides a common framework to support FPGA Device Feature
List (DFL), and also feature dev drivers under this DFL framework to provide
interfaces for userspace applications to configure, enumerate, open, and
access FPGA accelerators on DFL based FPGA device and enables system level
management functions such as FPGA partial reconfiguration, power management
and virtualization.

This patch series only adds the basic functions for FPGA accelerators and
partial reconfiguration. Patches for more functions, e.g power management
and virtualization, will be submitted after this series gets reviewed.

Note this patch series is only verified on DFL based Intel(R) FPGA PCIe
devices (e.g Intel Server Platform with In-package FPGA and Intel FPGA PCIe
Acceleration Cards).

Patch 1: add a document for FPGA DFL framework driver overview, including
Device Feature List (DFL) introduction, the HW architecture, driver
organization, device enumeration, virtualization and opens.

Patch 2: add region_id for fpga_image_info data structure, which allows
driver to pass region id information to fpga-mgr for FPGA reconfiguration
function. (Used by Patch 14)

Patch 3: add a 'status' sysfs interface to fpga-mgr class, it reflects
the status of the fpga-mgr including reconfiguration errors. (Used by
Patch 16)

Patch 4-7: add FPGA device feature list support, it provides common
enumeration interfaces which creates container device (FPGA base region)
and all feature devices by walking through all the 'Device Feature Lists'
provided low level drivers.

Patch 8-9: implement FPGA PCIe device driver. It locates all 'Device
Feature Lists' in PCIe device memory and invokes common interfaces from
above device feature list framework to finish the enumeration.

Patch 10-15: implement FPGA Management Engine (FME) driver. It's a
platform driver matching with the FME platform device created by above
device feature list framework during enumeration. Sysfs and device file
ioctls are exposed as user interfaces to allow partial reconfiguration
to Accelerated Function Units (AFUs) from user space applications.

Patch 16-19: implement FPGA manager/bridge/region platform drivers for
Intel FPGA Management Engine (FME). These platform drivers match with
platform devices created by above FME driver, they use the generic
fpga-mgr/bridge/region class infrastructure to implement FPGA partial
reconfiguration function.

Patch 20-24: implement FPGA Accelerated Function Unit (AFU) driver.
It's a platform driver matching with AFU platform device created by above
device feature list framework during enumeration. It provides user
interfaces to expose the AFU MMIO region, map/unmap dma buffer, and
control the port which AFU connects to.

Changes from v3:
- Fix SPDX license issue.
- Rename documentation to dfl.txt, add introduction for Device Feature List
  (DFL) and re-organize the content.
- Rename to FPGA Device Feature List (DFL) drivers from Intel FPGA device
  drivers for better reuse purposes. Unified driver and files to dfl-*.*
- Remove static feature_info table from common enumeration code, and switch
  to use feature id for sub feature driver matching.
- Remove next_afu register checking for AFU from common enumeration code.
- Remove interface_id sysfs for dfl-fme-mgr and use per fpga-region
  compat_id instead. (new patch 13, 15, 19).
- Add more comments for driver data structures and functions.
- Fix typos, issues in debug message/commit message and other places.

Changes from v2:
- Split common enumeration code from pcie driver to a separated module
  which for device feature list support.
- Drop fpga-dev class and switch to use fpga base region as container.
- Update the intel-fpga.txt documentation for new driver organization.
- Rename feature device drivers for future code reuse.
- Rebase code due to fpga APIs changes
- replace bitfields with marco and shift.
- fix typos, checkpatch issue and other comments.

Changes from v1:
- Use GPLv2 license instead of Dual BSD/GPL.
- Move the code to drivers/fpga folder.
- Update the intel-fpga.txt documentation for new driver organization.
- Add documentation for new sysfs interfaces.
- Switch to use common fpga-region interface for partial reconfiguration
  (PR) function in FME. It creates fpga-region/fpga-mgr/fpga-bridge
  platform devices and leave the implementation to their platform drivers.
- Add platform drivers for FME fpga-mgr/bridge/region platform devices.
- Fix kbuild warnings, typos and other comments.

This patch series depends on the below patchset from Alan Tull.
[PATCH v3 0/5] fpga: don't use drvdata in common fpga code[2]

[1] https://marc.info/?l=linux-api&m=151176589114857&w=2
[2] https://marc.info/?l=linux-fpga&m=151803761521039&w=2

Kang Luwei (3):
  fpga: dfl: add FPGA Management Engine driver basic framework
  fpga: dfl: fme: add header sub feature support
  fpga: dfl: fme: add partial reconfiguration sub feature support

Wu Hao (18):
  docs: fpga: add a document for FPGA Device Feature List (DFL)
    Framework Overview
  fpga: mgr: add region_id to fpga_image_info
  fpga: mgr: add status for fpga-manager
  fpga: add device feature list support
  fpga: dfl: add chardev support for feature devices
  fpga: dfl: adds fpga_cdev_find_port
  fpga: dfl-pci: add enumeration for feature devices
  fpga: dfl: fme: add FPGA_GET_API_VERSION/CHECK_EXTENSION ioctls
    support
  fpga: region: add compat_id support
  fpga: dfl-fme-pr: add compat_id support for dfl-fme-region platform
    device.
  fpga: dfl: add fpga manager platform driver for FME
  fpga: dfl: add fpga bridge platform driver for FME
  fpga: dfl: add fpga region platform driver for FME
  fpga: dfl-fme-region: add compat_id support
  fpga: dfl: add FPGA Accelerated Function Unit driver basic framework
  fpga: dfl: afu: add header sub feature support
  fpga: dfl: afu: add FPGA_GET_API_VERSION/CHECK_EXTENSION ioctls
    support
  fpga: dfl: afu: add FPGA_PORT_DMA_MAP/UNMAP ioctls support

Xiao Guangrong (2):
  fpga: dfl: add feature device infrastructure
  fpga: dfl: afu: add user afu sub feature support

Zhang Yi (1):
  fpga: add FPGA DFL PCIe device driver

 Documentation/ABI/testing/sysfs-class-fpga-manager |  24 +
 Documentation/ABI/testing/sysfs-class-fpga-region  |   5 +
 Documentation/ABI/testing/sysfs-platform-dfl-fme   |  23 +
 Documentation/ABI/testing/sysfs-platform-dfl-port  |  16 +
 Documentation/fpga/dfl.txt                         | 382 ++++++++
 Documentation/ioctl/ioctl-number.txt               |   1 +
 drivers/fpga/Kconfig                               |  68 ++
 drivers/fpga/Makefile                              |  14 +
 drivers/fpga/dfl-afu-dma-region.c                  | 463 ++++++++++
 drivers/fpga/dfl-afu-main.c                        | 476 ++++++++++
 drivers/fpga/dfl-afu-region.c                      | 165 ++++
 drivers/fpga/dfl-afu.h                             | 100 +++
 drivers/fpga/dfl-fme-br.c                          |  85 ++
 drivers/fpga/dfl-fme-main.c                        | 281 ++++++
 drivers/fpga/dfl-fme-mgr.c                         | 290 ++++++
 drivers/fpga/dfl-fme-pr.c                          | 517 +++++++++++
 drivers/fpga/dfl-fme-pr.h                          | 115 +++
 drivers/fpga/dfl-fme-region.c                      |  90 ++
 drivers/fpga/dfl-fme.h                             |  38 +
 drivers/fpga/dfl-pci.c                             | 322 +++++++
 drivers/fpga/dfl.c                                 | 982 +++++++++++++++++++++
 drivers/fpga/dfl.h                                 | 462 ++++++++++
 drivers/fpga/fpga-mgr.c                            |  28 +
 drivers/fpga/fpga-region.c                         |  19 +
 include/linux/fpga/fpga-mgr.h                      |  11 +
 include/linux/fpga/fpga-region.h                   |  13 +
 include/uapi/linux/fpga-dfl.h                      | 176 ++++
 27 files changed, 5166 insertions(+)
 create mode 100644 Documentation/ABI/testing/sysfs-class-fpga-region
 create mode 100644 Documentation/ABI/testing/sysfs-platform-dfl-fme
 create mode 100644 Documentation/ABI/testing/sysfs-platform-dfl-port
 create mode 100644 Documentation/fpga/dfl.txt
 create mode 100644 drivers/fpga/dfl-afu-dma-region.c
 create mode 100644 drivers/fpga/dfl-afu-main.c
 create mode 100644 drivers/fpga/dfl-afu-region.c
 create mode 100644 drivers/fpga/dfl-afu.h
 create mode 100644 drivers/fpga/dfl-fme-br.c
 create mode 100644 drivers/fpga/dfl-fme-main.c
 create mode 100644 drivers/fpga/dfl-fme-mgr.c
 create mode 100644 drivers/fpga/dfl-fme-pr.c
 create mode 100644 drivers/fpga/dfl-fme-pr.h
 create mode 100644 drivers/fpga/dfl-fme-region.c
 create mode 100644 drivers/fpga/dfl-fme.h
 create mode 100644 drivers/fpga/dfl-pci.c
 create mode 100644 drivers/fpga/dfl.c
 create mode 100644 drivers/fpga/dfl.h
 create mode 100644 include/uapi/linux/fpga-dfl.h

-- 
2.7.4

[PATCH v4 03/24] fpga: mgr: add status for fpga-manager

From: Wu Hao <hidden>
Date: 2018-02-13 09:35:26

This patch adds status to fpga-manager data structure, to allow
driver to store full/partial reconfiguration errors and other
status information, and adds one status callback to fpga_manager_ops
to allow fpga_manager to collect latest status when failures are
detected.

The following sysfs file is created:
* /sys/class/fpga_manager/<fpga>/status
  Return status of fpga manager, including reconfiguration errors.

Signed-off-by: Wu Hao <redacted>
---
v3: add one line description for status
    add status callback function to fpga_manager_ops
    update fpga-mgr status if any failure or during initialization
    s/INCOMPATIBLE_BS_ERR/INCOMPATIBLE_IMAGE_ERR/
v4: simply code per suggestion from Alan.
    add supported status strings (and descriptions) in sysfs document.
    remove unused error code (SECURE_LOAD_ERR).
---
 Documentation/ABI/testing/sysfs-class-fpga-manager | 24 +++++++++++++++++++
 drivers/fpga/fpga-mgr.c                            | 28 ++++++++++++++++++++++
 include/linux/fpga/fpga-mgr.h                      |  9 +++++++
 3 files changed, 61 insertions(+)

diff --git a/Documentation/ABI/testing/sysfs-class-fpga-manager b/Documentation/ABI/testing/sysfs-class-fpga-manager
index 23056c5..60db277 100644
--- a/Documentation/ABI/testing/sysfs-class-fpga-manager
+++ b/Documentation/ABI/testing/sysfs-class-fpga-manager

@@ -35,3 +35,27 @@ Description:	Read fpga manager state as a string.
 		* write complete	= Doing post programming steps
 		* write complete error	= Error while doing post programming
 		* operating		= FPGA is programmed and operating
+
+What:		/sys/class/fpga_manager/<fpga>/status
+Date:		February 2018
+KernelVersion:	4.16
+Contact:	Wu Hao <hao.wu@intel.com>
+Description:	Read fpga manager status as a string.
+		If FPGA programming operation fails, it could be caused by crc
+		error or incompatible bitstream image. The intent of this
+		interface is to provide more detailed information for FPGA
+		programming errors to userspace. This is a list of strings for
+		the supported status.
+
+		* reconfig operation error 	- invalid operations detected by
+						  reconfiguration hardware.
+						  e.g start reconfiguration
+						  with errors not cleared
+		* reconfig CRC error		- CRC error detected by
+						  reconfiguration hardware.
+		* reconfig incompatible image	- reconfiguration image is
+						  incompatible with hardware
+		* reconfig IP protocol error	- protocol errors detected by
+						  reconfiguration hardware
+		* reconfig fifo overflow error	- FIFO overflow detected by
+						  reconfiguration hardware

diff --git a/drivers/fpga/fpga-mgr.c b/drivers/fpga/fpga-mgr.c
index 245e7a6..50b7396 100644
--- a/drivers/fpga/fpga-mgr.c
+++ b/drivers/fpga/fpga-mgr.c

@@ -397,12 +397,40 @@ static ssize_t state_show(struct device *dev,
 	return sprintf(buf, "%s\n", state_str[mgr->state]);
 }
 
+static ssize_t status_show(struct device *dev,
+			   struct device_attribute *attr, char *buf)
+{
+	struct fpga_manager *mgr = to_fpga_manager(dev);
+	u64 status;
+	int len = 0;
+
+	if (!mgr->mops->status)
+		return -ENOENT;
+
+	status = mgr->mops->status(mgr);
+
+	if (status & FPGA_MGR_STATUS_OPERATION_ERR)
+		len += sprintf(buf + len, "reconfig operation error\n");
+	if (status & FPGA_MGR_STATUS_CRC_ERR)
+		len += sprintf(buf + len, "reconfig CRC error\n");
+	if (status & FPGA_MGR_STATUS_INCOMPATIBLE_IMAGE_ERR)
+		len += sprintf(buf + len, "reconfig incompatible image\n");
+	if (status & FPGA_MGR_STATUS_IP_PROTOCOL_ERR)
+		len += sprintf(buf + len, "reconfig IP protocol error\n");
+	if (status & FPGA_MGR_STATUS_FIFO_OVERFLOW_ERR)
+		len += sprintf(buf + len, "reconfig fifo overflow error\n");
+
+	return len;
+}
+
 static DEVICE_ATTR_RO(name);
 static DEVICE_ATTR_RO(state);
+static DEVICE_ATTR_RO(status);
 
 static struct attribute *fpga_mgr_attrs[] = {
 	&dev_attr_name.attr,
 	&dev_attr_state.attr,
+	&dev_attr_status.attr,
 	NULL,
 };
 ATTRIBUTE_GROUPS(fpga_mgr);

diff --git a/include/linux/fpga/fpga-mgr.h b/include/linux/fpga/fpga-mgr.h
index 6e98b66..6c79183 100644
--- a/include/linux/fpga/fpga-mgr.h
+++ b/include/linux/fpga/fpga-mgr.h

@@ -112,6 +112,7 @@ struct fpga_image_info {
  * struct fpga_manager_ops - ops for low level fpga manager drivers
  * @initial_header_size: Maximum number of bytes that should be passed into write_init
  * @state: returns an enum value of the FPGA's state
+ * @status: returns status of the FPGA, including reconfiguration error code
  * @write_init: prepare the FPGA to receive confuration data
  * @write: write count bytes of configuration data to the FPGA
  * @write_sg: write the scatter list of configuration data to the FPGA

@@ -126,6 +127,7 @@ struct fpga_image_info {
 struct fpga_manager_ops {
 	size_t initial_header_size;
 	enum fpga_mgr_states (*state)(struct fpga_manager *mgr);
+	u64 (*status)(struct fpga_manager *mgr);
 	int (*write_init)(struct fpga_manager *mgr,
 			  struct fpga_image_info *info,
 			  const char *buf, size_t count);

@@ -137,6 +139,13 @@ struct fpga_manager_ops {
 	const struct attribute_group **groups;
 };
 
+/* FPGA manager status: Partial/Full Reconfiguration errors */
+#define FPGA_MGR_STATUS_OPERATION_ERR		BIT(0)
+#define FPGA_MGR_STATUS_CRC_ERR			BIT(1)
+#define FPGA_MGR_STATUS_INCOMPATIBLE_IMAGE_ERR	BIT(2)
+#define FPGA_MGR_STATUS_IP_PROTOCOL_ERR		BIT(3)
+#define FPGA_MGR_STATUS_FIFO_OVERFLOW_ERR	BIT(4)
+
 /**
  * struct fpga_manager - fpga manager structure
  * @name: name of low level fpga manager

-- 
2.7.4

[PATCH v4 08/24] fpga: add FPGA DFL PCIe device driver

From: Wu Hao <hidden>
Date: 2018-02-13 09:35:37

From: Zhang Yi <redacted>

This patch implements the basic framework of the driver for FPGA PCIe
device which implements the Device Feature List (DFL) in its MMIO space.
This driver is verified on Intel(R) PCIe based FPGA DFL devices, including
both integrated (e.g Intel Server Platform with In-package FPGA) and
discrete (e.g Intel FPGA PCIe Acceleration Cards) solutions.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Zhang Yi <redacted>
Signed-off-by: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
Signed-off-by: Wu Hao <redacted>
---
v2: move the code to drivers/fpga folder as suggested by Alan Tull.
    switch to GPLv2 license.
    fix comments from Moritz Fischer.
v3: switch to pci_set_dma_mask/consistent_dma_mask() function.
    remove pci_save_state() in probe function.
    rename driver to INTEL_FPGA_DFL_PCI and intel-dfl-pci.c to indicate
    this driver supports Intel FPGA PCI devices which implement DFL.
    improve Kconfig description for INTEL_FPGA_DFL_PCI
v4: rename to FPGA_DFL_PCI (dfl-pci.c) for better reuse.
    fix SPDX license issue.
---
 drivers/fpga/Kconfig   |  15 ++++++
 drivers/fpga/Makefile  |   3 ++
 drivers/fpga/dfl-pci.c | 127 +++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 145 insertions(+)
 create mode 100644 drivers/fpga/dfl-pci.c

diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
index 01ad31f..87f3d44 100644
--- a/drivers/fpga/Kconfig
+++ b/drivers/fpga/Kconfig

@@ -140,4 +140,19 @@ config FPGA_DFL
 	  Gate Array (FPGA) solutions which implement Device Feature List.
 	  It provides enumeration APIs, and feature device infrastructure.
 
+config FPGA_DFL_PCI
+	tristate "FPGA Device Feature List (DFL) PCIe Device Driver"
+	depends on PCI && FPGA_DFL
+	help
+	  Select this option to enable PCIe driver for PCIe based
+	  Field-Programmable Gate Array (FPGA) solutions which implemented
+	  the Device Feature List (DFL). This driver provides interfaces
+	  for userspace applications to configure, enumerate, open and access
+	  FPGA accelerators on the FPGA DFL devices, enables system level
+	  management functions such as FPGA partial reconfiguration, power
+	  management, and virtualization with DFL framework and DFL feature
+	  device drivers.
+
+	  To compile this as a module, choose M here.
+
 endif # FPGA

diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index c4c62b9..4375630 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile

@@ -30,3 +30,6 @@ obj-$(CONFIG_OF_FPGA_REGION)		+= of-fpga-region.o
 
 # FPGA Device Feature List Support
 obj-$(CONFIG_FPGA_DFL)			+= dfl.o
+
+# Drivers for FPGAs which implement DFL
+obj-$(CONFIG_FPGA_DFL_PCI)		+= dfl-pci.o

diff --git a/drivers/fpga/dfl-pci.c b/drivers/fpga/dfl-pci.c
new file mode 100644
index 0000000..d91ea42
--- /dev/null
+++ b/drivers/fpga/dfl-pci.c

@@ -0,0 +1,127 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Driver for FPGA Device Feature List (DFL) PCIe device
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Zhang Yi <Yi.Z.Zhang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *   Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
+ *   Joseph Grecco <joe.grecco-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *   Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *   Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *   Ananda Ravuri <ananda.ravuri-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *   Henry Mitchel <henry.mitchel-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ */
+
+#include <linux/pci.h>
+#include <linux/types.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/stddef.h>
+#include <linux/errno.h>
+#include <linux/aer.h>
+
+#define DRV_VERSION	"0.8"
+#define DRV_NAME	"dfl-pci"
+
+/* PCI Device ID */
+#define PCIE_DEVICE_ID_PF_INT_5_X	0xBCBD
+#define PCIE_DEVICE_ID_PF_INT_6_X	0xBCC0
+#define PCIE_DEVICE_ID_PF_DSC_1_X	0x09C4
+/* VF Device */
+#define PCIE_DEVICE_ID_VF_INT_5_X	0xBCBF
+#define PCIE_DEVICE_ID_VF_INT_6_X	0xBCC1
+#define PCIE_DEVICE_ID_VF_DSC_1_X	0x09C5
+
+static struct pci_device_id cci_pcie_id_tbl[] = {
+	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIE_DEVICE_ID_PF_INT_5_X),},
+	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIE_DEVICE_ID_VF_INT_5_X),},
+	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIE_DEVICE_ID_PF_INT_6_X),},
+	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIE_DEVICE_ID_VF_INT_6_X),},
+	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIE_DEVICE_ID_PF_DSC_1_X),},
+	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIE_DEVICE_ID_VF_DSC_1_X),},
+	{0,}
+};
+MODULE_DEVICE_TABLE(pci, cci_pcie_id_tbl);
+
+static
+int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
+{
+	int ret;
+
+	ret = pci_enable_device(pcidev);
+	if (ret < 0) {
+		dev_err(&pcidev->dev, "Failed to enable device %d.\n", ret);
+		return ret;
+	}
+
+	ret = pci_enable_pcie_error_reporting(pcidev);
+	if (ret && ret != -EINVAL)
+		dev_info(&pcidev->dev, "PCIE AER unavailable %d.\n", ret);
+
+	ret = pci_request_regions(pcidev, DRV_NAME);
+	if (ret) {
+		dev_err(&pcidev->dev, "Failed to request regions.\n");
+		goto disable_error_report_exit;
+	}
+
+	pci_set_master(pcidev);
+
+	if (!pci_set_dma_mask(pcidev, DMA_BIT_MASK(64))) {
+		ret = pci_set_consistent_dma_mask(pcidev, DMA_BIT_MASK(64));
+		if (ret)
+			goto release_region_exit;
+	} else if (!pci_set_dma_mask(pcidev, DMA_BIT_MASK(32))) {
+		ret = pci_set_consistent_dma_mask(pcidev, DMA_BIT_MASK(32));
+		if (ret)
+			goto release_region_exit;
+	} else {
+		ret = -EIO;
+		dev_err(&pcidev->dev, "No suitable DMA support available.\n");
+		goto release_region_exit;
+	}
+
+	/* TODO: create and add the platform device per feature list */
+	return 0;
+
+release_region_exit:
+	pci_release_regions(pcidev);
+disable_error_report_exit:
+	pci_disable_pcie_error_reporting(pcidev);
+	pci_disable_device(pcidev);
+	return ret;
+}
+
+static void cci_pci_remove(struct pci_dev *pcidev)
+{
+	pci_release_regions(pcidev);
+	pci_disable_pcie_error_reporting(pcidev);
+	pci_disable_device(pcidev);
+}
+
+static struct pci_driver cci_pci_driver = {
+	.name = DRV_NAME,
+	.id_table = cci_pcie_id_tbl,
+	.probe = cci_pci_probe,
+	.remove = cci_pci_remove,
+};
+
+static int __init ccidrv_init(void)
+{
+	pr_info("FPGA DFL PCIe Driver: Version %s\n", DRV_VERSION);
+
+	return pci_register_driver(&cci_pci_driver);
+}
+
+static void __exit ccidrv_exit(void)
+{
+	pci_unregister_driver(&cci_pci_driver);
+}
+
+module_init(ccidrv_init);
+module_exit(ccidrv_exit);
+
+MODULE_DESCRIPTION("FPGA DFL PCIe Device Driver");
+MODULE_AUTHOR("Intel Corporation");
+MODULE_LICENSE("GPL v2");

-- 
2.7.4

[PATCH v4 07/24] fpga: dfl: add feature device infrastructure

From: Wu Hao <hidden>
Date: 2018-02-13 09:35:38

From: Xiao Guangrong <redacted>

This patch abstracts the common operations of the sub features, and defines
the feature_ops data structure, including init, uinit and ioctl function
pointers. And this patch adds some common helper functions for FME and AFU
drivers, e.g feature_dev_use_begin/end which are used to ensure exclusive
usage of the feature device file.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Kang Luwei <redacted>
Signed-off-by: Zhang Yi <redacted>
Signed-off-by: Xiao Guangrong <redacted>
Signed-off-by: Wu Hao <redacted>
---
v2: rebased
v3: use const for feature_ops.
    replace pci related function.
v4: rebase and add more comments in code.
---
 drivers/fpga/dfl.c | 59 +++++++++++++++++++++++++++++++++++++
 drivers/fpga/dfl.h | 85 +++++++++++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 143 insertions(+), 1 deletion(-)

diff --git a/drivers/fpga/dfl.c b/drivers/fpga/dfl.c
index 38dc819..c0aad87 100644
--- a/drivers/fpga/dfl.c
+++ b/drivers/fpga/dfl.c

@@ -74,6 +74,65 @@ static enum fpga_id_type feature_dev_id_type(struct platform_device *pdev)
 	return FPGA_ID_MAX;
 }
 
+void fpga_dev_feature_uinit(struct platform_device *pdev)
+{
+	struct feature *feature;
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+
+	fpga_dev_for_each_feature(pdata, feature)
+		if (feature->ops) {
+			feature->ops->uinit(pdev, feature);
+			feature->ops = NULL;
+		}
+}
+EXPORT_SYMBOL_GPL(fpga_dev_feature_uinit);
+
+static int
+feature_instance_init(struct platform_device *pdev,
+		      struct feature_platform_data *pdata,
+		      struct feature *feature, struct feature_driver *drv)
+{
+	int ret;
+
+	WARN_ON(!feature->ioaddr);
+
+	ret = drv->ops->init(pdev, feature);
+	if (ret)
+		return ret;
+
+	feature->ops = drv->ops;
+
+	return ret;
+}
+
+int fpga_dev_feature_init(struct platform_device *pdev,
+			  struct feature_driver *feature_drvs)
+{
+	struct feature *feature;
+	struct feature_driver *drv = feature_drvs;
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	int ret;
+
+	while (drv->ops) {
+		fpga_dev_for_each_feature(pdata, feature) {
+			/* match feature and drv using id */
+			if (feature->id == drv->id) {
+				ret = feature_instance_init(pdev, pdata,
+							    feature, drv);
+				if (ret)
+					goto exit;
+			}
+		}
+		drv++;
+	}
+
+	return 0;
+exit:
+	fpga_dev_feature_uinit(pdev);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(fpga_dev_feature_init);
+
 struct fpga_chardev_info {
 	const char *name;
 	dev_t devt;

diff --git a/drivers/fpga/dfl.h b/drivers/fpga/dfl.h
index d6ecda1..ede1e95 100644
--- a/drivers/fpga/dfl.h
+++ b/drivers/fpga/dfl.h

@@ -132,6 +132,17 @@
 #define PORT_CTRL_SFTRST_ACK	BIT(4)			/* HW ack for reset */
 
 /**
+ * struct feature_driver - sub feature's driver
+ *
+ * @id:	sub feature id.
+ * @ops: ops of this sub feature.
+ */
+struct feature_driver {
+	u64 id;
+	const struct feature_ops *ops;
+};
+
+/**
  * struct feature - sub feature of the feature devices
  *
  * @id:	sub feature id.

@@ -139,13 +150,17 @@
  *		    this index is used to find its mmio resource from the
  *		    feature dev (platform device)'s reources.
  * @ioaddr: mapped mmio resource address.
+ * @ops: ops of this sub feature.
  */
 struct feature {
 	u64 id;
 	int resource_index;
 	void __iomem *ioaddr;
+	const struct feature_ops *ops;
 };
 
+#define DEV_STATUS_IN_USE	0
+
 /**
  * struct feature_platform_data - platform data for feature devices
  *

@@ -155,6 +170,8 @@ struct feature {
  * @dev: ptr to platform device linked with this platform data.
  * @disable_count: count for port disable.
  * @num: number for sub features.
+ * @dev_status: dev status (e.g DEV_STATUS_IN_USE).
+ * @private: ptr to feature dev private data.
  * @features: sub features of this feature dev.
  */
 struct feature_platform_data {

@@ -163,11 +180,44 @@ struct feature_platform_data {
 	struct cdev cdev;
 	struct platform_device *dev;
 	unsigned int disable_count;
-
+	unsigned long dev_status;
+	void *private;
 	int num;
 	struct feature features[0];
 };
 
+static inline int feature_dev_use_begin(struct feature_platform_data *pdata)
+{
+	/* Test and set IN_USE flags to ensure file is exclusively used */
+	if (test_and_set_bit_lock(DEV_STATUS_IN_USE, &pdata->dev_status))
+		return -EBUSY;
+
+	return 0;
+}
+
+static inline void feature_dev_use_end(struct feature_platform_data *pdata)
+{
+	clear_bit_unlock(DEV_STATUS_IN_USE, &pdata->dev_status);
+}
+
+static inline void
+fpga_pdata_set_private(struct feature_platform_data *pdata, void *private)
+{
+	pdata->private = private;
+}
+
+static inline void *fpga_pdata_get_private(struct feature_platform_data *pdata)
+{
+	return pdata->private;
+}
+
+struct feature_ops {
+	int (*init)(struct platform_device *pdev, struct feature *feature);
+	void (*uinit)(struct platform_device *pdev, struct feature *feature);
+	long (*ioctl)(struct platform_device *pdev, struct feature *feature,
+		      unsigned int cmd, unsigned long arg);
+};
+
 #define FPGA_FEATURE_DEV_FME		"dfl-fme"
 #define FPGA_FEATURE_DEV_PORT		"dfl-port"

@@ -177,6 +227,10 @@ static inline int feature_platform_data_size(const int num)
 		num * sizeof(struct feature);
 }
 
+void fpga_dev_feature_uinit(struct platform_device *pdev);
+int fpga_dev_feature_init(struct platform_device *pdev,
+			  struct feature_driver *feature_drvs);
+
 enum fpga_devt_type {
 	FPGA_DEVT_FME,
 	FPGA_DEVT_PORT,

@@ -257,6 +311,15 @@ static inline int fpga_port_reset(struct platform_device *pdev)
 	return ret;
 }
 
+static inline
+struct platform_device *fpga_inode_to_feature_dev(struct inode *inode)
+{
+	struct feature_platform_data *pdata;
+
+	pdata = container_of(inode->i_cdev, struct feature_platform_data, cdev);
+	return pdata->dev;
+}
+
 #define fpga_dev_for_each_feature(pdata, feature)			    \
 	for ((feature) = (pdata)->features;				    \
 	   (feature) < (pdata)->features + (pdata)->num; (feature)++)

@@ -284,6 +347,17 @@ static inline void __iomem *get_feature_ioaddr_by_id(struct device *dev, u64 id)
 	return NULL;
 }
 
+static inline bool is_feature_present(struct device *dev, u64 id)
+{
+	return !!get_feature_ioaddr_by_id(dev, id);
+}
+
+static inline struct device *
+fpga_pdata_to_parent(struct feature_platform_data *pdata)
+{
+	return pdata->dev->dev.parent->parent;
+}
+
 static inline bool feature_is_fme(void __iomem *base)
 {
 	u64 v = readq(base + DFH);

@@ -376,4 +450,13 @@ fpga_cdev_find_port(struct fpga_cdev *cdev, void *data,
 
 	return pdev;
 }
+
+static inline struct fpga_cdev *
+fpga_pdata_to_fpga_cdev(struct feature_platform_data *pdata)
+{
+	struct device *dev = pdata->dev->dev.parent;
+	struct fpga_region *region = to_fpga_region(dev);
+
+	return container_of(region, struct fpga_cdev, region);
+}
 #endif /* __FPGA_DFL_H */

-- 
2.7.4

[PATCH v4 13/24] fpga: region: add compat_id support

From: Wu Hao <hidden>
Date: 2018-02-13 09:35:48

This patch introduces a compat_id member and sysfs interface for each
fpga-region, e.g userspace applications could read the compat_id
from the sysfs interface for compatibility checking before PR.

Signed-off-by: Wu Hao <redacted>
---
 Documentation/ABI/testing/sysfs-class-fpga-region |  5 +++++
 drivers/fpga/fpga-region.c                        | 19 +++++++++++++++++++
 include/linux/fpga/fpga-region.h                  | 13 +++++++++++++
 3 files changed, 37 insertions(+)
 create mode 100644 Documentation/ABI/testing/sysfs-class-fpga-region

diff --git a/Documentation/ABI/testing/sysfs-class-fpga-region b/Documentation/ABI/testing/sysfs-class-fpga-region
new file mode 100644
index 0000000..419d930
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-class-fpga-region

@@ -0,0 +1,5 @@
+What:		/sys/class/fpga_region/<region>/compat_id
+Date:		February 2018
+KernelVersion:	4.16
+Contact:	Wu Hao <hao.wu@intel.com>
+Description:	FPGA region id for compatibility check.

diff --git a/drivers/fpga/fpga-region.c b/drivers/fpga/fpga-region.c
index 660a91b..babec96 100644
--- a/drivers/fpga/fpga-region.c
+++ b/drivers/fpga/fpga-region.c

@@ -162,6 +162,24 @@ int fpga_region_program_fpga(struct fpga_region *region)
 }
 EXPORT_SYMBOL_GPL(fpga_region_program_fpga);
 
+static ssize_t compat_id_show(struct device *dev,
+			      struct device_attribute *attr, char *buf)
+{
+	struct fpga_region *region = to_fpga_region(dev);
+
+	return sprintf(buf, "%016llx%016llx\n",
+		       (unsigned long long)region->compat_id.id_h,
+		       (unsigned long long)region->compat_id.id_l);
+}
+
+static DEVICE_ATTR_RO(compat_id);
+
+static struct attribute *fpga_region_attrs[] = {
+	&dev_attr_compat_id.attr,
+	NULL,
+};
+ATTRIBUTE_GROUPS(fpga_region);
+
 int fpga_region_register(struct fpga_region *region)
 {
 	struct device *dev = region->parent;

@@ -226,6 +244,7 @@ static int __init fpga_region_init(void)
 	if (IS_ERR(fpga_region_class))
 		return PTR_ERR(fpga_region_class);
 
+	fpga_region_class->dev_groups = fpga_region_groups;
 	fpga_region_class->dev_release = fpga_region_dev_release;
 
 	return 0;

diff --git a/include/linux/fpga/fpga-region.h b/include/linux/fpga/fpga-region.h
index 423c87e..bf97dcc 100644
--- a/include/linux/fpga/fpga-region.h
+++ b/include/linux/fpga/fpga-region.h

@@ -6,6 +6,17 @@
 #include <linux/fpga/fpga-bridge.h>
 
 /**
+ * struct fpga_region_compat_id - FPGA Region id for compatibility check
+ *
+ * @id_h: high 64bit of the compat_id
+ * @id_l: low 64bit of the compat_id
+ */
+struct fpga_region_compat_id {
+	u64 id_h;
+	u64 id_l;
+};
+
+/**
  * struct fpga_region - FPGA Region structure
  * @dev: FPGA Region device
  * @parent: parent device

@@ -13,6 +24,7 @@
  * @bridge_list: list of FPGA bridges specified in region
  * @mgr: FPGA manager
  * @info: FPGA image info
+ * @compat_id: FPGA region id for compatibility check.
  * @priv: private data
  * @get_bridges: optional function to get bridges to a list
  * @groups: optional attribute groups.

@@ -24,6 +36,7 @@ struct fpga_region {
 	struct list_head bridge_list;
 	struct fpga_manager *mgr;
 	struct fpga_image_info *info;
+	struct fpga_region_compat_id compat_id;
 	void *priv;
 	int (*get_bridges)(struct fpga_region *region);
 	const struct attribute_group **groups;

-- 
2.7.4

[PATCH v4 12/24] fpga: dfl: fme: add FPGA_GET_API_VERSION/CHECK_EXTENSION ioctls support

From: Wu Hao <hidden>
Date: 2018-02-13 09:35:49

FPGA_GET_API_VERSION and FPGA_CHECK_EXTENSION ioctls are common ones which
need to be supported by all feature devices drivers including FME and AFU.
Userspace application can use these ioctl interfaces to get the API info
and check if specific extension is supported or not in current driver.

This patch implements above 2 ioctls in FPGA Management Engine (FME)
driver.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Xiao Guangrong <redacted>
Signed-off-by: Wu Hao <redacted>
Acked-by: Alan Tull <atull@kernel.org>
Acked-by: Moritz Fischer <mdf@kernel.org>
---
v2: switched to GPLv2 license.
v3: rename intel-fpga.h to fpga-dfl.h and rebased.
v4: fix SPDX license issue.
    add Acked-by from Alan and Moritz.
---
 Documentation/ioctl/ioctl-number.txt |  1 +
 drivers/fpga/dfl-fme-main.c          | 12 +++++++++
 include/uapi/linux/fpga-dfl.h        | 48 ++++++++++++++++++++++++++++++++++++
 3 files changed, 61 insertions(+)
 create mode 100644 include/uapi/linux/fpga-dfl.h

diff --git a/Documentation/ioctl/ioctl-number.txt b/Documentation/ioctl/ioctl-number.txt
index 6501389..f1e7baa 100644
--- a/Documentation/ioctl/ioctl-number.txt
+++ b/Documentation/ioctl/ioctl-number.txt

@@ -324,6 +324,7 @@ Code  Seq#(hex)	Include File		Comments
 0xB3	00	linux/mmc/ioctl.h
 0xB4	00-0F	linux/gpio.h		<mailto:linux-gpio@vger.kernel.org>
 0xB5	00-0F	uapi/linux/rpmsg.h	<mailto:linux-remoteproc@vger.kernel.org>
+0xB6	all	linux/fpga-dfl.h
 0xC0	00-0F	linux/usb/iowarrior.h
 0xCA	00-0F	uapi/misc/cxl.h
 0xCA	10-2F	uapi/misc/ocxl.h

diff --git a/drivers/fpga/dfl-fme-main.c b/drivers/fpga/dfl-fme-main.c
index 056ae24..1a9929c 100644
--- a/drivers/fpga/dfl-fme-main.c
+++ b/drivers/fpga/dfl-fme-main.c

@@ -16,6 +16,7 @@
 
 #include <linux/kernel.h>
 #include <linux/module.h>
+#include <linux/fpga-dfl.h>
 
 #include "dfl.h"

@@ -114,6 +115,13 @@ static struct feature_driver fme_feature_drvs[] = {
 	},
 };
 
+static long fme_ioctl_check_extension(struct feature_platform_data *pdata,
+				      unsigned long arg)
+{
+	/* No extension support for now */
+	return 0;
+}
+
 static int fme_open(struct inode *inode, struct file *filp)
 {
 	struct platform_device *fdev = fpga_inode_to_feature_dev(inode);

@@ -154,6 +162,10 @@ static long fme_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
 	dev_dbg(&pdev->dev, "%s cmd 0x%x\n", __func__, cmd);
 
 	switch (cmd) {
+	case FPGA_GET_API_VERSION:
+		return FPGA_API_VERSION;
+	case FPGA_CHECK_EXTENSION:
+		return fme_ioctl_check_extension(pdata, arg);
 	default:
 		/*
 		 * Let sub-feature's ioctl function to handle the cmd

diff --git a/include/uapi/linux/fpga-dfl.h b/include/uapi/linux/fpga-dfl.h
new file mode 100644
index 0000000..9321ee9
--- /dev/null
+++ b/include/uapi/linux/fpga-dfl.h

@@ -0,0 +1,48 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Header File for FPGA DFL User API
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Zhang Yi <yi.z.zhang@intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ */
+
+#ifndef _UAPI_LINUX_FPGA_DFL_H
+#define _UAPI_LINUX_FPGA_DFL_H
+
+#define FPGA_API_VERSION 0
+
+/*
+ * The IOCTL interface for DFL based FPGA is designed for extensibility by
+ * embedding the structure length (argsz) and flags into structures passed
+ * between kernel and userspace. This design referenced the VFIO IOCTL
+ * interface (include/uapi/linux/vfio.h).
+ */
+
+#define FPGA_MAGIC 0xB6
+
+#define FPGA_BASE 0
+
+/**
+ * FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)
+ *
+ * Report the version of the driver API.
+ * Return: Driver API Version.
+ */
+
+#define FPGA_GET_API_VERSION	_IO(FPGA_MAGIC, FPGA_BASE + 0)
+
+/**
+ * FPGA_CHECK_EXTENSION - _IO(FPGA_MAGIC, FPGA_BASE + 1)
+ *
+ * Check whether an extension is supported.
+ * Return: 0 if not supported, otherwise the extension is supported.
+ */
+
+#define FPGA_CHECK_EXTENSION	_IO(FPGA_MAGIC, FPGA_BASE + 1)
+
+#endif /* _UAPI_LINUX_FPGA_DFL_H */

-- 
2.7.4

[PATCH v4 16/24] fpga: dfl: add fpga manager platform driver for FME

From: Wu Hao <hidden>
Date: 2018-02-13 09:35:54

This patch adds fpga manager driver for FPGA Management Engine (FME). It
implements fpga_manager_ops for FPGA Partial Reconfiguration function.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Kang Luwei <redacted>
Signed-off-by: Xiao Guangrong <redacted>
Signed-off-by: Wu Hao <redacted>
---
v3: rename driver to dfl-fpga-fme-mgr
    implemented status callback for fpga manager
    rebased due to fpga api changes
v4: rename to dfl-fme-mgr, and fix SPDX license issue
    add pr_credit comments and improve dev_err message
    remove interface_id sysfs interface
    include dfl-fme-pr.h instead of dfl.h
---
 drivers/fpga/Kconfig       |   6 +
 drivers/fpga/Makefile      |   1 +
 drivers/fpga/dfl-fme-mgr.c | 290 +++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 297 insertions(+)
 create mode 100644 drivers/fpga/dfl-fme-mgr.c

diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
index 103d5e2..89f76e8 100644
--- a/drivers/fpga/Kconfig
+++ b/drivers/fpga/Kconfig

@@ -150,6 +150,12 @@ config FPGA_DFL_FME
 	  FPGA platform level management features. There shall be 1 FME
 	  per DFL based FPGA device.
 
+config FPGA_DFL_FME_MGR
+	tristate "FPGA DFL FME Manager Driver"
+	depends on FPGA_DFL_FME
+	help
+	  Say Y to enable FPGA Manager driver for FPGA Management Engine.
+
 config FPGA_DFL_PCI
 	tristate "FPGA Device Feature List (DFL) PCIe Device Driver"
 	depends on PCI && FPGA_DFL

diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index 3c44fc9..f82814a 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile

@@ -31,6 +31,7 @@ obj-$(CONFIG_OF_FPGA_REGION)		+= of-fpga-region.o
 # FPGA Device Feature List Support
 obj-$(CONFIG_FPGA_DFL)			+= dfl.o
 obj-$(CONFIG_FPGA_DFL_FME)		+= dfl-fme.o
+obj-$(CONFIG_FPGA_DFL_FME_MGR)		+= dfl-fme-mgr.o
 
 dfl-fme-objs := dfl-fme-main.o dfl-fme-pr.o

diff --git a/drivers/fpga/dfl-fme-mgr.c b/drivers/fpga/dfl-fme-mgr.c
new file mode 100644
index 0000000..2f92c29
--- /dev/null
+++ b/drivers/fpga/dfl-fme-mgr.c

@@ -0,0 +1,290 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * FPGA Manager Driver for FPGA Management Engine (FME)
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Christopher Rauer <christopher.rauer@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ */
+
+#include <linux/bitfield.h>
+#include <linux/module.h>
+#include <linux/iopoll.h>
+#include <linux/fpga/fpga-mgr.h>
+
+#include "dfl-fme-pr.h"
+
+#define PR_WAIT_TIMEOUT   8000000
+#define PR_HOST_STATUS_IDLE	0
+
+struct fme_mgr_priv {
+	void __iomem *ioaddr;
+	u64 pr_error;
+};
+
+static u64 pr_error_to_mgr_status(u64 err)
+{
+	u64 status = 0;
+
+	if (err & FME_PR_ERR_OPERATION_ERR)
+		status |= FPGA_MGR_STATUS_OPERATION_ERR;
+	if (err & FME_PR_ERR_CRC_ERR)
+		status |= FPGA_MGR_STATUS_CRC_ERR;
+	if (err & FME_PR_ERR_INCOMPATIBLE_BS)
+		status |= FPGA_MGR_STATUS_INCOMPATIBLE_IMAGE_ERR;
+	if (err & FME_PR_ERR_PROTOCOL_ERR)
+		status |= FPGA_MGR_STATUS_IP_PROTOCOL_ERR;
+	if (err & FME_PR_ERR_FIFO_OVERFLOW)
+		status |= FPGA_MGR_STATUS_FIFO_OVERFLOW_ERR;
+
+	return status;
+}
+
+static u64 fme_mgr_pr_error_handle(void __iomem *fme_pr)
+{
+	u64 pr_status, pr_error;
+
+	pr_status = readq(fme_pr + FME_PR_STS);
+	if (!(pr_status & FME_PR_STS_PR_STS))
+		return 0;
+
+	pr_error = readq(fme_pr + FME_PR_ERR);
+	writeq(pr_error, fme_pr + FME_PR_ERR);
+
+	return pr_error;
+}
+
+static int fme_mgr_write_init(struct fpga_manager *mgr,
+			      struct fpga_image_info *info,
+			      const char *buf, size_t count)
+{
+	struct device *dev = &mgr->dev;
+	struct fme_mgr_priv *priv = mgr->priv;
+	void __iomem *fme_pr = priv->ioaddr;
+	u64 pr_ctrl, pr_status;
+
+	if (!(info->flags & FPGA_MGR_PARTIAL_RECONFIG)) {
+		dev_err(dev, "only supports partial reconfiguration.\n");
+		return -EINVAL;
+	}
+
+	dev_dbg(dev, "resetting PR before initiated PR\n");
+
+	pr_ctrl = readq(fme_pr + FME_PR_CTRL);
+	pr_ctrl |= FME_PR_CTRL_PR_RST;
+	writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
+
+	if (readq_poll_timeout(fme_pr + FME_PR_CTRL, pr_ctrl,
+			       pr_ctrl & FME_PR_CTRL_PR_RSTACK, 1,
+			       PR_WAIT_TIMEOUT)) {
+		dev_err(dev, "PR Reset ACK timeout\n");
+		return -ETIMEDOUT;
+	}
+
+	pr_ctrl = readq(fme_pr + FME_PR_CTRL);
+	pr_ctrl &= ~FME_PR_CTRL_PR_RST;
+	writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
+
+	dev_dbg(dev,
+		"waiting for PR resource in HW to be initialized and ready\n");
+
+	if (readq_poll_timeout(fme_pr + FME_PR_STS, pr_status,
+			       (pr_status & FME_PR_STS_PR_STS) ==
+			       FME_PR_STS_PR_STS_IDLE, 1, PR_WAIT_TIMEOUT)) {
+		dev_err(dev, "PR Status timeout\n");
+		priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
+		return -ETIMEDOUT;
+	}
+
+	dev_dbg(dev, "check and clear previous PR error\n");
+	priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
+	if (priv->pr_error)
+		dev_dbg(dev, "previous PR error detected %llx\n",
+			(unsigned long long)priv->pr_error);
+
+	dev_dbg(dev, "set PR port ID\n");
+
+	pr_ctrl = readq(fme_pr + FME_PR_CTRL);
+	pr_ctrl &= ~FME_PR_CTRL_PR_RGN_ID;
+	pr_ctrl |= FIELD_PREP(FME_PR_CTRL_PR_RGN_ID, info->region_id);
+	writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
+
+	return 0;
+}
+
+static int fme_mgr_write(struct fpga_manager *mgr,
+			 const char *buf, size_t count)
+{
+	struct device *dev = &mgr->dev;
+	struct fme_mgr_priv *priv = mgr->priv;
+	void __iomem *fme_pr = priv->ioaddr;
+	u64 pr_ctrl, pr_status, pr_data;
+	int delay = 0, pr_credit, i = 0;
+
+	dev_dbg(dev, "start request\n");
+
+	pr_ctrl = readq(fme_pr + FME_PR_CTRL);
+	pr_ctrl |= FME_PR_CTRL_PR_START;
+	writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
+
+	dev_dbg(dev, "pushing data from bitstream to HW\n");
+
+	/*
+	 * driver can push data to PR hardware using PR_DATA register once HW
+	 * has enough pr_credit (> 1), pr_credit reduces one for every 32bit
+	 * pr data write to PR_DATA register. If pr_credit <= 1, driver needs
+	 * to wait for enough pr_credit from hardware by polling.
+	 */
+	pr_status = readq(fme_pr + FME_PR_STS);
+	pr_credit = FIELD_GET(FME_PR_STS_PR_CREDIT, pr_status);
+
+	while (count > 0) {
+		while (pr_credit <= 1) {
+			if (delay++ > PR_WAIT_TIMEOUT) {
+				dev_err(dev, "PR_CREDIT timeout\n");
+				return -ETIMEDOUT;
+			}
+			udelay(1);
+
+			pr_status = readq(fme_pr + FME_PR_STS);
+			pr_credit = FIELD_GET(FME_PR_STS_PR_CREDIT, pr_status);
+		}
+
+		if (count >= 4) {
+			pr_data = 0;
+			pr_data |= FIELD_PREP(FME_PR_DATA_PR_DATA_RAW,
+					      *(((u32 *)buf) + i));
+			writeq(pr_data, fme_pr + FME_PR_DATA);
+			count -= 4;
+			pr_credit--;
+			i++;
+		} else {
+			WARN_ON(1);
+			return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+static int fme_mgr_write_complete(struct fpga_manager *mgr,
+				  struct fpga_image_info *info)
+{
+	struct device *dev = &mgr->dev;
+	struct fme_mgr_priv *priv = mgr->priv;
+	void __iomem *fme_pr = priv->ioaddr;
+	u64 pr_ctrl;
+
+	pr_ctrl = readq(fme_pr + FME_PR_CTRL);
+	pr_ctrl |= FME_PR_CTRL_PR_COMPLETE;
+	writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
+
+	dev_dbg(dev, "green bitstream push complete\n");
+	dev_dbg(dev, "waiting for HW to release PR resource\n");
+
+	if (readq_poll_timeout(fme_pr + FME_PR_CTRL, pr_ctrl,
+			       !(pr_ctrl & FME_PR_CTRL_PR_START), 1,
+			       PR_WAIT_TIMEOUT)) {
+		dev_err(dev, "PR Completion ACK timeout.\n");
+		return -ETIMEDOUT;
+	}
+
+	dev_dbg(dev, "PR operation complete, checking status\n");
+	priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
+	if (priv->pr_error) {
+		dev_dbg(dev, "PR error detected %llx\n",
+			(unsigned long long)priv->pr_error);
+		return -EIO;
+	}
+
+	dev_dbg(dev, "PR done successfully\n");
+
+	return 0;
+}
+
+static enum fpga_mgr_states fme_mgr_state(struct fpga_manager *mgr)
+{
+	return FPGA_MGR_STATE_UNKNOWN;
+}
+
+static u64 fme_mgr_status(struct fpga_manager *mgr)
+{
+	struct fme_mgr_priv *priv = mgr->priv;
+
+	return pr_error_to_mgr_status(priv->pr_error);
+}
+
+static const struct fpga_manager_ops fme_mgr_ops = {
+	.write_init = fme_mgr_write_init,
+	.write = fme_mgr_write,
+	.write_complete = fme_mgr_write_complete,
+	.state = fme_mgr_state,
+	.status = fme_mgr_status,
+};
+
+static int fme_mgr_probe(struct platform_device *pdev)
+{
+	struct device *dev = &pdev->dev;
+	struct fme_mgr_priv *priv;
+	struct fpga_manager *mgr;
+	struct resource *res;
+	int ret;
+
+	priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
+	if (!priv)
+		return -ENOMEM;
+
+	res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+	priv->ioaddr = devm_ioremap(dev, res->start, resource_size(res));
+	if (IS_ERR(priv->ioaddr))
+		return PTR_ERR(priv->ioaddr);
+
+	mgr = devm_kzalloc(dev, sizeof(*mgr), GFP_KERNEL);
+	if (!mgr)
+		return -ENOMEM;
+
+	mgr->name = "DFL FPGA Manager";
+	mgr->mops = &fme_mgr_ops;
+	mgr->priv = priv;
+	mgr->parent = dev;
+	platform_set_drvdata(pdev, mgr);
+
+	ret = fpga_mgr_register(mgr);
+	if (ret)
+		dev_err(dev, "unable to register FPGA manager\n");
+
+	return ret;
+}
+
+static int fme_mgr_remove(struct platform_device *pdev)
+{
+	struct fpga_manager *mgr = platform_get_drvdata(pdev);
+
+	fpga_mgr_unregister(mgr);
+
+	return 0;
+}
+
+static struct platform_driver fme_mgr_driver = {
+	.driver	= {
+		.name    = FPGA_DFL_FME_MGR,
+	},
+	.probe   = fme_mgr_probe,
+	.remove  = fme_mgr_remove,
+};
+
+module_platform_driver(fme_mgr_driver);
+
+MODULE_DESCRIPTION("FPGA Manager for DFL FPGA Management Engine");
+MODULE_AUTHOR("Intel Corporation");
+MODULE_LICENSE("GPL v2");
+MODULE_ALIAS("platform:dfl-fme-mgr");

-- 
2.7.4

[PATCH v4 19/24] fpga: dfl-fme-region: add compat_id support

From: Wu Hao <hidden>
Date: 2018-02-13 09:36:03

This patch adds compat_id support, configure the compat_id value
of fpga_region, then userspace could read it from the common sysfs
interface compat_id of fpga_region.

Signed-off-by: Wu Hao <redacted>
---
 drivers/fpga/dfl-fme-region.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/fpga/dfl-fme-region.c b/drivers/fpga/dfl-fme-region.c
index 1f5f33d..bcef98a 100644
--- a/drivers/fpga/dfl-fme-region.c
+++ b/drivers/fpga/dfl-fme-region.c

@@ -48,6 +48,7 @@ static int fme_region_probe(struct platform_device *pdev)
 	region->get_bridges = fme_region_get_bridges;
 	region->priv = pdata;
 	region->parent = dev;
+	region->compat_id = pdata->compat_id;
 	platform_set_drvdata(pdev, region);
 
 	ret = fpga_region_register(region);

-- 
2.7.4

[PATCH v4 20/24] fpga: dfl: add FPGA Accelerated Function Unit driver basic framework

From: Wu Hao <hidden>
Date: 2018-02-13 09:36:05

On DFL FPGA devices, the Accelerated Function Unit (AFU), can be
reprogrammed for different functions. It connects to the FPGA
infrastructure("blue bistream") via a Port. Port CSRs are implemented
separately from the AFU CSRs to provide control and status of the Port.
Once valid green bitstream is programmed into the AFU, it allows access
to the AFU CSRs in the AFU MMIO space.

This patch only implements basic driver framework for AFU, including
device file operation framework.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Xiao Guangrong <redacted>
Signed-off-by: Wu Hao <redacted>
---
v3: rename driver to dfl-afu-main.c
v4: rename to dfl-port and fix SPDX license issue.
---
 drivers/fpga/Kconfig        |   9 +++
 drivers/fpga/Makefile       |   2 +
 drivers/fpga/dfl-afu-main.c | 159 ++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 170 insertions(+)
 create mode 100644 drivers/fpga/dfl-afu-main.c

diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
index 65d54a4..4c6b45f 100644
--- a/drivers/fpga/Kconfig
+++ b/drivers/fpga/Kconfig

@@ -168,6 +168,15 @@ config FPGA_DFL_FME_REGION
 	help
 	  Say Y to enable FPGA Region driver for FPGA Management Engine.
 
+config FPGA_DFL_AFU
+	tristate "FPGA DFL AFU Driver"
+	depends on FPGA_DFL
+	help
+	  This is the driver for FPGA Accelerated Function Unit (AFU) which
+	  implements AFU and Port management features. A User AFU connects
+	  to the FPGA infrastructure via a Port. There may be more than 1
+	  Port/AFU per DFL based FPGA device.
+
 config FPGA_DFL_PCI
 	tristate "FPGA Device Feature List (DFL) PCIe Device Driver"
 	depends on PCI && FPGA_DFL

diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index 163894e..5c9607b 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile

@@ -34,8 +34,10 @@ obj-$(CONFIG_FPGA_DFL_FME)		+= dfl-fme.o
 obj-$(CONFIG_FPGA_DFL_FME_MGR)		+= dfl-fme-mgr.o
 obj-$(CONFIG_FPGA_DFL_FME_BRIDGE)	+= dfl-fme-br.o
 obj-$(CONFIG_FPGA_DFL_FME_REGION)	+= dfl-fme-region.o
+obj-$(CONFIG_FPGA_DFL_AFU)		+= dfl-afu.o
 
 dfl-fme-objs := dfl-fme-main.o dfl-fme-pr.o
+dfl-afu-objs := dfl-afu-main.o
 
 # Drivers for FPGAs which implement DFL
 obj-$(CONFIG_FPGA_DFL_PCI)		+= dfl-pci.o

diff --git a/drivers/fpga/dfl-afu-main.c b/drivers/fpga/dfl-afu-main.c
new file mode 100644
index 0000000..70db28c
--- /dev/null
+++ b/drivers/fpga/dfl-afu-main.c

@@ -0,0 +1,159 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Driver for FPGA Accelerated Function Unit (AFU)
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Wu Hao <hao.wu@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+
+#include "dfl.h"
+
+static int port_hdr_init(struct platform_device *pdev, struct feature *feature)
+{
+	dev_dbg(&pdev->dev, "PORT HDR Init.\n");
+
+	return 0;
+}
+
+static void port_hdr_uinit(struct platform_device *pdev,
+			   struct feature *feature)
+{
+	dev_dbg(&pdev->dev, "PORT HDR UInit.\n");
+}
+
+static const struct feature_ops port_hdr_ops = {
+	.init = port_hdr_init,
+	.uinit = port_hdr_uinit,
+};
+
+static struct feature_driver port_feature_drvs[] = {
+	{
+		.id = PORT_FEATURE_ID_HEADER,
+		.ops = &port_hdr_ops,
+	},
+	{
+		.ops = NULL,
+	}
+};
+
+static int afu_open(struct inode *inode, struct file *filp)
+{
+	struct platform_device *fdev = fpga_inode_to_feature_dev(inode);
+	struct feature_platform_data *pdata;
+	int ret;
+
+	pdata = dev_get_platdata(&fdev->dev);
+	if (WARN_ON(!pdata))
+		return -ENODEV;
+
+	ret = feature_dev_use_begin(pdata);
+	if (ret)
+		return ret;
+
+	dev_dbg(&fdev->dev, "Device File Open\n");
+	filp->private_data = fdev;
+
+	return 0;
+}
+
+static int afu_release(struct inode *inode, struct file *filp)
+{
+	struct platform_device *pdev = filp->private_data;
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+
+	dev_dbg(&pdev->dev, "Device File Release\n");
+
+	feature_dev_use_end(pdata);
+
+	return 0;
+}
+
+static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
+{
+	struct platform_device *pdev = filp->private_data;
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	struct feature *f;
+	long ret;
+
+	dev_dbg(&pdev->dev, "%s cmd 0x%x\n", __func__, cmd);
+
+	switch (cmd) {
+	default:
+		/*
+		 * Let sub-feature's ioctl function to handle the cmd
+		 * Sub-feature's ioctl returns -ENODEV when cmd is not
+		 * handled in this sub feature, and returns 0 and other
+		 * error code if cmd is handled.
+		 */
+		fpga_dev_for_each_feature(pdata, f)
+			if (f->ops && f->ops->ioctl) {
+				ret = f->ops->ioctl(pdev, f, cmd, arg);
+				if (ret == -ENODEV)
+					continue;
+				else
+					return ret;
+			}
+	}
+
+	return -EINVAL;
+}
+
+static const struct file_operations afu_fops = {
+	.owner = THIS_MODULE,
+	.open = afu_open,
+	.release = afu_release,
+	.unlocked_ioctl = afu_ioctl,
+};
+
+static int afu_probe(struct platform_device *pdev)
+{
+	int ret;
+
+	dev_dbg(&pdev->dev, "%s\n", __func__);
+
+	ret = fpga_dev_feature_init(pdev, port_feature_drvs);
+	if (ret)
+		return ret;
+
+	ret = fpga_register_dev_ops(pdev, &afu_fops, THIS_MODULE);
+	if (ret)
+		fpga_dev_feature_uinit(pdev);
+
+	return ret;
+}
+
+static int afu_remove(struct platform_device *pdev)
+{
+	dev_dbg(&pdev->dev, "%s\n", __func__);
+
+	fpga_dev_feature_uinit(pdev);
+	fpga_unregister_dev_ops(pdev);
+
+	return 0;
+}
+
+static struct platform_driver afu_driver = {
+	.driver	= {
+		.name    = FPGA_FEATURE_DEV_PORT,
+	},
+	.probe   = afu_probe,
+	.remove  = afu_remove,
+};
+
+module_platform_driver(afu_driver);
+
+MODULE_DESCRIPTION("FPGA Accelerated Function Unit driver");
+MODULE_AUTHOR("Intel Corporation");
+MODULE_LICENSE("GPL v2");
+MODULE_ALIAS("platform:dfl-port");

-- 
2.7.4

[PATCH v4 18/24] fpga: dfl: add fpga region platform driver for FME

From: Wu Hao <hidden>
Date: 2018-02-13 09:36:06

This patch adds fpga region platform driver for FPGA Management Engine.
It register an fpga region with given fpga manager / bridge device.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Wu Hao <redacted>
Acked-by: Alan Tull <atull@kernel.org>
Acked-by: Moritz Fischer <mdf@kernel.org>
---
v3: rename driver to fpga-dfl-fme-region
    fix fpga_mgr_put order problem in remove function.
    rebased due to fpga api changes.
v4: rename to dfl-fme-region, fix SPDX license issue
    include dfl-fme-pr.h instead of dfl-fme.h and dfl.h
    add Acked-by from Alan and Moritz
---
 drivers/fpga/Kconfig          |  6 +++
 drivers/fpga/Makefile         |  1 +
 drivers/fpga/dfl-fme-region.c | 89 +++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 96 insertions(+)
 create mode 100644 drivers/fpga/dfl-fme-region.c

diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
index a8f939a..65d54a4 100644
--- a/drivers/fpga/Kconfig
+++ b/drivers/fpga/Kconfig

@@ -162,6 +162,12 @@ config FPGA_DFL_FME_BRIDGE
 	help
 	  Say Y to enable FPGA Bridge driver for FPGA Management Engine.
 
+config FPGA_DFL_FME_REGION
+	tristate "FPGA DFL FME Region Driver"
+	depends on FPGA_DFL_FME
+	help
+	  Say Y to enable FPGA Region driver for FPGA Management Engine.
+
 config FPGA_DFL_PCI
 	tristate "FPGA Device Feature List (DFL) PCIe Device Driver"
 	depends on PCI && FPGA_DFL

diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index 75096e9..163894e 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile

@@ -33,6 +33,7 @@ obj-$(CONFIG_FPGA_DFL)			+= dfl.o
 obj-$(CONFIG_FPGA_DFL_FME)		+= dfl-fme.o
 obj-$(CONFIG_FPGA_DFL_FME_MGR)		+= dfl-fme-mgr.o
 obj-$(CONFIG_FPGA_DFL_FME_BRIDGE)	+= dfl-fme-br.o
+obj-$(CONFIG_FPGA_DFL_FME_REGION)	+= dfl-fme-region.o
 
 dfl-fme-objs := dfl-fme-main.o dfl-fme-pr.o

diff --git a/drivers/fpga/dfl-fme-region.c b/drivers/fpga/dfl-fme-region.c
new file mode 100644
index 0000000..1f5f33d
--- /dev/null
+++ b/drivers/fpga/dfl-fme-region.c

@@ -0,0 +1,89 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * FPGA Region Driver for FPGA Management Engine (FME)
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Wu Hao <hao.wu@intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ */
+
+#include <linux/module.h>
+#include <linux/fpga/fpga-region.h>
+
+#include "dfl-fme-pr.h"
+
+static int fme_region_get_bridges(struct fpga_region *region)
+{
+	struct fme_region_pdata *pdata = region->priv;
+	struct device *dev = &pdata->br->dev;
+
+	return fpga_bridge_get_to_list(dev, region->info, &region->bridge_list);
+}
+
+static int fme_region_probe(struct platform_device *pdev)
+{
+	struct device *dev = &pdev->dev;
+	struct fme_region_pdata *pdata = dev_get_platdata(dev);
+	struct fpga_region *region;
+	struct fpga_manager *mgr;
+	int ret;
+
+	mgr = fpga_mgr_get(&pdata->mgr->dev);
+	if (IS_ERR(mgr))
+		return -EPROBE_DEFER;
+
+	region = devm_kzalloc(dev, sizeof(*region), GFP_KERNEL);
+	if (!region) {
+		ret = -ENOMEM;
+		goto eprobe_mgr_put;
+	}
+
+	region->mgr = mgr;
+	region->get_bridges = fme_region_get_bridges;
+	region->priv = pdata;
+	region->parent = dev;
+	platform_set_drvdata(pdev, region);
+
+	ret = fpga_region_register(region);
+	if (ret)
+		goto eprobe_mgr_put;
+
+	dev_dbg(dev, "DFL FME FPGA Region probed\n");
+
+	return 0;
+
+eprobe_mgr_put:
+	fpga_mgr_put(mgr);
+	return ret;
+}
+
+static int fme_region_remove(struct platform_device *pdev)
+{
+	struct fpga_region *region = dev_get_drvdata(&pdev->dev);
+
+	fpga_region_unregister(region);
+	fpga_mgr_put(region->mgr);
+
+	return 0;
+}
+
+static struct platform_driver fme_region_driver = {
+	.driver	= {
+		.name    = FPGA_DFL_FME_REGION,
+	},
+	.probe   = fme_region_probe,
+	.remove  = fme_region_remove,
+};
+
+module_platform_driver(fme_region_driver);
+
+MODULE_DESCRIPTION("FPGA Region for DFL FPGA Management Engine");
+MODULE_AUTHOR("Intel Corporation");
+MODULE_LICENSE("GPL v2");
+MODULE_ALIAS("platform:dfl-fme-region");

-- 
2.7.4

[PATCH v4 22/24] fpga: dfl: afu: add FPGA_GET_API_VERSION/CHECK_EXTENSION ioctls support

From: Wu Hao <hidden>
Date: 2018-02-13 09:36:08

FPGA_GET_API_VERSION and FPGA_CHECK_EXTENSION ioctls are common ones which
need to be supported by all feature devices drivers including FME and AFU.
This patch implements above 2 ioctls in FPGA Accelerated Function Unit
(AFU) driver.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Xiao Guangrong <redacted>
Signed-off-by: Wu Hao <redacted>
Acked-by: Alan Tull <atull@kernel.org>
Acked-by: Moritz Fischer <mdf@kernel.org>
---
v2: rebased
v3: rebased as driver renamed to fpga-dfl-afu
    fix one checkpatch issue
v4: add Acked-by from Alan and Moritz.
---
 drivers/fpga/dfl-afu-main.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/fpga/dfl-afu-main.c b/drivers/fpga/dfl-afu-main.c
index b2f8798..bead242 100644
--- a/drivers/fpga/dfl-afu-main.c
+++ b/drivers/fpga/dfl-afu-main.c

@@ -121,6 +121,13 @@ static int afu_release(struct inode *inode, struct file *filp)
 	return 0;
 }
 
+static long afu_ioctl_check_extension(struct feature_platform_data *pdata,
+				      unsigned long arg)
+{
+	/* No extension support for now */
+	return 0;
+}
+
 static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
 {
 	struct platform_device *pdev = filp->private_data;

@@ -131,6 +138,10 @@ static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
 	dev_dbg(&pdev->dev, "%s cmd 0x%x\n", __func__, cmd);
 
 	switch (cmd) {
+	case FPGA_GET_API_VERSION:
+		return FPGA_API_VERSION;
+	case FPGA_CHECK_EXTENSION:
+		return afu_ioctl_check_extension(pdata, arg);
 	default:
 		/*
 		 * Let sub-feature's ioctl function to handle the cmd

-- 
2.7.4

[PATCH v4 24/24] fpga: dfl: afu: add FPGA_PORT_DMA_MAP/UNMAP ioctls support

From: Wu Hao <hidden>
Date: 2018-02-13 09:36:18

DMA memory regions are required for Accelerated Function Unit (AFU) usage.
These two ioctls allow user space applications to map user memory regions
for dma, and unmap them after use. Iova is returned from driver to user
space application via FPGA_PORT_DMA_MAP ioctl. Application needs to unmap
it after use, otherwise, driver will unmap them in device file release
operation.

Each AFU has its own rb tree to keep track of its mapped DMA regions.

Ioctl interfaces:
* FPGA_PORT_DMA_MAP
  Do the dma mapping per user_addr and length which provided by user.
  Return iova in provided struct afu_port_dma_map.

* FPGA_PORT_DMA_UNMAP
  Unmap the dma region per iova provided by user.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Xiao Guangrong <redacted>
Signed-off-by: Wu Hao <redacted>
---
v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
    switched to GPLv2 license.
    fixed kbuild warnings.
v3: improved commit description and fixed coding style issues.
    replaced fpga_pdata_to_pcidev with fpga_pdata_to_fpga_cdev
v4: rebase and fix SPDX license issue
---
 drivers/fpga/Makefile             |   2 +-
 drivers/fpga/dfl-afu-dma-region.c | 463 ++++++++++++++++++++++++++++++++++++++
 drivers/fpga/dfl-afu-main.c       |  61 ++++-
 drivers/fpga/dfl-afu.h            |  31 ++-
 include/uapi/linux/fpga-dfl.h     |  37 +++
 5 files changed, 591 insertions(+), 3 deletions(-)
 create mode 100644 drivers/fpga/dfl-afu-dma-region.c

diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index 041e3cd1..dd454ab 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile

@@ -37,7 +37,7 @@ obj-$(CONFIG_FPGA_DFL_FME_REGION)	+= dfl-fme-region.o
 obj-$(CONFIG_FPGA_DFL_AFU)		+= dfl-afu.o
 
 dfl-fme-objs := dfl-fme-main.o dfl-fme-pr.o
-dfl-afu-objs := dfl-afu-main.o dfl-afu-region.o
+dfl-afu-objs := dfl-afu-main.o dfl-afu-region.o dfl-afu-dma-region.o
 
 # Drivers for FPGAs which implement DFL
 obj-$(CONFIG_FPGA_DFL_PCI)		+= dfl-pci.o

diff --git a/drivers/fpga/dfl-afu-dma-region.c b/drivers/fpga/dfl-afu-dma-region.c
new file mode 100644
index 0000000..e0cfa86
--- /dev/null
+++ b/drivers/fpga/dfl-afu-dma-region.c

@@ -0,0 +1,463 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Driver for FPGA Accelerated Function Unit (AFU) DMA Region Management
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Wu Hao <hao.wu@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ */
+
+#include <linux/dma-mapping.h>
+#include <linux/sched/signal.h>
+#include <linux/uaccess.h>
+
+#include "dfl-afu.h"
+
+static void put_all_pages(struct page **pages, int npages)
+{
+	int i;
+
+	for (i = 0; i < npages; i++)
+		if (pages[i])
+			put_page(pages[i]);
+}
+
+void afu_dma_region_init(struct feature_platform_data *pdata)
+{
+	struct fpga_afu *afu = fpga_pdata_get_private(pdata);
+
+	afu->dma_regions = RB_ROOT;
+}
+
+/**
+ * afu_dma_adjust_locked_vm - adjust locked memory
+ * @dev: port device
+ * @npages: number of pages
+ * @incr: increase or decrease locked memory
+ *
+ * Increase or decrease the locked memory size with npages input.
+ *
+ * Return 0 on success.
+ * Return -ENOMEM if locked memory size is over the limit and no CAP_IPC_LOCK.
+ */
+static int afu_dma_adjust_locked_vm(struct device *dev, long npages, bool incr)
+{
+	unsigned long locked, lock_limit;
+	int ret = 0;
+
+	/* the task is exiting. */
+	if (!current->mm)
+		return 0;
+
+	down_write(&current->mm->mmap_sem);
+
+	if (incr) {
+		locked = current->mm->locked_vm + npages;
+		lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
+
+		if (locked > lock_limit && !capable(CAP_IPC_LOCK))
+			ret = -ENOMEM;
+		else
+			current->mm->locked_vm += npages;
+	} else {
+		if (WARN_ON_ONCE(npages > current->mm->locked_vm))
+			npages = current->mm->locked_vm;
+		current->mm->locked_vm -= npages;
+	}
+
+	dev_dbg(dev, "[%d] RLIMIT_MEMLOCK %c%ld %ld/%ld%s\n", current->pid,
+		incr ? '+' : '-', npages << PAGE_SHIFT,
+		current->mm->locked_vm << PAGE_SHIFT, rlimit(RLIMIT_MEMLOCK),
+		ret ? "- execeeded" : "");
+
+	up_write(&current->mm->mmap_sem);
+
+	return ret;
+}
+
+/**
+ * afu_dma_pin_pages - pin pages of given dma memory region
+ * @pdata: feature device platform data
+ * @region: dma memory region to be pinned
+ *
+ * Pin all the pages of given fpga_afu_dma_region.
+ * Return 0 for success or negative error code.
+ */
+static int afu_dma_pin_pages(struct feature_platform_data *pdata,
+			     struct fpga_afu_dma_region *region)
+{
+	int npages = region->length >> PAGE_SHIFT;
+	struct device *dev = &pdata->dev->dev;
+	int ret, pinned;
+
+	ret = afu_dma_adjust_locked_vm(dev, npages, true);
+	if (ret)
+		return ret;
+
+	region->pages = kcalloc(npages, sizeof(struct page *), GFP_KERNEL);
+	if (!region->pages) {
+		ret = -ENOMEM;
+		goto unlock_vm;
+	}
+
+	pinned = get_user_pages_fast(region->user_addr, npages, 1,
+				     region->pages);
+	if (pinned < 0) {
+		ret = pinned;
+		goto put_pages;
+	} else if (pinned != npages) {
+		ret = -EFAULT;
+		goto free_pages;
+	}
+
+	dev_dbg(dev, "%d pages pinned\n", pinned);
+
+	return 0;
+
+put_pages:
+	put_all_pages(region->pages, pinned);
+free_pages:
+	kfree(region->pages);
+unlock_vm:
+	afu_dma_adjust_locked_vm(dev, npages, false);
+	return ret;
+}
+
+/**
+ * afu_dma_unpin_pages - unpin pages of given dma memory region
+ * @pdata: feature device platform data
+ * @region: dma memory region to be unpinned
+ *
+ * Unpin all the pages of given fpga_afu_dma_region.
+ * Return 0 for success or negative error code.
+ */
+static void afu_dma_unpin_pages(struct feature_platform_data *pdata,
+				struct fpga_afu_dma_region *region)
+{
+	long npages = region->length >> PAGE_SHIFT;
+	struct device *dev = &pdata->dev->dev;
+
+	put_all_pages(region->pages, npages);
+	kfree(region->pages);
+	afu_dma_adjust_locked_vm(dev, npages, false);
+
+	dev_dbg(dev, "%ld pages unpinned\n", npages);
+}
+
+/**
+ * afu_dma_check_continuous_pages - check if pages are continuous
+ * @region: dma memory region
+ *
+ * Return true if pages of given dma memory region have continuous physical
+ * address, otherwise return false.
+ */
+static bool afu_dma_check_continuous_pages(struct fpga_afu_dma_region *region)
+{
+	int npages = region->length >> PAGE_SHIFT;
+	int i;
+
+	for (i = 0; i < npages - 1; i++)
+		if (page_to_pfn(region->pages[i]) + 1 !=
+				page_to_pfn(region->pages[i + 1]))
+			return false;
+
+	return true;
+}
+
+/**
+ * dma_region_check_iova - check if memory area is fully contained in the region
+ * @region: dma memory region
+ * @iova: address of the dma memory area
+ * @size: size of the dma memory area
+ *
+ * Compare the dma memory area defined by @iova and @size with given dma region.
+ * Return true if memory area is fully contained in the region, otherwise false.
+ */
+static bool dma_region_check_iova(struct fpga_afu_dma_region *region,
+				  u64 iova, u64 size)
+{
+	if (!size && region->iova != iova)
+		return false;
+
+	return (region->iova <= iova) &&
+		(region->length + region->iova >= iova + size);
+}
+
+/**
+ * afu_dma_region_add - add given dma region to rbtree
+ * @pdata: feature device platform data
+ * @region: dma region to be added
+ *
+ * Return 0 for success, -EEXIST if dma region has already been added.
+ *
+ * Needs to be called with pdata->lock heold.
+ */
+static int afu_dma_region_add(struct feature_platform_data *pdata,
+			      struct fpga_afu_dma_region *region)
+{
+	struct fpga_afu *afu = fpga_pdata_get_private(pdata);
+	struct rb_node **new, *parent = NULL;
+
+	dev_dbg(&pdata->dev->dev, "add region (iova = %llx)\n",
+		(unsigned long long)region->iova);
+
+	new = &afu->dma_regions.rb_node;
+
+	while (*new) {
+		struct fpga_afu_dma_region *this;
+
+		this = container_of(*new, struct fpga_afu_dma_region, node);
+
+		parent = *new;
+
+		if (dma_region_check_iova(this, region->iova, region->length))
+			return -EEXIST;
+
+		if (region->iova < this->iova)
+			new = &((*new)->rb_left);
+		else if (region->iova > this->iova)
+			new = &((*new)->rb_right);
+		else
+			return -EEXIST;
+	}
+
+	rb_link_node(&region->node, parent, new);
+	rb_insert_color(&region->node, &afu->dma_regions);
+
+	return 0;
+}
+
+/**
+ * afu_dma_region_remove - remove given dma region from rbtree
+ * @pdata: feature device platform data
+ * @region: dma region to be removed
+ *
+ * Needs to be called with pdata->lock heold.
+ */
+static void afu_dma_region_remove(struct feature_platform_data *pdata,
+				  struct fpga_afu_dma_region *region)
+{
+	struct fpga_afu *afu;
+
+	dev_dbg(&pdata->dev->dev, "del region (iova = %llx)\n",
+		(unsigned long long)region->iova);
+
+	afu = fpga_pdata_get_private(pdata);
+	rb_erase(&region->node, &afu->dma_regions);
+}
+
+/**
+ * afu_dma_region_destroy - destroy all regions in rbtree
+ * @pdata: feature device platform data
+ *
+ * Needs to be called with pdata->lock heold.
+ */
+void afu_dma_region_destroy(struct feature_platform_data *pdata)
+{
+	struct fpga_afu *afu = fpga_pdata_get_private(pdata);
+	struct rb_node *node = rb_first(&afu->dma_regions);
+	struct fpga_afu_dma_region *region;
+
+	while (node) {
+		region = container_of(node, struct fpga_afu_dma_region, node);
+
+		dev_dbg(&pdata->dev->dev, "del region (iova = %llx)\n",
+			(unsigned long long)region->iova);
+
+		rb_erase(node, &afu->dma_regions);
+
+		if (region->iova)
+			dma_unmap_page(fpga_pdata_to_parent(pdata),
+				       region->iova, region->length,
+				       DMA_BIDIRECTIONAL);
+
+		if (region->pages)
+			afu_dma_unpin_pages(pdata, region);
+
+		node = rb_next(node);
+		kfree(region);
+	}
+}
+
+/**
+ * afu_dma_region_find - find the dma region from rbtree based on iova and size
+ * @pdata: feature device platform data
+ * @iova: address of the dma memory area
+ * @size: size of the dma memory area
+ *
+ * It finds the dma region from the rbtree based on @iova and @size:
+ * - if @size == 0, it finds the dma region which starts from @iova
+ * - otherwise, it finds the dma region which fully contains
+ *   [@iova, @iova+size)
+ * If nothing is matched returns NULL.
+ *
+ * Needs to be called with pdata->lock held.
+ */
+struct fpga_afu_dma_region *
+afu_dma_region_find(struct feature_platform_data *pdata, u64 iova, u64 size)
+{
+	struct fpga_afu *afu = fpga_pdata_get_private(pdata);
+	struct rb_node *node = afu->dma_regions.rb_node;
+	struct device *dev = &pdata->dev->dev;
+
+	while (node) {
+		struct fpga_afu_dma_region *region;
+
+		region = container_of(node, struct fpga_afu_dma_region, node);
+
+		if (dma_region_check_iova(region, iova, size)) {
+			dev_dbg(dev, "find region (iova = %llx)\n",
+				(unsigned long long)region->iova);
+			return region;
+		}
+
+		if (iova < region->iova)
+			node = node->rb_left;
+		else if (iova > region->iova)
+			node = node->rb_right;
+		else
+			/* the iova region is not fully covered. */
+			break;
+	}
+
+	dev_dbg(dev, "region with iova %llx and size %llx is not found\n",
+		(unsigned long long)iova, (unsigned long long)size);
+
+	return NULL;
+}
+
+/**
+ * afu_dma_region_find_iova - find the dma region from rbtree by iova
+ * @pdata: feature device platform data
+ * @iova: address of the dma region
+ *
+ * Needs to be called with pdata->lock held.
+ */
+static struct fpga_afu_dma_region *
+afu_dma_region_find_iova(struct feature_platform_data *pdata, u64 iova)
+{
+	return afu_dma_region_find(pdata, iova, 0);
+}
+
+/**
+ * afu_dma_map_region - map memory region for dma
+ * @pdata: feature device platform data
+ * @user_addr: address of the memory region
+ * @length: size of the memory region
+ * @iova: pointer of iova address
+ *
+ * Map memory region defined by @user_addr and @length, and return dma address
+ * of the memory region via @iova.
+ * Return 0 for success, otherwise error code.
+ */
+int afu_dma_map_region(struct feature_platform_data *pdata,
+		       u64 user_addr, u64 length, u64 *iova)
+{
+	struct fpga_afu_dma_region *region;
+	int ret;
+
+	/*
+	 * Check Inputs, only accept page-aligned user memory region with
+	 * valid length.
+	 */
+	if (!PAGE_ALIGNED(user_addr) || !PAGE_ALIGNED(length) || !length)
+		return -EINVAL;
+
+	/* Check overflow */
+	if (user_addr + length < user_addr)
+		return -EINVAL;
+
+	if (!access_ok(VERIFY_WRITE, (void __user *)(unsigned long)user_addr,
+		       length))
+		return -EINVAL;
+
+	region = kzalloc(sizeof(*region), GFP_KERNEL);
+	if (!region)
+		return -ENOMEM;
+
+	region->user_addr = user_addr;
+	region->length = length;
+
+	/* Pin the user memory region */
+	ret = afu_dma_pin_pages(pdata, region);
+	if (ret) {
+		dev_err(&pdata->dev->dev, "failed to pin memory region\n");
+		goto free_region;
+	}
+
+	/* Only accept continuous pages, return error else */
+	if (!afu_dma_check_continuous_pages(region)) {
+		dev_err(&pdata->dev->dev, "pages are not continuous\n");
+		ret = -EINVAL;
+		goto unpin_pages;
+	}
+
+	/* As pages are continuous then start to do DMA mapping */
+	region->iova = dma_map_page(fpga_pdata_to_parent(pdata),
+				    region->pages[0], 0,
+				    region->length,
+				    DMA_BIDIRECTIONAL);
+	if (dma_mapping_error(&pdata->dev->dev, region->iova)) {
+		dev_err(&pdata->dev->dev, "failed to map for dma\n");
+		ret = -EFAULT;
+		goto unpin_pages;
+	}
+
+	*iova = region->iova;
+
+	mutex_lock(&pdata->lock);
+	ret = afu_dma_region_add(pdata, region);
+	mutex_unlock(&pdata->lock);
+	if (ret) {
+		dev_err(&pdata->dev->dev, "failed to add dma region\n");
+		goto unmap_dma;
+	}
+
+	return 0;
+
+unmap_dma:
+	dma_unmap_page(fpga_pdata_to_parent(pdata),
+		       region->iova, region->length, DMA_BIDIRECTIONAL);
+unpin_pages:
+	afu_dma_unpin_pages(pdata, region);
+free_region:
+	kfree(region);
+	return ret;
+}
+
+/**
+ * afu_dma_unmap_region - unmap dma memory region
+ * @pdata: feature device platform data
+ * @iova: dma address of the region
+ *
+ * Unmap dma memory region based on @iova.
+ * Return 0 for success, otherwise error code.
+ */
+int afu_dma_unmap_region(struct feature_platform_data *pdata, u64 iova)
+{
+	struct fpga_afu_dma_region *region;
+
+	mutex_lock(&pdata->lock);
+	region = afu_dma_region_find_iova(pdata, iova);
+	if (!region) {
+		mutex_unlock(&pdata->lock);
+		return -EINVAL;
+	}
+
+	if (region->in_use) {
+		mutex_unlock(&pdata->lock);
+		return -EBUSY;
+	}
+
+	afu_dma_region_remove(pdata, region);
+	mutex_unlock(&pdata->lock);
+
+	dma_unmap_page(fpga_pdata_to_parent(pdata),
+		       region->iova, region->length, DMA_BIDIRECTIONAL);
+	afu_dma_unpin_pages(pdata, region);
+	kfree(region);
+
+	return 0;
+}

diff --git a/drivers/fpga/dfl-afu-main.c b/drivers/fpga/dfl-afu-main.c
index 9c0e4a8..5295aaf 100644
--- a/drivers/fpga/dfl-afu-main.c
+++ b/drivers/fpga/dfl-afu-main.c

@@ -178,7 +178,11 @@ static int afu_release(struct inode *inode, struct file *filp)
 
 	dev_dbg(&pdev->dev, "Device File Release\n");
 
-	fpga_port_reset(pdev);
+	mutex_lock(&pdata->lock);
+	__fpga_port_reset(pdev);
+	afu_dma_region_destroy(pdata);
+	mutex_unlock(&pdata->lock);
+
 	feature_dev_use_end(pdata);
 
 	return 0;

@@ -249,6 +253,55 @@ afu_ioctl_get_region_info(struct feature_platform_data *pdata, void __user *arg)
 	return 0;
 }
 
+static long
+afu_ioctl_dma_map(struct feature_platform_data *pdata, void __user *arg)
+{
+	struct fpga_port_dma_map map;
+	unsigned long minsz;
+	long ret;
+
+	minsz = offsetofend(struct fpga_port_dma_map, iova);
+
+	if (copy_from_user(&map, arg, minsz))
+		return -EFAULT;
+
+	if (map.argsz < minsz || map.flags)
+		return -EINVAL;
+
+	ret = afu_dma_map_region(pdata, map.user_addr, map.length, &map.iova);
+	if (ret)
+		return ret;
+
+	if (copy_to_user(arg, &map, sizeof(map))) {
+		afu_dma_unmap_region(pdata, map.iova);
+		return -EFAULT;
+	}
+
+	dev_dbg(&pdata->dev->dev, "dma map: ua=%llx, len=%llx, iova=%llx\n",
+		(unsigned long long)map.user_addr,
+		(unsigned long long)map.length,
+		(unsigned long long)map.iova);
+
+	return 0;
+}
+
+static long
+afu_ioctl_dma_unmap(struct feature_platform_data *pdata, void __user *arg)
+{
+	struct fpga_port_dma_unmap unmap;
+	unsigned long minsz;
+
+	minsz = offsetofend(struct fpga_port_dma_unmap, iova);
+
+	if (copy_from_user(&unmap, arg, minsz))
+		return -EFAULT;
+
+	if (unmap.argsz < minsz || unmap.flags)
+		return -EINVAL;
+
+	return afu_dma_unmap_region(pdata, unmap.iova);
+}
+
 static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
 {
 	struct platform_device *pdev = filp->private_data;

@@ -267,6 +320,10 @@ static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
 		return afu_ioctl_get_info(pdata, (void __user *)arg);
 	case FPGA_PORT_GET_REGION_INFO:
 		return afu_ioctl_get_region_info(pdata, (void __user *)arg);
+	case FPGA_PORT_DMA_MAP:
+		return afu_ioctl_dma_map(pdata, (void __user *)arg);
+	case FPGA_PORT_DMA_UNMAP:
+		return afu_ioctl_dma_unmap(pdata, (void __user *)arg);
 	default:
 		/*
 		 * Let sub-feature's ioctl function to handle the cmd

@@ -342,6 +399,7 @@ static int afu_dev_init(struct platform_device *pdev)
 	mutex_lock(&pdata->lock);
 	fpga_pdata_set_private(pdata, afu);
 	afu_region_init(pdata);
+	afu_dma_region_init(pdata);
 	mutex_unlock(&pdata->lock);
 	return 0;
 }

@@ -354,6 +412,7 @@ static int afu_dev_destroy(struct platform_device *pdev)
 	mutex_lock(&pdata->lock);
 	afu = fpga_pdata_get_private(pdata);
 	afu_region_destroy(pdata);
+	afu_dma_region_destroy(pdata);
 	fpga_pdata_set_private(pdata, NULL);
 	mutex_unlock(&pdata->lock);

diff --git a/drivers/fpga/dfl-afu.h b/drivers/fpga/dfl-afu.h
index af690cc..47165193 100644
--- a/drivers/fpga/dfl-afu.h
+++ b/drivers/fpga/dfl-afu.h

@@ -41,12 +41,32 @@ struct fpga_afu_region {
 };
 
 /**
+ * struct fpga_afu_dma_region - afu DMA region data structure
+ *
+ * @user_addr: region userspace virtual address.
+ * @length: region length.
+ * @iova: region IO virtual address.
+ * @pages: ptr to pages of this region.
+ * @node: rb tree node.
+ * @in_use: flag to indicate if this region is in_use.
+ */
+struct fpga_afu_dma_region {
+	u64 user_addr;
+	u64 length;
+	u64 iova;
+	struct page **pages;
+	struct rb_node node;
+	bool in_use;
+};
+
+/**
  * struct fpga_afu - afu device data structure
  *
  * @region_cur_offset: current region offset from start to the device fd.
  * @num_regions: num of regions.
  * @regions: the region link list of this afu feature device.
  * @num_umsgs: num of umsgs.
+ * @dma_regions: root of dma regions rb tree.
  * @pdata: afu platform device's pdata.
  */
 struct fpga_afu {

@@ -54,6 +74,7 @@ struct fpga_afu {
 	int num_regions;
 	u8 num_umsgs;
 	struct list_head regions;
+	struct rb_root dma_regions;
 
 	struct feature_platform_data *pdata;
 };

@@ -68,4 +89,12 @@ int afu_get_region_by_offset(struct feature_platform_data *pdata,
 			     u64 offset, u64 size,
 			     struct fpga_afu_region *pregion);
 
-#endif
+void afu_dma_region_init(struct feature_platform_data *pdata);
+void afu_dma_region_destroy(struct feature_platform_data *pdata);
+int afu_dma_map_region(struct feature_platform_data *pdata,
+		       u64 user_addr, u64 length, u64 *iova);
+int afu_dma_unmap_region(struct feature_platform_data *pdata, u64 iova);
+struct fpga_afu_dma_region *
+afu_dma_region_find(struct feature_platform_data *pdata, u64 iova, u64 size);
+
+#endif /* __DFL_AFU_H */

diff --git a/include/uapi/linux/fpga-dfl.h b/include/uapi/linux/fpga-dfl.h
index df41828..32f6336 100644
--- a/include/uapi/linux/fpga-dfl.h
+++ b/include/uapi/linux/fpga-dfl.h

@@ -112,6 +112,43 @@ struct fpga_port_region_info {
 
 #define FPGA_PORT_GET_REGION_INFO	_IO(FPGA_MAGIC, PORT_BASE + 2)
 
+/**
+ * FPGA_PORT_DMA_MAP - _IOWR(FPGA_MAGIC, PORT_BASE + 3,
+ *						struct fpga_port_dma_map)
+ *
+ * Map the dma memory per user_addr and length which are provided by caller.
+ * Driver fills the iova in provided struct afu_port_dma_map.
+ * This interface only accepts page-size aligned user memory for dma mapping.
+ * Return: 0 on success, -errno on failure.
+ */
+struct fpga_port_dma_map {
+	/* Input */
+	__u32 argsz;		/* Structure length */
+	__u32 flags;		/* Zero for now */
+	__u64 user_addr;        /* Process virtual address */
+	__u64 length;           /* Length of mapping (bytes)*/
+	/* Output */
+	__u64 iova;             /* IO virtual address */
+};
+
+#define FPGA_PORT_DMA_MAP	_IO(FPGA_MAGIC, PORT_BASE + 3)
+
+/**
+ * FPGA_PORT_DMA_UNMAP - _IOW(FPGA_MAGIC, PORT_BASE + 4,
+ *						struct fpga_port_dma_unmap)
+ *
+ * Unmap the dma memory per iova provided by caller.
+ * Return: 0 on success, -errno on failure.
+ */
+struct fpga_port_dma_unmap {
+	/* Input */
+	__u32 argsz;		/* Structure length */
+	__u32 flags;		/* Zero for now */
+	__u64 iova;		/* IO virtual address */
+};
+
+#define FPGA_PORT_DMA_UNMAP	_IO(FPGA_MAGIC, PORT_BASE + 4)
+
 /* IOCTLs for FME file descriptor */
 
 /**

-- 
2.7.4

[PATCH v4 23/24] fpga: dfl: afu: add user afu sub feature support

From: Wu Hao <hidden>
Date: 2018-02-13 09:37:19

From: Xiao Guangrong <redacted>

User Accelerated Function Unit sub feature exposes the MMIO region of
the AFU. After valid green bitstream (GBS) is programmed and port is
enabled, then this MMIO region could be accessed.

This patch adds support to enumerate the AFU MMIO region and expose it
to userspace via mmap file operation. Below interfaces are exposed to user:

Sysfs interface:
* /sys/class/fpga_region/<regionX>/<dfl-port.x>/afu_id
  Read-only. Indicate which green bitstream is programmed to this AFU.

Ioctl interfaces:
* FPGA_PORT_GET_INFO
  Provide info to userspace on the number of supported region.
  Only UAFU region is supported now.

* FPGA_PORT_GET_REGION_INFO
  Provide region information, including access permission, region size,
  offset from the start of device fd.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Xiao Guangrong <redacted>
Signed-off-by: Wu Hao <redacted>
---
v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
    add sysfs documentation.
    switched to GPLv2 license.
v3: rename driver to fpga-dfl-afu
    fix coding style and checkpatch issue.
    only allow afu_id to be read when port isn't in reset.
v4: rebase and add more comments in code.
    fix SPDX license issue.
---
 Documentation/ABI/testing/sysfs-platform-dfl-port |   9 +
 drivers/fpga/Makefile                             |   2 +-
 drivers/fpga/dfl-afu-main.c                       | 211 +++++++++++++++++++++-
 drivers/fpga/dfl-afu-region.c                     | 165 +++++++++++++++++
 drivers/fpga/dfl-afu.h                            |  71 ++++++++
 include/uapi/linux/fpga-dfl.h                     |  47 +++++
 6 files changed, 501 insertions(+), 4 deletions(-)
 create mode 100644 drivers/fpga/dfl-afu-region.c
 create mode 100644 drivers/fpga/dfl-afu.h

diff --git a/Documentation/ABI/testing/sysfs-platform-dfl-port b/Documentation/ABI/testing/sysfs-platform-dfl-port
index 79e8332..10bda5a 100644
--- a/Documentation/ABI/testing/sysfs-platform-dfl-port
+++ b/Documentation/ABI/testing/sysfs-platform-dfl-port

@@ -5,3 +5,12 @@ Contact:	Wu Hao <hao.wu@intel.com>
 Description:	Read-only. It returns id of this port. One DFL FPGA device
 		may have more than one port. Userspace could use this id to
 		distinguish different ports under same FPGA device.
+
+What:		/sys/bus/platform/devices/dfl-port.0/afu_id
+Date:		Februray 2018
+KernelVersion:  4.16
+Contact:	Wu Hao <hao.wu@intel.com>
+Description:	Read-only. User can program different green bitstreams (GBS) to
+		FPGA Accelerator Function Unit (AFU) for different functions.
+		It returns uuid which could be used to identify which GBS is
+		programmed in this AFU.

diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index 5c9607b..041e3cd1 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile

@@ -37,7 +37,7 @@ obj-$(CONFIG_FPGA_DFL_FME_REGION)	+= dfl-fme-region.o
 obj-$(CONFIG_FPGA_DFL_AFU)		+= dfl-afu.o
 
 dfl-fme-objs := dfl-fme-main.o dfl-fme-pr.o
-dfl-afu-objs := dfl-afu-main.o
+dfl-afu-objs := dfl-afu-main.o dfl-afu-region.o
 
 # Drivers for FPGAs which implement DFL
 obj-$(CONFIG_FPGA_DFL_PCI)		+= dfl-pci.o

diff --git a/drivers/fpga/dfl-afu-main.c b/drivers/fpga/dfl-afu-main.c
index bead242..9c0e4a8 100644
--- a/drivers/fpga/dfl-afu-main.c
+++ b/drivers/fpga/dfl-afu-main.c

@@ -16,9 +16,10 @@
 
 #include <linux/kernel.h>
 #include <linux/module.h>
+#include <linux/uaccess.h>
 #include <linux/fpga-dfl.h>
 
-#include "dfl.h"
+#include "dfl-afu.h"
 
 static ssize_t
 id_show(struct device *dev, struct device_attribute *attr, char *buf)

@@ -78,12 +79,74 @@ static const struct feature_ops port_hdr_ops = {
 	.ioctl = port_hdr_ioctl,
 };
 
+static ssize_t
+afu_id_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(dev);
+	void __iomem *base;
+	u64 guidl, guidh;
+
+	base = get_feature_ioaddr_by_id(dev, PORT_FEATURE_ID_AFU);
+
+	mutex_lock(&pdata->lock);
+	if (pdata->disable_count) {
+		mutex_unlock(&pdata->lock);
+		return -EBUSY;
+	}
+
+	guidl = readq(base + GUID_L);
+	guidh = readq(base + GUID_H);
+	mutex_unlock(&pdata->lock);
+
+	return scnprintf(buf, PAGE_SIZE, "%016llx%016llx\n", guidh, guidl);
+}
+static DEVICE_ATTR_RO(afu_id);
+
+static const struct attribute *port_uafu_attrs[] = {
+	&dev_attr_afu_id.attr,
+	NULL
+};
+
+static int port_uafu_init(struct platform_device *pdev, struct feature *feature)
+{
+	struct resource *res = &pdev->resource[feature->resource_index];
+	u32 flags = FPGA_REGION_READ | FPGA_REGION_WRITE | FPGA_REGION_MMAP;
+	int ret;
+
+	dev_dbg(&pdev->dev, "PORT AFU Init.\n");
+
+	ret = afu_region_add(dev_get_platdata(&pdev->dev),
+			     FPGA_PORT_INDEX_UAFU, resource_size(res),
+			     res->start, flags);
+	if (ret)
+		return ret;
+
+	return sysfs_create_files(&pdev->dev.kobj, port_uafu_attrs);
+}
+
+static void port_uafu_uinit(struct platform_device *pdev,
+			    struct feature *feature)
+{
+	dev_dbg(&pdev->dev, "PORT AFU UInit.\n");
+
+	sysfs_remove_files(&pdev->dev.kobj, port_uafu_attrs);
+}
+
+static const struct feature_ops port_uafu_ops = {
+	.init = port_uafu_init,
+	.uinit = port_uafu_uinit,
+};
+
 static struct feature_driver port_feature_drvs[] = {
 	{
 		.id = PORT_FEATURE_ID_HEADER,
 		.ops = &port_hdr_ops,
 	},
 	{
+		.id = PORT_FEATURE_ID_AFU,
+		.ops = &port_uafu_ops,
+	},
+	{
 		.ops = NULL,
 	}
 };

@@ -128,6 +191,64 @@ static long afu_ioctl_check_extension(struct feature_platform_data *pdata,
 	return 0;
 }
 
+static long
+afu_ioctl_get_info(struct feature_platform_data *pdata, void __user *arg)
+{
+	struct fpga_port_info info;
+	struct fpga_afu *afu;
+	unsigned long minsz;
+
+	minsz = offsetofend(struct fpga_port_info, num_umsgs);
+
+	if (copy_from_user(&info, arg, minsz))
+		return -EFAULT;
+
+	if (info.argsz < minsz)
+		return -EINVAL;
+
+	mutex_lock(&pdata->lock);
+	afu = fpga_pdata_get_private(pdata);
+	info.flags = 0;
+	info.num_regions = afu->num_regions;
+	info.num_umsgs = afu->num_umsgs;
+	mutex_unlock(&pdata->lock);
+
+	if (copy_to_user(arg, &info, sizeof(info)))
+		return -EFAULT;
+
+	return 0;
+}
+
+static long
+afu_ioctl_get_region_info(struct feature_platform_data *pdata, void __user *arg)
+{
+	struct fpga_port_region_info rinfo;
+	struct fpga_afu_region region;
+	unsigned long minsz;
+	long ret;
+
+	minsz = offsetofend(struct fpga_port_region_info, offset);
+
+	if (copy_from_user(&rinfo, arg, minsz))
+		return -EFAULT;
+
+	if (rinfo.argsz < minsz || rinfo.padding)
+		return -EINVAL;
+
+	ret = afu_get_region_by_index(pdata, rinfo.index, &region);
+	if (ret)
+		return ret;
+
+	rinfo.flags = region.flags;
+	rinfo.size = region.size;
+	rinfo.offset = region.offset;
+
+	if (copy_to_user(arg, &rinfo, sizeof(rinfo)))
+		return -EFAULT;
+
+	return 0;
+}
+
 static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
 {
 	struct platform_device *pdev = filp->private_data;

@@ -142,6 +263,10 @@ static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
 		return FPGA_API_VERSION;
 	case FPGA_CHECK_EXTENSION:
 		return afu_ioctl_check_extension(pdata, arg);
+	case FPGA_PORT_GET_INFO:
+		return afu_ioctl_get_info(pdata, (void __user *)arg);
+	case FPGA_PORT_GET_REGION_INFO:
+		return afu_ioctl_get_region_info(pdata, (void __user *)arg);
 	default:
 		/*
 		 * Let sub-feature's ioctl function to handle the cmd

@@ -162,27 +287,106 @@ static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
 	return -EINVAL;
 }
 
+static int afu_mmap(struct file *filp, struct vm_area_struct *vma)
+{
+	struct fpga_afu_region region;
+	struct platform_device *pdev = filp->private_data;
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	u64 size = vma->vm_end - vma->vm_start;
+	u64 offset;
+	int ret;
+
+	if (!(vma->vm_flags & VM_SHARED))
+		return -EINVAL;
+
+	offset = vma->vm_pgoff << PAGE_SHIFT;
+	ret = afu_get_region_by_offset(pdata, offset, size, &region);
+	if (ret)
+		return ret;
+
+	if (!(region.flags & FPGA_REGION_MMAP))
+		return -EINVAL;
+
+	if ((vma->vm_flags & VM_READ) && !(region.flags & FPGA_REGION_READ))
+		return -EPERM;
+
+	if ((vma->vm_flags & VM_WRITE) && !(region.flags & FPGA_REGION_WRITE))
+		return -EPERM;
+
+	vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
+
+	return remap_pfn_range(vma, vma->vm_start,
+			(region.phys + (offset - region.offset)) >> PAGE_SHIFT,
+			size, vma->vm_page_prot);
+}
+
 static const struct file_operations afu_fops = {
 	.owner = THIS_MODULE,
 	.open = afu_open,
 	.release = afu_release,
 	.unlocked_ioctl = afu_ioctl,
+	.mmap = afu_mmap,
 };
 
+static int afu_dev_init(struct platform_device *pdev)
+{
+	struct fpga_afu *afu;
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+
+	afu = devm_kzalloc(&pdev->dev, sizeof(*afu), GFP_KERNEL);
+	if (!afu)
+		return -ENOMEM;
+
+	afu->pdata = pdata;
+
+	mutex_lock(&pdata->lock);
+	fpga_pdata_set_private(pdata, afu);
+	afu_region_init(pdata);
+	mutex_unlock(&pdata->lock);
+	return 0;
+}
+
+static int afu_dev_destroy(struct platform_device *pdev)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	struct fpga_afu *afu;
+
+	mutex_lock(&pdata->lock);
+	afu = fpga_pdata_get_private(pdata);
+	afu_region_destroy(pdata);
+	fpga_pdata_set_private(pdata, NULL);
+	mutex_unlock(&pdata->lock);
+
+	devm_kfree(&pdev->dev, afu);
+
+	return 0;
+}
+
 static int afu_probe(struct platform_device *pdev)
 {
 	int ret;
 
 	dev_dbg(&pdev->dev, "%s\n", __func__);
 
+	ret = afu_dev_init(pdev);
+	if (ret)
+		goto exit;
+
 	ret = fpga_dev_feature_init(pdev, port_feature_drvs);
 	if (ret)
-		return ret;
+		goto dev_destroy;
 
 	ret = fpga_register_dev_ops(pdev, &afu_fops, THIS_MODULE);
-	if (ret)
+	if (ret) {
 		fpga_dev_feature_uinit(pdev);
+		goto dev_destroy;
+	}
+
+	return 0;
 
+dev_destroy:
+	afu_dev_destroy(pdev);
+exit:
 	return ret;
 }

@@ -192,6 +396,7 @@ static int afu_remove(struct platform_device *pdev)
 
 	fpga_dev_feature_uinit(pdev);
 	fpga_unregister_dev_ops(pdev);
+	afu_dev_destroy(pdev);
 
 	return 0;
 }

diff --git a/drivers/fpga/dfl-afu-region.c b/drivers/fpga/dfl-afu-region.c
new file mode 100644
index 0000000..e3a467b
--- /dev/null
+++ b/drivers/fpga/dfl-afu-region.c

@@ -0,0 +1,165 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Driver for FPGA Accelerated Function Unit (AFU) Region Management
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Wu Hao <hao.wu@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ */
+#include "dfl-afu.h"
+
+/**
+ * afu_region_init - init function for afu region support
+ * @pdata: afu platform device's pdata.
+ */
+void afu_region_init(struct feature_platform_data *pdata)
+{
+	struct fpga_afu *afu = fpga_pdata_get_private(pdata);
+
+	INIT_LIST_HEAD(&afu->regions);
+}
+
+#define for_each_region(region, afu)	\
+	list_for_each_entry((region), &(afu)->regions, node)
+
+static struct fpga_afu_region *get_region_by_index(struct fpga_afu *afu,
+						   u32 region_index)
+{
+	struct fpga_afu_region *region;
+
+	for_each_region(region, afu)
+		if (region->index == region_index)
+			return region;
+
+	return NULL;
+}
+
+/**
+ * afu_region_add - add a region to given feature dev.
+ *
+ * @region_index: region index.
+ * @region_size: region size.
+ * @phys: region's physical address of this region.
+ * @flags: region flags (access permission).
+ *
+ * Return: 0 on success, negative error code otherwise.
+ */
+int afu_region_add(struct feature_platform_data *pdata, u32 region_index,
+		   u64 region_size, u64 phys, u32 flags)
+{
+	struct fpga_afu_region *region;
+	struct fpga_afu *afu;
+	int ret = 0;
+
+	region = devm_kzalloc(&pdata->dev->dev, sizeof(*region), GFP_KERNEL);
+	if (!region)
+		return -ENOMEM;
+
+	region->index = region_index;
+	region->size = region_size;
+	region->phys = phys;
+	region->flags = flags;
+
+	mutex_lock(&pdata->lock);
+
+	afu = fpga_pdata_get_private(pdata);
+
+	/* check if @index already exists */
+	if (get_region_by_index(afu, region_index)) {
+		mutex_unlock(&pdata->lock);
+		ret = -EEXIST;
+		goto exit;
+	}
+
+	region_size = PAGE_ALIGN(region_size);
+	region->offset = afu->region_cur_offset;
+	list_add(&region->node, &afu->regions);
+
+	afu->region_cur_offset += region_size;
+	afu->num_regions++;
+	mutex_unlock(&pdata->lock);
+
+	return 0;
+
+exit:
+	devm_kfree(&pdata->dev->dev, region);
+	return ret;
+}
+
+/**
+ * afu_region_destroy - destroy all regions under given feature dev.
+ * @pdata: afu platform device's pdata.
+ */
+void afu_region_destroy(struct feature_platform_data *pdata)
+{
+	struct fpga_afu_region *tmp, *region;
+	struct fpga_afu *afu = fpga_pdata_get_private(pdata);
+
+	list_for_each_entry_safe(region, tmp, &afu->regions, node)
+		devm_kfree(&pdata->dev->dev, region);
+}
+
+/**
+ * afu_get_region_by_index - find an afu region by index.
+ * @pdata: afu platform device's pdata.
+ * @region_index: region index.
+ * @pregion: ptr to region for result.
+ *
+ * Return: 0 on success, negative error code otherwise.
+ */
+int afu_get_region_by_index(struct feature_platform_data *pdata,
+			    u32 region_index, struct fpga_afu_region *pregion)
+{
+	struct fpga_afu_region *region;
+	struct fpga_afu *afu;
+	int ret = 0;
+
+	mutex_lock(&pdata->lock);
+	afu = fpga_pdata_get_private(pdata);
+	region = get_region_by_index(afu, region_index);
+	if (!region) {
+		ret = -EINVAL;
+		goto exit;
+	}
+	*pregion = *region;
+exit:
+	mutex_unlock(&pdata->lock);
+	return ret;
+}
+
+/**
+ * afu_get_region_by_offset - find an afu region by offset and size
+ *
+ * @pdata: afu platform device's pdata.
+ * @offset: region offset from start of the device fd.
+ * @size: region size.
+ * @pregion: ptr to region for result.
+ *
+ * Find the region which fully contains the region described by input
+ * parameters (offset and size) from the feature dev's region link list.
+ *
+ * Return: 0 on success, negative error code otherwise.
+ */
+int afu_get_region_by_offset(struct feature_platform_data *pdata,
+			     u64 offset, u64 size,
+			     struct fpga_afu_region *pregion)
+{
+	struct fpga_afu_region *region;
+	struct fpga_afu *afu;
+	int ret = 0;
+
+	mutex_lock(&pdata->lock);
+	afu = fpga_pdata_get_private(pdata);
+	for_each_region(region, afu)
+		if (region->offset <= offset &&
+		    region->offset + region->size >= offset + size) {
+			*pregion = *region;
+			goto exit;
+		}
+	ret = -EINVAL;
+exit:
+	mutex_unlock(&pdata->lock);
+	return ret;
+}

diff --git a/drivers/fpga/dfl-afu.h b/drivers/fpga/dfl-afu.h
new file mode 100644
index 0000000..af690cc
--- /dev/null
+++ b/drivers/fpga/dfl-afu.h

@@ -0,0 +1,71 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Header file for FPGA Accelerated Function Unit (AFU) Driver
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *     Wu Hao <hao.wu@intel.com>
+ *     Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *     Joseph Grecco <joe.grecco@intel.com>
+ *     Enno Luebbers <enno.luebbers@intel.com>
+ *     Tim Whisonant <tim.whisonant@intel.com>
+ *     Ananda Ravuri <ananda.ravuri@intel.com>
+ *     Henry Mitchel <henry.mitchel@intel.com>
+ */
+
+#ifndef __DFL_AFU_H
+#define __DFL_AFU_H
+
+#include <linux/mm.h>
+
+#include "dfl.h"
+
+/**
+ * struct fpga_afu_region - afu region data structure
+ *
+ * @index: region index.
+ * @flags: region flags (access permission).
+ * @size: region size.
+ * @offset: region offset from start of the device fd.
+ * @phys: region's physical address.
+ * @node: node to add to afu feature dev's region list.
+ */
+struct fpga_afu_region {
+	u32 index;
+	u32 flags;
+	u64 size;
+	u64 offset;
+	u64 phys;
+	struct list_head node;
+};
+
+/**
+ * struct fpga_afu - afu device data structure
+ *
+ * @region_cur_offset: current region offset from start to the device fd.
+ * @num_regions: num of regions.
+ * @regions: the region link list of this afu feature device.
+ * @num_umsgs: num of umsgs.
+ * @pdata: afu platform device's pdata.
+ */
+struct fpga_afu {
+	u64 region_cur_offset;
+	int num_regions;
+	u8 num_umsgs;
+	struct list_head regions;
+
+	struct feature_platform_data *pdata;
+};
+
+void afu_region_init(struct feature_platform_data *pdata);
+int afu_region_add(struct feature_platform_data *pdata, u32 region_index,
+		   u64 region_size, u64 phys, u32 flags);
+void afu_region_destroy(struct feature_platform_data *pdata);
+int afu_get_region_by_index(struct feature_platform_data *pdata,
+			    u32 region_index, struct fpga_afu_region *pregion);
+int afu_get_region_by_offset(struct feature_platform_data *pdata,
+			     u64 offset, u64 size,
+			     struct fpga_afu_region *pregion);
+
+#endif

diff --git a/include/uapi/linux/fpga-dfl.h b/include/uapi/linux/fpga-dfl.h
index 727bec2..df41828 100644
--- a/include/uapi/linux/fpga-dfl.h
+++ b/include/uapi/linux/fpga-dfl.h

@@ -65,6 +65,53 @@
 
 #define FPGA_PORT_RESET		_IO(FPGA_MAGIC, PORT_BASE + 0)
 
+/**
+ * FPGA_PORT_GET_INFO - _IOR(FPGA_MAGIC, PORT_BASE + 1, struct fpga_port_info)
+ *
+ * Retrieve information about the fpga port.
+ * Driver fills the info in provided struct fpga_port_info.
+ * Return: 0 on success, -errno on failure.
+ */
+struct fpga_port_info {
+	/* Input */
+	__u32 argsz;		/* Structure length */
+	/* Output */
+	__u32 flags;		/* Zero for now */
+	__u32 num_regions;	/* The number of supported regions */
+	__u32 num_umsgs;	/* The number of allocated umsgs */
+};
+
+#define FPGA_PORT_GET_INFO	_IO(FPGA_MAGIC, PORT_BASE + 1)
+
+/**
+ * FPGA_PORT_GET_REGION_INFO - _IOWR(FPGA_MAGIC, PORT_BASE + 2,
+ *						struct fpga_port_region_info)
+ *
+ * Retrieve information about a device region.
+ * Caller provides struct fpga_port_region_info with index value set.
+ * Driver returns the region info in other fields.
+ * Return: 0 on success, -errno on failure.
+ */
+struct fpga_port_region_info {
+	/* input */
+	__u32 argsz;		/* Structure length */
+	/* Output */
+	__u32 flags;		/* Access permission */
+#define FPGA_REGION_READ	(1 << 0)	/* Region is readable */
+#define FPGA_REGION_WRITE	(1 << 1)	/* Region is writable */
+#define FPGA_REGION_MMAP	(1 << 2)	/* Can be mmaped to userspace */
+	/* Input */
+	__u32 index;		/* Region index */
+#define FPGA_PORT_INDEX_UAFU	0		/* User AFU */
+#define FPGA_PORT_INDEX_STP	1		/* Signal Tap */
+	__u32 padding;
+	/* Output */
+	__u64 size;		/* Region size (bytes) */
+	__u64 offset;		/* Region offset from start of device fd */
+};
+
+#define FPGA_PORT_GET_REGION_INFO	_IO(FPGA_MAGIC, PORT_BASE + 2)
+
 /* IOCTLs for FME file descriptor */
 
 /**

-- 
2.7.4

[PATCH v4 21/24] fpga: dfl: afu: add header sub feature support

From: Wu Hao <hidden>
Date: 2018-02-13 09:39:38

The port header register set is always present for port, it is mainly
for capability, control and status of the ports that AFU connected to.

This patch implements header sub feature support. Below user interfaces
are created by this patch.

Sysfs interface:
* /sys/class/fpga_region/<regionX>/<dfl-port.x>/id
  Read-only. Port ID.

Ioctl interface:
* FPGA_PORT_RESET
  Reset the FPGA Port and its AFU.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
Signed-off-by: Wu Hao <redacted>
Acked-by: Alan Tull <redacted>
---
v3: rename driver name to fpga-dfl-afu
    add more description for reset ioctl.
    fix some checkpatch issues.
v4: rebase.
    add Acked-by from Alan.
---
 Documentation/ABI/testing/sysfs-platform-dfl-port |  7 ++++
 drivers/fpga/dfl-afu-main.c                       | 44 ++++++++++++++++++++++-
 include/uapi/linux/fpga-dfl.h                     | 17 +++++++++
 3 files changed, 67 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/ABI/testing/sysfs-platform-dfl-port

diff --git a/Documentation/ABI/testing/sysfs-platform-dfl-port b/Documentation/ABI/testing/sysfs-platform-dfl-port
new file mode 100644
index 0000000..79e8332
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-platform-dfl-port

@@ -0,0 +1,7 @@
+What:		/sys/bus/platform/devices/dfl-port.0/id
+Date:		Februray 2018
+KernelVersion:  4.16
+Contact:	Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+Description:	Read-only. It returns id of this port. One DFL FPGA device
+		may have more than one port. Userspace could use this id to
+		distinguish different ports under same FPGA device.

diff --git a/drivers/fpga/dfl-afu-main.c b/drivers/fpga/dfl-afu-main.c
index 70db28c..b2f8798 100644
--- a/drivers/fpga/dfl-afu-main.c
+++ b/drivers/fpga/dfl-afu-main.c

@@ -16,25 +16,66 @@
 
 #include <linux/kernel.h>
 #include <linux/module.h>
+#include <linux/fpga-dfl.h>
 
 #include "dfl.h"
 
+static ssize_t
+id_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+	int id = fpga_port_id(to_platform_device(dev));
+
+	return scnprintf(buf, PAGE_SIZE, "%d\n", id);
+}
+static DEVICE_ATTR_RO(id);
+
+static const struct attribute *port_hdr_attrs[] = {
+	&dev_attr_id.attr,
+	NULL,
+};
+
 static int port_hdr_init(struct platform_device *pdev, struct feature *feature)
 {
 	dev_dbg(&pdev->dev, "PORT HDR Init.\n");
 
-	return 0;
+	fpga_port_reset(pdev);
+
+	return sysfs_create_files(&pdev->dev.kobj, port_hdr_attrs);
 }
 
 static void port_hdr_uinit(struct platform_device *pdev,
 			   struct feature *feature)
 {
 	dev_dbg(&pdev->dev, "PORT HDR UInit.\n");
+
+	sysfs_remove_files(&pdev->dev.kobj, port_hdr_attrs);
+}
+
+static long
+port_hdr_ioctl(struct platform_device *pdev, struct feature *feature,
+	       unsigned int cmd, unsigned long arg)
+{
+	long ret;
+
+	switch (cmd) {
+	case FPGA_PORT_RESET:
+		if (!arg)
+			ret = fpga_port_reset(pdev);
+		else
+			ret = -EINVAL;
+		break;
+	default:
+		dev_dbg(&pdev->dev, "%x cmd not handled", cmd);
+		ret = -ENODEV;
+	}
+
+	return ret;
 }
 
 static const struct feature_ops port_hdr_ops = {
 	.init = port_hdr_init,
 	.uinit = port_hdr_uinit,
+	.ioctl = port_hdr_ioctl,
 };
 
 static struct feature_driver port_feature_drvs[] = {

@@ -74,6 +115,7 @@ static int afu_release(struct inode *inode, struct file *filp)
 
 	dev_dbg(&pdev->dev, "Device File Release\n");
 
+	fpga_port_reset(pdev);
 	feature_dev_use_end(pdata);
 
 	return 0;

diff --git a/include/uapi/linux/fpga-dfl.h b/include/uapi/linux/fpga-dfl.h
index 50ee831..727bec2 100644
--- a/include/uapi/linux/fpga-dfl.h
+++ b/include/uapi/linux/fpga-dfl.h

@@ -28,8 +28,11 @@
 #define FPGA_MAGIC 0xB6
 
 #define FPGA_BASE 0
+#define PORT_BASE 0x40
 #define FME_BASE 0x80
 
+/* Common IOCTLs for both FME and AFU file descriptor */
+
 /**
  * FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)
  *

@@ -48,6 +51,20 @@
 
 #define FPGA_CHECK_EXTENSION	_IO(FPGA_MAGIC, FPGA_BASE + 1)
 
+/* IOCTLs for AFU file descriptor */
+
+/**
+ * FPGA_PORT_RESET - _IO(FPGA_MAGIC, PORT_BASE + 0)
+ *
+ * Reset the FPGA Port and its AFU. No parameters are supported.
+ * Userspace can do Port reset at any time, e.g during DMA or PR. But
+ * it should never cause any system level issue, only functional failure
+ * (e.g DMA or PR operation failure) and be recoverable from the failure.
+ * Return: 0 on success, -errno of failure
+ */
+
+#define FPGA_PORT_RESET		_IO(FPGA_MAGIC, PORT_BASE + 0)
+
 /* IOCTLs for FME file descriptor */
 
 /**

-- 
2.7.4

[PATCH v4 17/24] fpga: dfl: add fpga bridge platform driver for FME

From: Wu Hao <hidden>
Date: 2018-02-13 09:40:08

This patch adds fpga bridge platform driver for FPGA Management Engine.
It implements the enable_set call back for fpga bridge.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Wu Hao <redacted>
Acked-by: Alan Tull <redacted>
Acked-by: Moritz Fischer <redacted>
---
v3: rename driver to fpga-dfl-fme-br
    remove useless dev_dbg in probe function.
    rebased due to fpga api change.
v4: rename to dfl-fme-br and fix SPDX license issue
    include dfl-fme-pr.h instead of dfl-fme.h
    add Acked-by from Alan and Moritz
---
 drivers/fpga/Kconfig      |  6 ++++
 drivers/fpga/Makefile     |  1 +
 drivers/fpga/dfl-fme-br.c | 85 +++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 92 insertions(+)
 create mode 100644 drivers/fpga/dfl-fme-br.c

diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
index 89f76e8..a8f939a 100644
--- a/drivers/fpga/Kconfig
+++ b/drivers/fpga/Kconfig

@@ -156,6 +156,12 @@ config FPGA_DFL_FME_MGR
 	help
 	  Say Y to enable FPGA Manager driver for FPGA Management Engine.
 
+config FPGA_DFL_FME_BRIDGE
+	tristate "FPGA DFL FME Bridge Driver"
+	depends on FPGA_DFL_FME
+	help
+	  Say Y to enable FPGA Bridge driver for FPGA Management Engine.
+
 config FPGA_DFL_PCI
 	tristate "FPGA Device Feature List (DFL) PCIe Device Driver"
 	depends on PCI && FPGA_DFL

diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index f82814a..75096e9 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile

@@ -32,6 +32,7 @@ obj-$(CONFIG_OF_FPGA_REGION)		+= of-fpga-region.o
 obj-$(CONFIG_FPGA_DFL)			+= dfl.o
 obj-$(CONFIG_FPGA_DFL_FME)		+= dfl-fme.o
 obj-$(CONFIG_FPGA_DFL_FME_MGR)		+= dfl-fme-mgr.o
+obj-$(CONFIG_FPGA_DFL_FME_BRIDGE)	+= dfl-fme-br.o
 
 dfl-fme-objs := dfl-fme-main.o dfl-fme-pr.o

diff --git a/drivers/fpga/dfl-fme-br.c b/drivers/fpga/dfl-fme-br.c
new file mode 100644
index 0000000..46cee30
--- /dev/null
+++ b/drivers/fpga/dfl-fme-br.c

@@ -0,0 +1,85 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * FPGA Bridge Driver for FPGA Management Engine (FME)
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *   Joseph Grecco <joe.grecco-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *   Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *   Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *   Ananda Ravuri <ananda.ravuri-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *   Henry Mitchel <henry.mitchel-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ */
+
+#include <linux/module.h>
+#include <linux/fpga/fpga-bridge.h>
+
+#include "dfl.h"
+#include "dfl-fme-pr.h"
+
+static int fme_bridge_enable_set(struct fpga_bridge *bridge, bool enable)
+{
+	struct fme_br_pdata *pdata = bridge->priv;
+	int ret = 0;
+
+	if (enable)
+		fpga_port_enable(pdata->port);
+	else
+		ret = fpga_port_disable(pdata->port);
+
+	return ret;
+}
+
+static const struct fpga_bridge_ops fme_bridge_ops = {
+	.enable_set = fme_bridge_enable_set,
+};
+
+static int fme_br_probe(struct platform_device *pdev)
+{
+	struct device *dev = &pdev->dev;
+	struct fme_br_pdata *pdata = dev_get_platdata(dev);
+	struct fpga_bridge *br;
+	int ret;
+
+	br = devm_kzalloc(dev, sizeof(*br), GFP_KERNEL);
+	if (!br)
+		return -ENOMEM;
+
+	br->name = "FPGA FME Bridge";
+	br->br_ops = &fme_bridge_ops;
+	br->priv = pdata;
+	br->parent = dev;
+	platform_set_drvdata(pdev, br);
+
+	ret = fpga_bridge_register(br);
+	if (ret)
+		dev_err(dev, "unable to register FPGA Bridge\n");
+
+	return ret;
+}
+
+static int fme_br_remove(struct platform_device *pdev)
+{
+	struct fpga_bridge *br = platform_get_drvdata(pdev);
+
+	fpga_bridge_unregister(br);
+
+	return 0;
+}
+
+static struct platform_driver fme_br_driver = {
+	.driver	= {
+		.name    = FPGA_DFL_FME_BRIDGE,
+	},
+	.probe   = fme_br_probe,
+	.remove  = fme_br_remove,
+};
+
+module_platform_driver(fme_br_driver);
+
+MODULE_DESCRIPTION("FPGA Bridge for DFL FPGA Management Engine");
+MODULE_AUTHOR("Intel Corporation");
+MODULE_LICENSE("GPL v2");
+MODULE_ALIAS("platform:dfl-fme-bridge");

-- 
2.7.4

[PATCH v4 15/24] fpga: dfl-fme-pr: add compat_id support for dfl-fme-region platform device.

From: Wu Hao <hidden>
Date: 2018-02-13 09:40:39

This patch adds compat_id support when driver creates the platform
device for dfl-fme-region. It allows dfl-fme-region platform driver
to create fpga-region with correct compat_id.

Signed-off-by: Wu Hao <redacted>
---
 drivers/fpga/dfl-fme-pr.c | 20 ++++++++++++++++++++
 drivers/fpga/dfl-fme-pr.h |  3 +++
 2 files changed, 23 insertions(+)

diff --git a/drivers/fpga/dfl-fme-pr.c b/drivers/fpga/dfl-fme-pr.c
index 526e90b..c17170b 100644
--- a/drivers/fpga/dfl-fme-pr.c
+++ b/drivers/fpga/dfl-fme-pr.c

@@ -314,6 +314,25 @@ static void fpga_fme_destroy_bridges(struct feature_platform_data *pdata)
 }
 
 /**
+ * fpga_fme_get_region_compat_id - read region compat_id from hardware
+ *
+ * @dev: fme device.
+ */
+static struct fpga_region_compat_id
+fpga_fme_get_region_compat_id(struct device *dev)
+{
+	struct fpga_region_compat_id compat_id;
+	void __iomem *fme_pr;
+
+	fme_pr = get_feature_ioaddr_by_id(dev, FME_FEATURE_ID_PR_MGMT);
+
+	compat_id.id_l = readq(fme_pr + FME_PR_INTFC_ID_L);
+	compat_id.id_h = readq(fme_pr + FME_PR_INTFC_ID_H);
+
+	return compat_id;
+}
+
+/**
  * fpga_fme_create_region - create fpga region platform device as child
  *
  * @pdata: fme platform device's pdata

@@ -339,6 +358,7 @@ fpga_fme_create_region(struct feature_platform_data *pdata,
 
 	region_pdata.mgr = mgr;
 	region_pdata.br = br;
+	region_pdata.compat_id = fpga_fme_get_region_compat_id(dev);
 
 	/*
 	 * Each FPGA device may have more than one port, so allocate platform

diff --git a/drivers/fpga/dfl-fme-pr.h b/drivers/fpga/dfl-fme-pr.h
index 11bd001..ef6c95a 100644
--- a/drivers/fpga/dfl-fme-pr.h
+++ b/drivers/fpga/dfl-fme-pr.h

@@ -19,6 +19,7 @@
 #define __DFL_FME_PR_H
 
 #include <linux/platform_device.h>
+#include <linux/fpga/fpga-region.h>
 
 /**
  * struct fme_region - FME fpga region data structure

@@ -39,11 +40,13 @@ struct fme_region {
  * @mgr: platform device of the FPGA manager.
  * @br: platform device of the FPGA bridge.
  * @region_id: region id (same as port_id).
+ * @compat_id: compat id of the FPGA region.
  */
 struct fme_region_pdata {
 	struct platform_device *mgr;
 	struct platform_device *br;
 	int region_id;
+	struct fpga_region_compat_id compat_id;
 };
 
 /**

-- 
2.7.4

[PATCH v4 14/24] fpga: dfl: fme: add partial reconfiguration sub feature support

From: Wu Hao <hidden>
Date: 2018-02-13 09:41:15

From: Kang Luwei <redacted>

Partial Reconfiguration (PR) is the most important function for FME. It
allows reconfiguration for given Port/Accelerated Function Unit (AFU).

It creates platform devices for fpga-mgr, fpga-regions and fpga-bridges,
and invokes fpga-region's interface (fpga_region_program_fpga) for PR
operation once PR request received via ioctl. Below user space interface
is exposed by this sub feature.

Ioctl interface:
* FPGA_FME_PORT_PR
  Do partial reconfiguration per information from userspace, including
  target port(AFU), buffer size and address info. It returns error code
  to userspace if failed. For detailed PR error information, user needs
  to read fpga-mgr's status sysfs interface.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Kang Luwei <redacted>
Signed-off-by: Xiao Guangrong <redacted>
Signed-off-by: Wu Hao <redacted>
---
v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
    switched to GPLv2 license.
    removed status from FPGA_FME_PORT_PR ioctl data structure.
    added platform devices creation for fpga-mgr/fpga-region/fpga-bridge.
    switched to fpga-region interface fpga_region_program_fpga for PR.
    fixed comments from Alan Tull on FPGA_MGR_PARTIAL_RECONFIG flag usage.
    fixed kbuild warnings.
v3: rename driver files to dfl-fme-*.
    rebase due to fpga APIs change.
    replace bitfields.
    switch to fpga_cdev_find_port to find port device.
v4: rebase and correct comments for some function.
    fix SPDX license issue.
    remove unnecessary input parameter for destroy_bridge/region function.
    add dfl-fme-pr.h for PR sub feature data structure and registers.
---
 drivers/fpga/Makefile         |   2 +-
 drivers/fpga/dfl-fme-main.c   |  45 +++-
 drivers/fpga/dfl-fme-pr.c     | 497 ++++++++++++++++++++++++++++++++++++++++++
 drivers/fpga/dfl-fme-pr.h     | 113 ++++++++++
 drivers/fpga/dfl-fme.h        |  38 ++++
 include/uapi/linux/fpga-dfl.h |  27 +++
 6 files changed, 720 insertions(+), 2 deletions(-)
 create mode 100644 drivers/fpga/dfl-fme-pr.c
 create mode 100644 drivers/fpga/dfl-fme-pr.h
 create mode 100644 drivers/fpga/dfl-fme.h

diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index fbd1c85..3c44fc9 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile

@@ -32,7 +32,7 @@ obj-$(CONFIG_OF_FPGA_REGION)		+= of-fpga-region.o
 obj-$(CONFIG_FPGA_DFL)			+= dfl.o
 obj-$(CONFIG_FPGA_DFL_FME)		+= dfl-fme.o
 
-dfl-fme-objs := dfl-fme-main.o
+dfl-fme-objs := dfl-fme-main.o dfl-fme-pr.o
 
 # Drivers for FPGAs which implement DFL
 obj-$(CONFIG_FPGA_DFL_PCI)		+= dfl-pci.o

diff --git a/drivers/fpga/dfl-fme-main.c b/drivers/fpga/dfl-fme-main.c
index 1a9929c..967a44c 100644
--- a/drivers/fpga/dfl-fme-main.c
+++ b/drivers/fpga/dfl-fme-main.c

@@ -19,6 +19,7 @@
 #include <linux/fpga-dfl.h>
 
 #include "dfl.h"
+#include "dfl-fme.h"
 
 static ssize_t ports_num_show(struct device *dev,
 			      struct device_attribute *attr, char *buf)

@@ -111,6 +112,10 @@ static struct feature_driver fme_feature_drvs[] = {
 		.ops = &fme_hdr_ops,
 	},
 	{
+		.id = FME_FEATURE_ID_PR_MGMT,
+		.ops = &pr_mgmt_ops,
+	},
+	{
 		.ops = NULL,
 	},
 };

@@ -194,14 +199,49 @@ static const struct file_operations fme_fops = {
 	.unlocked_ioctl = fme_ioctl,
 };
 
+static int fme_dev_init(struct platform_device *pdev)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	struct fpga_fme *fme;
+
+	fme = devm_kzalloc(&pdev->dev, sizeof(*fme), GFP_KERNEL);
+	if (!fme)
+		return -ENOMEM;
+
+	fme->pdata = pdata;
+
+	mutex_lock(&pdata->lock);
+	fpga_pdata_set_private(pdata, fme);
+	mutex_unlock(&pdata->lock);
+
+	return 0;
+}
+
+static void fme_dev_destroy(struct platform_device *pdev)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	struct fpga_fme *fme;
+
+	mutex_lock(&pdata->lock);
+	fme = fpga_pdata_get_private(pdata);
+	fpga_pdata_set_private(pdata, NULL);
+	mutex_unlock(&pdata->lock);
+
+	devm_kfree(&pdev->dev, fme);
+}
+
 static int fme_probe(struct platform_device *pdev)
 {
 	int ret;
 
-	ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
+	ret = fme_dev_init(pdev);
 	if (ret)
 		goto exit;
 
+	ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
+	if (ret)
+		goto dev_destroy;
+
 	ret = fpga_register_dev_ops(pdev, &fme_fops, THIS_MODULE);
 	if (ret)
 		goto feature_uinit;

@@ -210,6 +250,8 @@ static int fme_probe(struct platform_device *pdev)
 
 feature_uinit:
 	fpga_dev_feature_uinit(pdev);
+dev_destroy:
+	fme_dev_destroy(pdev);
 exit:
 	return ret;
 }

@@ -218,6 +260,7 @@ static int fme_remove(struct platform_device *pdev)
 {
 	fpga_dev_feature_uinit(pdev);
 	fpga_unregister_dev_ops(pdev);
+	fme_dev_destroy(pdev);
 
 	return 0;
 }

diff --git a/drivers/fpga/dfl-fme-pr.c b/drivers/fpga/dfl-fme-pr.c
new file mode 100644
index 0000000..526e90b
--- /dev/null
+++ b/drivers/fpga/dfl-fme-pr.c

@@ -0,0 +1,497 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Driver for FPGA Management Engine (FME) Partial Reconfiguration
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Christopher Rauer <christopher.rauer@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ */
+
+#include <linux/types.h>
+#include <linux/device.h>
+#include <linux/vmalloc.h>
+#include <linux/uaccess.h>
+#include <linux/fpga/fpga-mgr.h>
+#include <linux/fpga/fpga-bridge.h>
+#include <linux/fpga/fpga-region.h>
+#include <linux/fpga-dfl.h>
+
+#include "dfl.h"
+#include "dfl-fme.h"
+#include "dfl-fme-pr.h"
+
+static struct fme_region *
+find_fme_region_by_port_id(struct fpga_fme *fme, int port_id)
+{
+	struct fme_region *fme_region;
+
+	list_for_each_entry(fme_region, &fme->region_list, node)
+		if (fme_region->port_id == port_id)
+			return fme_region;
+
+	return NULL;
+}
+
+static int fpga_fme_region_match(struct device *dev, const void *data)
+{
+	return dev->parent == data;
+}
+
+static struct fpga_region *
+fpga_fme_region_find(struct fpga_fme *fme, int port_id)
+{
+	struct fme_region *fme_region;
+	struct fpga_region *region;
+
+	fme_region = find_fme_region_by_port_id(fme, port_id);
+	if (!fme_region)
+		return NULL;
+
+	region = fpga_region_class_find(NULL, &fme_region->region->dev,
+					fpga_fme_region_match);
+	if (!region)
+		return NULL;
+
+	return region;
+}
+
+static int fme_pr(struct platform_device *pdev, unsigned long arg)
+{
+	void __user *argp = (void __user *)arg;
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	struct fpga_fme *fme;
+	struct fpga_image_info *info;
+	struct fpga_region *region;
+	struct fpga_fme_port_pr port_pr;
+	unsigned long minsz;
+	void __iomem *fme_hdr;
+	void *buf = NULL;
+	int ret = 0;
+	u64 v;
+
+	minsz = offsetofend(struct fpga_fme_port_pr, buffer_address);
+
+	if (copy_from_user(&port_pr, argp, minsz))
+		return -EFAULT;
+
+	if (port_pr.argsz < minsz || port_pr.flags)
+		return -EINVAL;
+
+	if (!IS_ALIGNED(port_pr.buffer_size, 4))
+		return -EINVAL;
+
+	/* get fme header region */
+	fme_hdr = get_feature_ioaddr_by_id(&pdev->dev,
+					   FME_FEATURE_ID_HEADER);
+
+	/* check port id */
+	v = readq(fme_hdr + FME_HDR_CAP);
+	if (port_pr.port_id >= FIELD_GET(FME_CAP_NUM_PORTS, v)) {
+		dev_dbg(&pdev->dev, "port number more than maximum\n");
+		return -EINVAL;
+	}
+
+	if (!access_ok(VERIFY_READ,
+		       (void __user *)(unsigned long)port_pr.buffer_address,
+		       port_pr.buffer_size))
+		return -EFAULT;
+
+	buf = vmalloc(port_pr.buffer_size);
+	if (!buf)
+		return -ENOMEM;
+
+	if (copy_from_user(buf,
+			   (void __user *)(unsigned long)port_pr.buffer_address,
+			   port_pr.buffer_size)) {
+		ret = -EFAULT;
+		goto free_exit;
+	}
+
+	/* prepare fpga_image_info for PR */
+	info = fpga_image_info_alloc(&pdev->dev);
+	if (!info) {
+		ret = -ENOMEM;
+		goto free_exit;
+	}
+
+	info->flags |= FPGA_MGR_PARTIAL_RECONFIG;
+
+	mutex_lock(&pdata->lock);
+	fme = fpga_pdata_get_private(pdata);
+	/* fme device has been unregistered. */
+	if (!fme) {
+		ret = -EINVAL;
+		goto unlock_exit;
+	}
+
+	region = fpga_fme_region_find(fme, port_pr.port_id);
+	if (!region) {
+		ret = -EINVAL;
+		goto unlock_exit;
+	}
+
+	fpga_image_info_free(region->info);
+
+	info->buf = buf;
+	info->count = port_pr.buffer_size;
+	info->region_id = port_pr.port_id;
+	region->info = info;
+
+	ret = fpga_region_program_fpga(region);
+
+	if (region->get_bridges)
+		fpga_bridges_put(&region->bridge_list);
+
+	put_device(&region->dev);
+unlock_exit:
+	mutex_unlock(&pdata->lock);
+free_exit:
+	vfree(buf);
+	if (copy_to_user((void __user *)arg, &port_pr, minsz))
+		return -EFAULT;
+
+	return ret;
+}
+
+/**
+ * fpga_fme_create_mgr - create fpga mgr platform device as child device
+ *
+ * @pdata: fme platform_device's pdata
+ *
+ * Return: mgr platform device if successful, and error code otherwise.
+ */
+static struct platform_device *
+fpga_fme_create_mgr(struct feature_platform_data *pdata)
+{
+	struct platform_device *mgr, *fme = pdata->dev;
+	struct feature *feature;
+	struct resource res;
+	struct resource *pres;
+	int ret = -ENOMEM;
+
+	feature = get_feature_by_id(&pdata->dev->dev, FME_FEATURE_ID_PR_MGMT);
+	if (!feature)
+		return ERR_PTR(-ENODEV);
+
+	/*
+	 * Each FME has only one fpga-mgr, so allocate platform device using
+	 * the same FME platform device id.
+	 */
+	mgr = platform_device_alloc(FPGA_DFL_FME_MGR, fme->id);
+	if (!mgr)
+		return ERR_PTR(ret);
+
+	mgr->dev.parent = &fme->dev;
+
+	pres = platform_get_resource(fme, IORESOURCE_MEM,
+				     feature->resource_index);
+	if (!pres) {
+		ret = -ENODEV;
+		goto create_mgr_err;
+	}
+
+	memset(&res, 0, sizeof(struct resource));
+
+	res.start = pres->start;
+	res.end = pres->end;
+	res.name = pres->name;
+	res.flags = IORESOURCE_MEM;
+
+	ret = platform_device_add_resources(mgr, &res, 1);
+	if (ret)
+		goto create_mgr_err;
+
+	ret = platform_device_add(mgr);
+	if (ret)
+		goto create_mgr_err;
+
+	return mgr;
+
+create_mgr_err:
+	platform_device_put(mgr);
+	return ERR_PTR(ret);
+}
+
+/**
+ * fpga_fme_destroy_mgr - destroy fpga mgr platform device
+ * @pdata: fme platform device's pdata
+ */
+static void fpga_fme_destroy_mgr(struct feature_platform_data *pdata)
+{
+	struct fpga_fme *priv = fpga_pdata_get_private(pdata);
+
+	platform_device_unregister(priv->mgr);
+}
+
+/**
+ * fpga_fme_create_bridge - create fme fpga bridge platform device as child
+ *
+ * @pdata: fme platform device's pdata
+ * @port_id: port id for the bridge to be created.
+ *
+ * Return: bridge platform device if successful, and error code otherwise.
+ */
+static struct fme_bridge *
+fpga_fme_create_bridge(struct feature_platform_data *pdata, int port_id)
+{
+	struct device *dev = &pdata->dev->dev;
+	struct fme_br_pdata br_pdata;
+	struct fme_bridge *fme_br;
+	int ret = -ENOMEM;
+
+	fme_br = devm_kzalloc(dev, sizeof(*fme_br), GFP_KERNEL);
+	if (!fme_br)
+		return ERR_PTR(ret);
+
+	br_pdata.port = fpga_cdev_find_port(fpga_pdata_to_fpga_cdev(pdata),
+					    &port_id, fpga_port_check_id);
+	if (!br_pdata.port)
+		return ERR_PTR(-ENODEV);
+
+	/*
+	 * Each FPGA device may have more than one port, so allocate platform
+	 * device using the same port platform device id.
+	 */
+	fme_br->br = platform_device_alloc(FPGA_DFL_FME_BRIDGE,
+					   br_pdata.port->id);
+	if (!fme_br->br) {
+		ret = -ENOMEM;
+		goto create_br_err;
+	}
+
+	fme_br->br->dev.parent = dev;
+
+	ret = platform_device_add_data(fme_br->br, &br_pdata, sizeof(br_pdata));
+	if (ret)
+		goto create_br_err;
+
+	ret = platform_device_add(fme_br->br);
+	if (ret)
+		goto create_br_err;
+
+	return fme_br;
+
+create_br_err:
+	platform_device_put(fme_br->br);
+	put_device(&br_pdata.port->dev);
+	return ERR_PTR(ret);
+}
+
+/**
+ * fpga_fme_destroy_bridge - destroy fpga bridge platform device
+ * @fme_br: fme bridge to destroy
+ */
+static void fpga_fme_destroy_bridge(struct fme_bridge *fme_br)
+{
+	struct fme_br_pdata *br_pdata = dev_get_platdata(&fme_br->br->dev);
+
+	put_device(&br_pdata->port->dev);
+	platform_device_unregister(fme_br->br);
+}
+
+/**
+ * fpga_fme_destroy_bridge - destroy all fpga bridge platform device
+ * @pdata: fme platform device's pdata
+ */
+static void fpga_fme_destroy_bridges(struct feature_platform_data *pdata)
+{
+	struct fpga_fme *priv = fpga_pdata_get_private(pdata);
+	struct fme_bridge *fbridge, *tmp;
+
+	list_for_each_entry_safe(fbridge, tmp, &priv->bridge_list, node) {
+		list_del(&fbridge->node);
+		fpga_fme_destroy_bridge(fbridge);
+	}
+}
+
+/**
+ * fpga_fme_create_region - create fpga region platform device as child
+ *
+ * @pdata: fme platform device's pdata
+ * @mgr: mgr platform device needed for region
+ * @br: br platform device needed for region
+ * @port_id: port id
+ *
+ * Return: fme region if successful, and error code otherwise.
+ */
+static struct fme_region *
+fpga_fme_create_region(struct feature_platform_data *pdata,
+		       struct platform_device *mgr,
+		       struct platform_device *br, int port_id)
+{
+	struct device *dev = &pdata->dev->dev;
+	struct fme_region_pdata region_pdata;
+	struct fme_region *fme_region;
+	int ret = -ENOMEM;
+
+	fme_region = devm_kzalloc(dev, sizeof(*fme_region), GFP_KERNEL);
+	if (!fme_region)
+		return ERR_PTR(ret);
+
+	region_pdata.mgr = mgr;
+	region_pdata.br = br;
+
+	/*
+	 * Each FPGA device may have more than one port, so allocate platform
+	 * device using the same port platform device id.
+	 */
+	fme_region->region = platform_device_alloc(FPGA_DFL_FME_REGION, br->id);
+	if (!fme_region->region)
+		return ERR_PTR(ret);
+
+	fme_region->region->dev.parent = dev;
+
+	ret = platform_device_add_data(fme_region->region, &region_pdata,
+				       sizeof(region_pdata));
+	if (ret)
+		goto create_region_err;
+
+	ret = platform_device_add(fme_region->region);
+	if (ret)
+		goto create_region_err;
+
+	fme_region->port_id = port_id;
+
+	return fme_region;
+
+create_region_err:
+	platform_device_put(fme_region->region);
+	return ERR_PTR(ret);
+}
+
+/**
+ * fpga_fme_destroy_region - destroy fme region
+ * @fme_region: fme region to destroy
+ */
+static void fpga_fme_destroy_region(struct fme_region *fme_region)
+{
+	platform_device_unregister(fme_region->region);
+}
+
+/**
+ * fpga_fme_destroy_regions - destroy all fme regions
+ * @pdata: fme platform device's pdata
+ */
+static void fpga_fme_destroy_regions(struct feature_platform_data *pdata)
+{
+	struct fpga_fme *priv = fpga_pdata_get_private(pdata);
+	struct fme_region *fme_region, *tmp;
+
+	list_for_each_entry_safe(fme_region, tmp, &priv->region_list, node) {
+		list_del(&fme_region->node);
+		fpga_fme_destroy_region(fme_region);
+	}
+}
+
+static int pr_mgmt_init(struct platform_device *pdev, struct feature *feature)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	void __iomem *fme_hdr;
+	struct platform_device *mgr;
+	struct fme_region *fme_region;
+	struct fme_bridge *fme_br;
+	struct fpga_fme *priv;
+	int ret = -ENODEV, i = 0;
+	u64 fme_cap, port_offset;
+
+	fme_hdr = get_feature_ioaddr_by_id(&pdev->dev,
+					   FME_FEATURE_ID_HEADER);
+
+	mutex_lock(&pdata->lock);
+	priv = fpga_pdata_get_private(pdata);
+
+	/* Initialize the region and bridge sub device list */
+	INIT_LIST_HEAD(&priv->region_list);
+	INIT_LIST_HEAD(&priv->bridge_list);
+
+	/* Create fpga mgr platform device */
+	mgr = fpga_fme_create_mgr(pdata);
+	if (IS_ERR(mgr)) {
+		dev_err(&pdev->dev, "fail to create fpga mgr pdev\n");
+		goto unlock;
+	}
+
+	priv->mgr = mgr;
+
+	/* Read capability register to check number of regions and bridges */
+	fme_cap = readq(fme_hdr + FME_HDR_CAP);
+	for (; i < FIELD_GET(FME_CAP_NUM_PORTS, fme_cap); i++) {
+		port_offset = readq(fme_hdr + FME_HDR_PORT_OFST(i));
+		if (!(port_offset & FME_PORT_OFST_IMP))
+			continue;
+
+		/* Create bridge for each port */
+		fme_br = fpga_fme_create_bridge(pdata, i);
+		if (IS_ERR(fme_br)) {
+			ret = PTR_ERR(fme_br);
+			goto destroy_region;
+		}
+
+		list_add(&fme_br->node, &priv->bridge_list);
+
+		/* Create region for each port */
+		fme_region = fpga_fme_create_region(pdata, mgr, fme_br->br, i);
+		if (!fme_region) {
+			ret = PTR_ERR(fme_region);
+			goto destroy_region;
+		}
+
+		list_add(&fme_region->node, &priv->region_list);
+	}
+	mutex_unlock(&pdata->lock);
+
+	return 0;
+
+destroy_region:
+	fpga_fme_destroy_regions(pdata);
+	fpga_fme_destroy_bridges(pdata);
+	fpga_fme_destroy_mgr(pdata);
+unlock:
+	mutex_unlock(&pdata->lock);
+	return ret;
+}
+
+static void pr_mgmt_uinit(struct platform_device *pdev, struct feature *feature)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	struct fpga_fme *priv;
+
+	mutex_lock(&pdata->lock);
+	priv = fpga_pdata_get_private(pdata);
+
+	fpga_fme_destroy_regions(pdata);
+	fpga_fme_destroy_bridges(pdata);
+	fpga_fme_destroy_mgr(pdata);
+	mutex_unlock(&pdata->lock);
+}
+
+static long fme_pr_ioctl(struct platform_device *pdev, struct feature *feature,
+			 unsigned int cmd, unsigned long arg)
+{
+	long ret;
+
+	switch (cmd) {
+	case FPGA_FME_PORT_PR:
+		ret = fme_pr(pdev, arg);
+		break;
+	default:
+		ret = -ENODEV;
+	}
+
+	return ret;
+}
+
+const struct feature_ops pr_mgmt_ops = {
+	.init = pr_mgmt_init,
+	.uinit = pr_mgmt_uinit,
+	.ioctl = fme_pr_ioctl,
+};

diff --git a/drivers/fpga/dfl-fme-pr.h b/drivers/fpga/dfl-fme-pr.h
new file mode 100644
index 0000000..11bd001
--- /dev/null
+++ b/drivers/fpga/dfl-fme-pr.h

@@ -0,0 +1,113 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Header file for FPGA Management Engine (FME) Partial Reconfiguration Driver
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ */
+
+#ifndef __DFL_FME_PR_H
+#define __DFL_FME_PR_H
+
+#include <linux/platform_device.h>
+
+/**
+ * struct fme_region - FME fpga region data structure
+ *
+ * @region: platform device of the FPGA region.
+ * @node: used to link fme_region to a list.
+ * @port_id: indicate which port this region connected to.
+ */
+struct fme_region {
+	struct platform_device *region;
+	struct list_head node;
+	int port_id;
+};
+
+/**
+ * struct fme_region_pdata - platform data for FME region platform device.
+ *
+ * @mgr: platform device of the FPGA manager.
+ * @br: platform device of the FPGA bridge.
+ * @region_id: region id (same as port_id).
+ */
+struct fme_region_pdata {
+	struct platform_device *mgr;
+	struct platform_device *br;
+	int region_id;
+};
+
+/**
+ * struct fme_bridge - FME fpga bridge data structure
+ *
+ * @br: platform device of the FPGA bridge.
+ * @node: used to link fme_bridge to a list.
+ */
+struct fme_bridge {
+	struct platform_device *br;
+	struct list_head node;
+};
+
+/**
+ * struct fme_bridge_pdata - platform data for FME bridge platform device.
+ *
+ * @port: platform device of the port feature dev.
+ */
+struct fme_br_pdata {
+	struct platform_device *port;
+};
+
+#define FPGA_DFL_FME_MGR	"dfl-fme-mgr"
+#define FPGA_DFL_FME_BRIDGE	"dfl-fme-bridge"
+#define FPGA_DFL_FME_REGION	"dfl-fme-region"
+
+/* FME Partial Reconfiguration Sub Feature Register Set */
+#define FME_PR_DFH		0x0
+#define FME_PR_CTRL		0x8
+#define FME_PR_STS		0x10
+#define FME_PR_DATA		0x18
+#define FME_PR_ERR		0x20
+#define FME_PR_INTFC_ID_H	0xA8
+#define FME_PR_INTFC_ID_L	0xB0
+
+/* FME PR Control Register Bitfield */
+#define FME_PR_CTRL_PR_RST	BIT(0)  /* Reset PR engine */
+#define FME_PR_CTRL_PR_RSTACK	BIT(4)  /* Ack for PR engine reset */
+#define FME_PR_CTRL_PR_RGN_ID	GENMASK_ULL(9, 7)       /* PR Region ID */
+#define FME_PR_CTRL_PR_START	BIT(12) /* Start to request for PR service */
+#define FME_PR_CTRL_PR_COMPLETE	BIT(13) /* PR data push complete notification */
+
+/* FME PR Status Register Bitfield */
+/* Number of available entries in HW queue inside the PR engine. */
+#define FME_PR_STS_PR_CREDIT	GENMASK_ULL(8, 0)
+#define FME_PR_STS_PR_STS	BIT(16) /* PR operation status */
+#define FME_PR_STS_PR_STS_IDLE	0
+#define FME_PR_STS_PR_CTRLR_STS	GENMASK_ULL(22, 20)     /* Controller status */
+#define FME_PR_STS_PR_HOST_STS	GENMASK_ULL(27, 24)     /* PR host status */
+
+/* FME PR Data Register Bitfield */
+/* PR data from the raw-binary file. */
+#define FME_PR_DATA_PR_DATA_RAW	GENMASK_ULL(32, 0)
+
+/* FME PR Error Register */
+/* PR Operation errors detected. */
+#define FME_PR_ERR_OPERATION_ERR	BIT(0)
+/* CRC error detected. */
+#define FME_PR_ERR_CRC_ERR		BIT(1)
+/* Incompatible PR bitstream detected. */
+#define FME_PR_ERR_INCOMPATIBLE_BS	BIT(2)
+/* PR data push protocol violated. */
+#define FME_PR_ERR_PROTOCOL_ERR		BIT(3)
+/* PR data fifo overflow error detected */
+#define FME_PR_ERR_FIFO_OVERFLOW	BIT(4)
+
+#endif /* __DFL_FME_PR_H */

diff --git a/drivers/fpga/dfl-fme.h b/drivers/fpga/dfl-fme.h
new file mode 100644
index 0000000..c8bd48a
--- /dev/null
+++ b/drivers/fpga/dfl-fme.h

@@ -0,0 +1,38 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Header file for FPGA Management Engine (FME) Driver
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ */
+
+#ifndef __DFL_FME_H
+#define __DFL_FME_H
+
+/**
+ * struct fpga_fme - fpga fme private data
+ *
+ * @mgr: FME's FPGA manager platform device.
+ * @region_list: link list of FME's FPGA regions.
+ * @bridge_list: link list of FME's FPGA bridges.
+ * @pdata: fme platform device's pdata.
+ */
+struct fpga_fme {
+	struct platform_device *mgr;
+	struct list_head region_list;
+	struct list_head bridge_list;
+	struct feature_platform_data *pdata;
+};
+
+extern const struct feature_ops pr_mgmt_ops;
+
+#endif /* __DFL_FME_H */

diff --git a/include/uapi/linux/fpga-dfl.h b/include/uapi/linux/fpga-dfl.h
index 9321ee9..50ee831 100644
--- a/include/uapi/linux/fpga-dfl.h
+++ b/include/uapi/linux/fpga-dfl.h

@@ -14,6 +14,8 @@
 #ifndef _UAPI_LINUX_FPGA_DFL_H
 #define _UAPI_LINUX_FPGA_DFL_H
 
+#include <linux/types.h>
+
 #define FPGA_API_VERSION 0
 
 /*

@@ -26,6 +28,7 @@
 #define FPGA_MAGIC 0xB6
 
 #define FPGA_BASE 0
+#define FME_BASE 0x80
 
 /**
  * FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)

@@ -45,4 +48,28 @@
 
 #define FPGA_CHECK_EXTENSION	_IO(FPGA_MAGIC, FPGA_BASE + 1)
 
+/* IOCTLs for FME file descriptor */
+
+/**
+ * FPGA_FME_PORT_PR - _IOW(FPGA_MAGIC, FME_BASE + 0, struct fpga_fme_port_pr)
+ *
+ * Driver does Partial Reconfiguration based on Port ID and Buffer (Image)
+ * provided by caller.
+ * Return: 0 on success, -errno on failure.
+ * If FPGA_FME_PORT_PR returns -EIO, that indicates the HW has detected
+ * some errors during PR, under this case, the user can fetch HW error info
+ * from the status of FME's fpga manager.
+ */
+
+struct fpga_fme_port_pr {
+	/* Input */
+	__u32 argsz;		/* Structure length */
+	__u32 flags;		/* Zero for now */
+	__u32 port_id;
+	__u32 buffer_size;
+	__u64 buffer_address;	/* Userspace address to the buffer for PR */
+};
+
+#define FPGA_FME_PORT_PR	_IO(FPGA_MAGIC, FME_BASE + 0)
+
 #endif /* _UAPI_LINUX_FPGA_DFL_H */

-- 
2.7.4

[PATCH v4 11/24] fpga: dfl: fme: add header sub feature support

From: Wu Hao <hidden>
Date: 2018-02-13 09:41:41

From: Kang Luwei <redacted>

The Header Register set is always present for FPGA Management Engine (FME),
this patch implements init and uinit function for header sub feature and
introduce several read-only sysfs interfaces for the capability and status.

Sysfs interfaces:
* /sys/class/fpga_region/<regionX>/<dfl-fme.x>/ports_num
  Read-only. Number of ports implemented

* /sys/class/fpga_region/<regionX>/<dfl-fme.x>/bitstream_id
  Read-only. Blue Bitstream (static FPGA region) identifier number. It contains
  the detailed version and other information of this static FPGA region.

* /sys/class/fpga_region/<regionX>/<dfl-fme.x>/bitstream_metadata
  Read-only. Blue Bitstream (static FPGA region) meta data. It contains the
  synthesis date, seed and other information of this static FPGA region.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Kang Luwei <redacted>
Signed-off-by: Xiao Guangrong <redacted>
Signed-off-by: Wu Hao <redacted>
---
v2: add sysfs documentation
v3: rename driver to fpga-dfl-fme.
    improve sysfs doc and commit description.
    replace bitfield.
v4: rebase and switch to use id for sub feature matching.
    add more description for bitstream_id/metadata.
---
 Documentation/ABI/testing/sysfs-platform-dfl-fme | 23 ++++++++
 drivers/fpga/dfl-fme-main.c                      | 68 ++++++++++++++++++++++++
 2 files changed, 91 insertions(+)
 create mode 100644 Documentation/ABI/testing/sysfs-platform-dfl-fme

diff --git a/Documentation/ABI/testing/sysfs-platform-dfl-fme b/Documentation/ABI/testing/sysfs-platform-dfl-fme
new file mode 100644
index 0000000..23a13a5
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-platform-dfl-fme

@@ -0,0 +1,23 @@
+What:		/sys/bus/platform/devices/dfl-fme.0/ports_num
+Date:		February 2018
+KernelVersion:  4.16
+Contact:	Wu Hao <hao.wu@intel.com>
+Description:	Read-only. One DFL FPGA device may have more than 1
+		port/Accelerator Function Unit (AFU). It returns the
+		number of ports on the FPGA device when read it.
+
+What:		/sys/bus/platform/devices/dfl-fme.0/bitstream_id
+Date:		February 2018
+KernelVersion:  4.16
+Contact:	Wu Hao <hao.wu@intel.com>
+Description:	Read-only. It returns Blue Bitstream (static FPGA region)
+		identifier number, which includes the detailed version
+		and other information of this static FPGA region.
+
+What:		/sys/bus/platform/devices/dfl-fme.0/bitstream_meta
+Date:		February 2018
+KernelVersion:  4.16
+Contact:	Wu Hao <hao.wu@intel.com>
+Description:	Read-only. It returns Blue Bitstream (static FPGA region)
+		meta data, which includes the synthesis date, seed and other
+		information of this static FPGA region.

diff --git a/drivers/fpga/dfl-fme-main.c b/drivers/fpga/dfl-fme-main.c
index ebe6b52..056ae24 100644
--- a/drivers/fpga/dfl-fme-main.c
+++ b/drivers/fpga/dfl-fme-main.c

@@ -19,9 +19,76 @@
 
 #include "dfl.h"
 
+static ssize_t ports_num_show(struct device *dev,
+			      struct device_attribute *attr, char *buf)
+{
+	void __iomem *base;
+	u64 v;
+
+	base = get_feature_ioaddr_by_id(dev, FME_FEATURE_ID_HEADER);
+
+	v = readq(base + FME_HDR_CAP);
+
+	return scnprintf(buf, PAGE_SIZE, "%u\n",
+			 (unsigned int)FIELD_GET(FME_CAP_NUM_PORTS, v));
+}
+static DEVICE_ATTR_RO(ports_num);
+
+/*
+ * Blue Bitstream (static FPGA region) identifier number. It contains the
+ * detailed version and other information of this static FPGA region.
+ */
+static ssize_t bitstream_id_show(struct device *dev,
+				 struct device_attribute *attr, char *buf)
+{
+	void __iomem *base;
+	u64 v;
+
+	base = get_feature_ioaddr_by_id(dev, FME_FEATURE_ID_HEADER);
+
+	v = readq(base + FME_HDR_BITSTREAM_ID);
+
+	return scnprintf(buf, PAGE_SIZE, "0x%llx\n", (unsigned long long)v);
+}
+static DEVICE_ATTR_RO(bitstream_id);
+
+/*
+ * Blue Bitstream (static FPGA region) meta data. It contains the synthesis
+ * date, seed and other information of this static FPGA region.
+ */
+static ssize_t bitstream_metadata_show(struct device *dev,
+				       struct device_attribute *attr, char *buf)
+{
+	void __iomem *base;
+	u64 v;
+
+	base = get_feature_ioaddr_by_id(dev, FME_FEATURE_ID_HEADER);
+
+	v = readq(base + FME_HDR_BITSTREAM_MD);
+
+	return scnprintf(buf, PAGE_SIZE, "0x%llx\n", (unsigned long long)v);
+}
+static DEVICE_ATTR_RO(bitstream_metadata);
+
+static const struct attribute *fme_hdr_attrs[] = {
+	&dev_attr_ports_num.attr,
+	&dev_attr_bitstream_id.attr,
+	&dev_attr_bitstream_metadata.attr,
+	NULL,
+};
+
 static int fme_hdr_init(struct platform_device *pdev, struct feature *feature)
 {
+	void __iomem *base = feature->ioaddr;
+	int ret;
+
 	dev_dbg(&pdev->dev, "FME HDR Init.\n");
+	dev_dbg(&pdev->dev, "FME cap %llx.\n",
+		(unsigned long long)readq(base + FME_HDR_CAP));
+
+	ret = sysfs_create_files(&pdev->dev.kobj, fme_hdr_attrs);
+	if (ret)
+		return ret;
 
 	return 0;
 }

@@ -29,6 +96,7 @@ static int fme_hdr_init(struct platform_device *pdev, struct feature *feature)
 static void fme_hdr_uinit(struct platform_device *pdev, struct feature *feature)
 {
 	dev_dbg(&pdev->dev, "FME HDR UInit.\n");
+	sysfs_remove_files(&pdev->dev.kobj, fme_hdr_attrs);
 }
 
 static const struct feature_ops fme_hdr_ops = {

-- 
2.7.4

[PATCH v4 10/24] fpga: dfl: add FPGA Management Engine driver basic framework

From: Wu Hao <hidden>
Date: 2018-02-13 09:42:40

From: Kang Luwei <redacted>

The FPGA Management Engine (FME) provides power, thermal management,
performance counters, partial reconfiguration and other functions. For each
function, it is packaged into a private feature linked to the FME feature
device in the 'Device Feature List'. It's a platform device created by
DFL framework.

This patch adds the basic framework of FME platform driver. It defines
sub feature drivers to handle the different sub features, including init,
uinit and ioctl. It also registers the file operations for the device file.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Kang Luwei <redacted>
Signed-off-by: Xiao Guangrong <redacted>
Signed-off-by: Wu Hao <redacted>
---
v3: rename driver from intel-fpga-fme to dfl-fme
    rename Kconfig from INTEL_FPGA_FME to FPGA_DFL_FME
v4: fix SPDX license issue, use dfl-fme as module name
---
 drivers/fpga/Kconfig        |  10 +++
 drivers/fpga/Makefile       |   3 +
 drivers/fpga/dfl-fme-main.c | 158 ++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 171 insertions(+)
 create mode 100644 drivers/fpga/dfl-fme-main.c

diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
index 87f3d44..103d5e2 100644
--- a/drivers/fpga/Kconfig
+++ b/drivers/fpga/Kconfig

@@ -140,6 +140,16 @@ config FPGA_DFL
 	  Gate Array (FPGA) solutions which implement Device Feature List.
 	  It provides enumeration APIs, and feature device infrastructure.
 
+config FPGA_DFL_FME
+	tristate "FPGA DFL FME Driver"
+	depends on FPGA_DFL
+	help
+	  The FPGA Management Engine (FME) is a feature device implemented
+	  under Device Feature List (DFL) framework. Select this option to
+	  enable the platform device driver for FME which implements all
+	  FPGA platform level management features. There shall be 1 FME
+	  per DFL based FPGA device.
+
 config FPGA_DFL_PCI
 	tristate "FPGA Device Feature List (DFL) PCIe Device Driver"
 	depends on PCI && FPGA_DFL

diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index 4375630..fbd1c85 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile

@@ -30,6 +30,9 @@ obj-$(CONFIG_OF_FPGA_REGION)		+= of-fpga-region.o
 
 # FPGA Device Feature List Support
 obj-$(CONFIG_FPGA_DFL)			+= dfl.o
+obj-$(CONFIG_FPGA_DFL_FME)		+= dfl-fme.o
+
+dfl-fme-objs := dfl-fme-main.o
 
 # Drivers for FPGAs which implement DFL
 obj-$(CONFIG_FPGA_DFL_PCI)		+= dfl-pci.o

diff --git a/drivers/fpga/dfl-fme-main.c b/drivers/fpga/dfl-fme-main.c
new file mode 100644
index 0000000..ebe6b52
--- /dev/null
+++ b/drivers/fpga/dfl-fme-main.c

@@ -0,0 +1,158 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Driver for FPGA Management Engine (FME)
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+
+#include "dfl.h"
+
+static int fme_hdr_init(struct platform_device *pdev, struct feature *feature)
+{
+	dev_dbg(&pdev->dev, "FME HDR Init.\n");
+
+	return 0;
+}
+
+static void fme_hdr_uinit(struct platform_device *pdev, struct feature *feature)
+{
+	dev_dbg(&pdev->dev, "FME HDR UInit.\n");
+}
+
+static const struct feature_ops fme_hdr_ops = {
+	.init = fme_hdr_init,
+	.uinit = fme_hdr_uinit,
+};
+
+static struct feature_driver fme_feature_drvs[] = {
+	{
+		.id = FME_FEATURE_ID_HEADER,
+		.ops = &fme_hdr_ops,
+	},
+	{
+		.ops = NULL,
+	},
+};
+
+static int fme_open(struct inode *inode, struct file *filp)
+{
+	struct platform_device *fdev = fpga_inode_to_feature_dev(inode);
+	struct feature_platform_data *pdata = dev_get_platdata(&fdev->dev);
+	int ret;
+
+	if (WARN_ON(!pdata))
+		return -ENODEV;
+
+	ret = feature_dev_use_begin(pdata);
+	if (ret)
+		return ret;
+
+	dev_dbg(&fdev->dev, "Device File Open\n");
+	filp->private_data = pdata;
+
+	return 0;
+}
+
+static int fme_release(struct inode *inode, struct file *filp)
+{
+	struct feature_platform_data *pdata = filp->private_data;
+	struct platform_device *pdev = pdata->dev;
+
+	dev_dbg(&pdev->dev, "Device File Release\n");
+	feature_dev_use_end(pdata);
+
+	return 0;
+}
+
+static long fme_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
+{
+	struct feature_platform_data *pdata = filp->private_data;
+	struct platform_device *pdev = pdata->dev;
+	struct feature *f;
+	long ret;
+
+	dev_dbg(&pdev->dev, "%s cmd 0x%x\n", __func__, cmd);
+
+	switch (cmd) {
+	default:
+		/*
+		 * Let sub-feature's ioctl function to handle the cmd
+		 * Sub-feature's ioctl returns -ENODEV when cmd is not
+		 * handled in this sub feature, and returns 0 and other
+		 * error code if cmd is handled.
+		 */
+		fpga_dev_for_each_feature(pdata, f) {
+			if (f->ops && f->ops->ioctl) {
+				ret = f->ops->ioctl(pdev, f, cmd, arg);
+				if (ret == -ENODEV)
+					continue;
+				else
+					return ret;
+			}
+		}
+	}
+
+	return -EINVAL;
+}
+
+static const struct file_operations fme_fops = {
+	.owner		= THIS_MODULE,
+	.open		= fme_open,
+	.release	= fme_release,
+	.unlocked_ioctl = fme_ioctl,
+};
+
+static int fme_probe(struct platform_device *pdev)
+{
+	int ret;
+
+	ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
+	if (ret)
+		goto exit;
+
+	ret = fpga_register_dev_ops(pdev, &fme_fops, THIS_MODULE);
+	if (ret)
+		goto feature_uinit;
+
+	return 0;
+
+feature_uinit:
+	fpga_dev_feature_uinit(pdev);
+exit:
+	return ret;
+}
+
+static int fme_remove(struct platform_device *pdev)
+{
+	fpga_dev_feature_uinit(pdev);
+	fpga_unregister_dev_ops(pdev);
+
+	return 0;
+}
+
+static struct platform_driver fme_driver = {
+	.driver	= {
+		.name    = FPGA_FEATURE_DEV_FME,
+	},
+	.probe   = fme_probe,
+	.remove  = fme_remove,
+};
+
+module_platform_driver(fme_driver);
+
+MODULE_DESCRIPTION("FPGA Management Engine driver");
+MODULE_AUTHOR("Intel Corporation");
+MODULE_LICENSE("GPL v2");
+MODULE_ALIAS("platform:dfl-fme");

-- 
2.7.4

[PATCH v4 06/24] fpga: dfl: adds fpga_cdev_find_port

From: Wu Hao <hidden>
Date: 2018-02-13 09:43:10

For feature devices, we need a method to find the port dedicated
to the device. This patch adds a function fpga_cdev_find_port
for this purpose. e.g FPGA Management Engine (FME) Partial
Reconfiguration sub feature, it uses this function to find
dedicated port on the device for PR function implementation.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Xiao Guangrong <redacted>
Signed-off-by: Wu Hao <redacted>
---
v3: s/fpga_for_each_port/fpga_cdev_find_port/
    move fpga_cdev_find_port to fpga-dfl module.
v4: improve description in commit message.
    add comments to remind user to put_device after use this function.
---
 drivers/fpga/dfl.c | 34 ++++++++++++++++++++++++++++++++++
 drivers/fpga/dfl.h | 20 ++++++++++++++++++++
 2 files changed, 54 insertions(+)

diff --git a/drivers/fpga/dfl.c b/drivers/fpga/dfl.c
index dcfe5ab..38dc819 100644
--- a/drivers/fpga/dfl.c
+++ b/drivers/fpga/dfl.c

@@ -783,6 +783,40 @@ void fpga_remove_feature_devs(struct fpga_cdev *cdev)
 }
 EXPORT_SYMBOL_GPL(fpga_remove_feature_devs);
 
+/**
+ * __fpga_cdev_find_port - find a port under given container device
+ *
+ * @cdev: container device
+ * @data: data passed to match function
+ * @match: match function used to find specific port from the port device list
+ *
+ * Find a port device under container device. This function needs to be
+ * invoked with lock held.
+ *
+ * This function returns NULL if the device doesn't match and non-zero if it
+ * does. If this function returns non-zero and a reference to the platform
+ * device of port can be obtained.
+ *
+ * NOTE: you will need to drop the device reference with put_device() after use.
+ */
+struct platform_device *
+__fpga_cdev_find_port(struct fpga_cdev *cdev, void *data,
+		      int (*match)(struct platform_device *, void *))
+{
+	struct feature_platform_data *pdata;
+	struct platform_device *port_dev;
+
+	list_for_each_entry(pdata, &cdev->port_dev_list, node) {
+		port_dev = pdata->dev;
+
+		if (match(port_dev, data) && get_device(&port_dev->dev))
+			return port_dev;
+	}
+
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(__fpga_cdev_find_port);
+
 int fpga_port_id(struct platform_device *pdev)
 {
 	void __iomem *base = get_feature_ioaddr_by_id(&pdev->dev,

diff --git a/drivers/fpga/dfl.h b/drivers/fpga/dfl.h
index 9b19399..d6ecda1 100644
--- a/drivers/fpga/dfl.h
+++ b/drivers/fpga/dfl.h

@@ -356,4 +356,24 @@ struct fpga_cdev {
 struct fpga_cdev *fpga_enumerate_feature_devs(struct fpga_enum_info *info);
 void fpga_remove_feature_devs(struct fpga_cdev *cdev);
 
+/*
+ * need to drop the device reference with put_device() after use port platform
+ * device returned by __fpga_cdev_find_port and fpga_cdev_find_port functions.
+ */
+struct platform_device *
+__fpga_cdev_find_port(struct fpga_cdev *cdev, void *data,
+		      int (*match)(struct platform_device *, void *));
+
+static inline struct platform_device *
+fpga_cdev_find_port(struct fpga_cdev *cdev, void *data,
+		    int (*match)(struct platform_device *, void *))
+{
+	struct platform_device *pdev;
+
+	mutex_lock(&cdev->lock);
+	pdev = __fpga_cdev_find_port(cdev, data, match);
+	mutex_unlock(&cdev->lock);
+
+	return pdev;
+}
 #endif /* __FPGA_DFL_H */

-- 
2.7.4

[PATCH v4 09/24] fpga: dfl-pci: add enumeration for feature devices

From: Wu Hao <hidden>
Date: 2018-02-13 09:43:10

The Device Feature List (DFL) is implemented in MMIO, and features
are linked via the DFLs. This patch enables pcie driver to prepare
enumeration information (e.g locations of all device feature lists
in MMIO) and use common APIs provided by the Device Feature List
framework to enumerate each feature device linked.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Zhang Yi <redacted>
Signed-off-by: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
Signed-off-by: Wu Hao <redacted>
---
v3: split from another patch
    use common functions from DFL framework for enumeration.
v4: rebase
---
 drivers/fpga/dfl-pci.c | 199 ++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 197 insertions(+), 2 deletions(-)

diff --git a/drivers/fpga/dfl-pci.c b/drivers/fpga/dfl-pci.c
index d91ea42..8ce8a94 100644
--- a/drivers/fpga/dfl-pci.c
+++ b/drivers/fpga/dfl-pci.c

@@ -22,9 +22,52 @@
 #include <linux/errno.h>
 #include <linux/aer.h>
 
+#include "dfl.h"
+
 #define DRV_VERSION	"0.8"
 #define DRV_NAME	"dfl-pci"
 
+struct cci_drvdata {
+	struct fpga_cdev *cdev;	/* container device */
+	struct list_head regions; /* list of pci bar mapping region */
+};
+
+/* pci bar mapping info */
+struct cci_region {
+	int bar;
+	void __iomem *ioaddr;	/* pointer to mapped bar region */
+	struct list_head node;
+};
+
+static void __iomem *cci_pci_ioremap_bar(struct pci_dev *pcidev, int bar)
+{
+	struct cci_drvdata *drvdata = pci_get_drvdata(pcidev);
+	struct cci_region *region;
+
+	list_for_each_entry(region, &drvdata->regions, node)
+		if (region->bar == bar) {
+			dev_dbg(&pcidev->dev, "BAR %d region exists\n", bar);
+			return region->ioaddr;
+		}
+
+	region = devm_kzalloc(&pcidev->dev, sizeof(*region), GFP_KERNEL);
+	if (!region)
+		return NULL;
+
+	region->bar = bar;
+	region->ioaddr = pci_ioremap_bar(pcidev, bar);
+	if (!region->ioaddr) {
+		dev_err(&pcidev->dev, "can't ioremap memory from BAR %d.\n",
+			bar);
+		devm_kfree(&pcidev->dev, region);
+		return NULL;
+	}
+
+	list_add(&region->node, &drvdata->regions);
+
+	return region->ioaddr;
+}
+
 /* PCI Device ID */
 #define PCIE_DEVICE_ID_PF_INT_5_X	0xBCBD
 #define PCIE_DEVICE_ID_PF_INT_6_X	0xBCC0

@@ -45,6 +88,143 @@ static struct pci_device_id cci_pcie_id_tbl[] = {
 };
 MODULE_DEVICE_TABLE(pci, cci_pcie_id_tbl);
 
+static int cci_init_drvdata(struct pci_dev *pcidev)
+{
+	struct cci_drvdata *drvdata;
+
+	drvdata = devm_kzalloc(&pcidev->dev, sizeof(*drvdata), GFP_KERNEL);
+	if (!drvdata)
+		return -ENOMEM;
+
+	INIT_LIST_HEAD(&drvdata->regions);
+
+	pci_set_drvdata(pcidev, drvdata);
+
+	return 0;
+}
+
+static void cci_pci_release_regions(struct pci_dev *pcidev)
+{
+	struct cci_drvdata *drvdata = pci_get_drvdata(pcidev);
+	struct cci_region *tmp, *region;
+
+	list_for_each_entry_safe(region, tmp, &drvdata->regions, node) {
+		list_del(&region->node);
+		if (region->ioaddr)
+			pci_iounmap(pcidev, region->ioaddr);
+		devm_kfree(&pcidev->dev, region);
+	}
+}
+
+static void cci_remove_drvdata(struct pci_dev *pcidev)
+{
+	struct cci_drvdata *drvdata = pci_get_drvdata(pcidev);
+
+	cci_pci_release_regions(pcidev);
+	pci_set_drvdata(pcidev, NULL);
+	devm_kfree(&pcidev->dev, drvdata);
+}
+
+static void cci_remove_feature_devs(struct pci_dev *pcidev)
+{
+	struct cci_drvdata *drvdata = pci_get_drvdata(pcidev);
+
+	/* remove all children feature devices */
+	fpga_remove_feature_devs(drvdata->cdev);
+}
+
+/* enumerate feature devices under pci device */
+static int cci_enumerate_feature_devs(struct pci_dev *pcidev)
+{
+	struct cci_drvdata *drvdata = pci_get_drvdata(pcidev);
+	struct fpga_cdev *cdev;
+	struct fpga_enum_info *info;
+	resource_size_t start, len;
+	void __iomem *base;
+	int port_num, bar, i, ret = 0;
+	u32 offset;
+	u64 v;
+
+	/* allocate enumeration info via pci_dev */
+	info = fpga_enum_info_alloc(&pcidev->dev);
+	if (!info)
+		return -ENOMEM;
+
+	/* start to find Device Feature List from Bar 0 */
+	base = cci_pci_ioremap_bar(pcidev, 0);
+	if (!base) {
+		ret = -ENOMEM;
+		goto enum_info_free_exit;
+	}
+
+	/*
+	 * PF device has FME and Ports/AFUs, and VF device only has 1 Port/AFU.
+	 * check them and add related "Device Feature List" info for the next
+	 * step enumeration.
+	 */
+	if (feature_is_fme(base)) {
+		start = pci_resource_start(pcidev, 0);
+		len = pci_resource_len(pcidev, 0);
+
+		fpga_enum_info_add_dfl(info, start, len, base);
+
+		/*
+		 * find more Device Feature Lists (e.g Ports) per information
+		 * indicated by FME module.
+		 */
+		v = readq(base + FME_HDR_CAP);
+		port_num = FIELD_GET(FME_CAP_NUM_PORTS, v);
+
+		WARN_ON(port_num > MAX_FPGA_PORT_NUM);
+
+		for (i = 0; i < port_num; i++) {
+			v = readq(base + FME_HDR_PORT_OFST(i));
+
+			/* skip ports which are not implemented. */
+			if (!(v & FME_PORT_OFST_IMP))
+				continue;
+
+			/*
+			 * add Port's Device Feature List information for next
+			 * step enumeration.
+			 */
+			bar = FIELD_GET(FME_PORT_OFST_BAR_ID, v);
+			offset = FIELD_GET(FME_PORT_OFST_DFH_OFST, v);
+			base = cci_pci_ioremap_bar(pcidev, bar);
+			if (!base)
+				continue;
+
+			start = pci_resource_start(pcidev, bar) + offset;
+			len = pci_resource_len(pcidev, bar) - offset;
+
+			fpga_enum_info_add_dfl(info, start, len, base + offset);
+		}
+	} else if (feature_is_port(base)) {
+		start = pci_resource_start(pcidev, 0);
+		len = pci_resource_len(pcidev, 0);
+
+		fpga_enum_info_add_dfl(info, start, len, base);
+	} else {
+		ret = -ENODEV;
+		goto enum_info_free_exit;
+	}
+
+	/* start enumeration with prepared enumeration information */
+	cdev = fpga_enumerate_feature_devs(info);
+	if (IS_ERR(cdev)) {
+		dev_err(&pcidev->dev, "Enumeration failure\n");
+		ret = PTR_ERR(cdev);
+		goto enum_info_free_exit;
+	}
+
+	drvdata->cdev = cdev;
+
+enum_info_free_exit:
+	fpga_enum_info_free(info);
+
+	return ret;
+}
+
 static
 int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
 {

@@ -82,9 +262,22 @@ int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
 		goto release_region_exit;
 	}
 
-	/* TODO: create and add the platform device per feature list */
-	return 0;
+	ret = cci_init_drvdata(pcidev);
+	if (ret) {
+		dev_err(&pcidev->dev, "Fail to init drvdata %d.\n", ret);
+		goto release_region_exit;
+	}
+
+	ret = cci_enumerate_feature_devs(pcidev);
+	if (ret) {
+		dev_err(&pcidev->dev, "enumeration failure %d.\n", ret);
+		goto remove_drvdata_exit;
+	}
+
+	return ret;
 
+remove_drvdata_exit:
+	cci_remove_drvdata(pcidev);
 release_region_exit:
 	pci_release_regions(pcidev);
 disable_error_report_exit:

@@ -95,6 +288,8 @@ int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
 
 static void cci_pci_remove(struct pci_dev *pcidev)
 {
+	cci_remove_feature_devs(pcidev);
+	cci_remove_drvdata(pcidev);
 	pci_release_regions(pcidev);
 	pci_disable_pcie_error_reporting(pcidev);
 	pci_disable_device(pcidev);

-- 
2.7.4

[PATCH v4 05/24] fpga: dfl: add chardev support for feature devices

From: Wu Hao <hidden>
Date: 2018-02-13 09:44:00

For feature devices drivers, both the FPGA Management Engine (FME) and
Accelerated Function Unit (AFU) driver need to expose user interfaces via
the device file, for example, mmap and ioctls.

This patch adds chardev support in the dfl driver for feature devices,
FME and AFU. It reserves the chardev regions for FME and AFU, and provide
interfaces for FME and AFU driver to register their device file operations.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Zhang Yi <redacted>
Signed-off-by: Xiao Guangrong <redacted>
Signed-off-by: Wu Hao <redacted>
---
v2: rebased
v3: move chardev support to fpga-dfl framework
v4: rebase, and add more comments in code.
---
 drivers/fpga/dfl.c | 104 ++++++++++++++++++++++++++++++++++++++++++++++++++++-
 drivers/fpga/dfl.h |  14 ++++++++
 2 files changed, 117 insertions(+), 1 deletion(-)

diff --git a/drivers/fpga/dfl.c b/drivers/fpga/dfl.c
index f50694e..dcfe5ab 100644
--- a/drivers/fpga/dfl.c
+++ b/drivers/fpga/dfl.c

@@ -74,6 +74,96 @@ static enum fpga_id_type feature_dev_id_type(struct platform_device *pdev)
 	return FPGA_ID_MAX;
 }
 
+struct fpga_chardev_info {
+	const char *name;
+	dev_t devt;
+};
+
+/* indexed by enum fpga_devt_type */
+struct fpga_chardev_info fpga_chrdevs[] = {
+	{.name = FPGA_FEATURE_DEV_FME},		/* FPGA_DEVT_FME */
+	{.name = FPGA_FEATURE_DEV_PORT},	/* FPGA_DEVT_AFU */
+};
+
+static void fpga_chardev_uinit(void)
+{
+	int i;
+
+	for (i = 0; i < FPGA_DEVT_MAX; i++)
+		if (MAJOR(fpga_chrdevs[i].devt)) {
+			unregister_chrdev_region(fpga_chrdevs[i].devt,
+						 MINORMASK);
+			fpga_chrdevs[i].devt = MKDEV(0, 0);
+		}
+}
+
+static int fpga_chardev_init(void)
+{
+	int i, ret;
+
+	for (i = 0; i < FPGA_DEVT_MAX; i++) {
+		ret = alloc_chrdev_region(&fpga_chrdevs[i].devt, 0, MINORMASK,
+					  fpga_chrdevs[i].name);
+		if (ret)
+			goto exit;
+	}
+
+	return 0;
+
+exit:
+	fpga_chardev_uinit();
+	return ret;
+}
+
+static dev_t fpga_get_devt(enum fpga_devt_type type, int id)
+{
+	WARN_ON(type >= FPGA_DEVT_MAX);
+
+	return MKDEV(MAJOR(fpga_chrdevs[type].devt), id);
+}
+
+/**
+ * fpga_register_dev_ops - register cdev ops for feature dev
+ *
+ * @pdev: feature dev.
+ * @fops: file operations for feature dev's cdev.
+ * @owner: owning module/driver.
+ *
+ * Return: 0 on success, negative error code otherwise.
+ */
+int fpga_register_dev_ops(struct platform_device *pdev,
+			  const struct file_operations *fops,
+			  struct module *owner)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+
+	cdev_init(&pdata->cdev, fops);
+	pdata->cdev.owner = owner;
+
+	/*
+	 * set parent to the feature device so that its refcount is
+	 * decreased after the last refcount of cdev is gone, that
+	 * makes sure the feature device is valid during device
+	 * file's life-cycle.
+	 */
+	pdata->cdev.kobj.parent = &pdev->dev.kobj;
+
+	return cdev_add(&pdata->cdev, pdev->dev.devt, 1);
+}
+EXPORT_SYMBOL_GPL(fpga_register_dev_ops);
+
+/**
+ * fpga_unregister_dev_ops - unregister cdev ops for feature dev
+ * @pdev: feature dev.
+ */
+void fpga_unregister_dev_ops(struct platform_device *pdev)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+
+	cdev_del(&pdata->cdev);
+}
+EXPORT_SYMBOL_GPL(fpga_unregister_dev_ops);
+
 /**
  * struct build_feature_devs_info - info collected during feature dev build.
  *

@@ -207,8 +297,12 @@ build_info_create_dev(struct build_feature_devs_info *binfo,
 		      void __iomem *ioaddr)
 {
 	struct platform_device *fdev;
+	enum fpga_devt_type devt_type = FPGA_DEVT_FME;
 	int ret;
 
+	if (type == PORT_ID)
+		devt_type = FPGA_DEVT_PORT;
+
 	/* we will create a new device, commit current device first */
 	ret = build_info_commit_dev(binfo);
 	if (ret)

@@ -232,6 +326,7 @@ build_info_create_dev(struct build_feature_devs_info *binfo,
 		return fdev->id;
 
 	fdev->dev.parent = &binfo->cdev->region.dev;
+	fdev->dev.devt = fpga_get_devt(devt_type, fdev->id);
 
 	return 0;
 }

@@ -769,13 +864,20 @@ EXPORT_SYMBOL_GPL(__fpga_port_disable);
 
 static int __init dfl_fpga_init(void)
 {
+	int ret;
+
 	fpga_ids_init();
 
-	return 0;
+	ret = fpga_chardev_init();
+	if (ret)
+		fpga_ids_destroy();
+
+	return ret;
 }
 
 static void __exit dfl_fpga_exit(void)
 {
+	fpga_chardev_uinit();
 	fpga_ids_destroy();
 }

diff --git a/drivers/fpga/dfl.h b/drivers/fpga/dfl.h
index 22dcf73..9b19399 100644
--- a/drivers/fpga/dfl.h
+++ b/drivers/fpga/dfl.h

@@ -15,6 +15,7 @@
 #define __FPGA_DFL_H
 
 #include <linux/bitfield.h>
+#include <linux/cdev.h>
 #include <linux/delay.h>
 #include <linux/fs.h>
 #include <linux/iopoll.h>

@@ -150,6 +151,7 @@ struct feature {
  *
  * @node: node to link feature devs to container device's port_dev_list.
  * @lock: mutex to protect platform data.
+ * @cdev: cdev of feature dev.
  * @dev: ptr to platform device linked with this platform data.
  * @disable_count: count for port disable.
  * @num: number for sub features.

@@ -158,6 +160,7 @@ struct feature {
 struct feature_platform_data {
 	struct list_head node;
 	struct mutex lock;
+	struct cdev cdev;
 	struct platform_device *dev;
 	unsigned int disable_count;

@@ -174,6 +177,17 @@ static inline int feature_platform_data_size(const int num)
 		num * sizeof(struct feature);
 }
 
+enum fpga_devt_type {
+	FPGA_DEVT_FME,
+	FPGA_DEVT_PORT,
+	FPGA_DEVT_MAX,
+};
+
+int fpga_register_dev_ops(struct platform_device *pdev,
+			  const struct file_operations *fops,
+			  struct module *owner);
+void fpga_unregister_dev_ops(struct platform_device *pdev);
+
 int fpga_port_id(struct platform_device *pdev);
 
 static inline int fpga_port_check_id(struct platform_device *pdev,

-- 
2.7.4

[PATCH v4 02/24] fpga: mgr: add region_id to fpga_image_info

From: Wu Hao <hidden>
Date: 2018-02-13 09:44:48

This patch adds region_id to fpga_image_info data structure, it
allows driver to pass region id information to fpga-mgr via
fpga_image_info for fpga reconfiguration function.

Signed-off-by: Wu Hao <redacted>
Acked-by: Moritz Fischer <mdf@kernel.org>
Acked-by: Alan Tull <atull@kernel.org>
---
v3: add one line comment for region_id
v4: add Acked-by from Moritz and Alan.
---
 include/linux/fpga/fpga-mgr.h | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/include/linux/fpga/fpga-mgr.h b/include/linux/fpga/fpga-mgr.h
index 1ad4822..6e98b66 100644
--- a/include/linux/fpga/fpga-mgr.h
+++ b/include/linux/fpga/fpga-mgr.h

@@ -88,6 +88,7 @@ enum fpga_mgr_states {
  * @sgt: scatter/gather table containing FPGA image
  * @buf: contiguous buffer containing FPGA image
  * @count: size of buf
+ * @region_id: id of target region
  * @dev: device that owns this
  * @overlay: Device Tree overlay
  */

@@ -100,6 +101,7 @@ struct fpga_image_info {
 	struct sg_table *sgt;
 	const char *buf;
 	size_t count;
+	int region_id;
 	struct device *dev;
 #ifdef CONFIG_OF
 	struct device_node *overlay;

-- 
2.7.4

[PATCH v4 04/24] fpga: add device feature list support

From: Wu Hao <hidden>
Date: 2018-02-13 09:44:49

Device Feature List (DFL) defines a feature list structure that creates
a link list of feature headers within the MMIO space to provide an
extensible way of adding features. This patch introduces a kernel module
to provide basic infrastructure to support FPGA devices which implement
the Device Feature List.

Usually there will be different features and their sub features linked into
the DFL. This code provides common APIs for feature enumeration, it creates
a container device (FPGA base region), walks through the DFLs and creates
platform devices for feature devices (Currently it only supports two
different feature devices, FPGA Management Engine (FME) and Port which
the Accelerator Function Unit (AFU) connected to). In order to enumerate
the DFLs, the common APIs required low level driver to provide necessary
enumeration information (e.g address for each device feature list for
given device) and fill it to the fpga_enum_info data structure. Please
refer to below description for APIs added for enumeration.

Functions for enumeration information preparation:
 *fpga_enum_info_alloc
   allocate enumeration information data structure.

 *fpga_enum_info_add_dfl
   add a device feature list to fpga_enum_info data structure.

 *fpga_enum_info_free
   free fpga_enum_info data structure and related resources.

Functions for feature device enumeration:
 *fpga_enumerate_feature_devs
   enumerate feature devices and return container device.

 *fpga_remove_feature_devs
   remove feature devices under given container device.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Zhang Yi <redacted>
Signed-off-by: Xiao Guangrong <redacted>
Signed-off-by: Wu Hao <redacted>
---
v3: split from another patch.
    separate dfl enumeration code from original pcie driver.
    provide common data structures and APIs for enumeration.
    update device feature list parsing process according to latest hw.
    add dperf/iperf/hssi sub feature placeholder according to latest hw.
    remove build_info_add_sub_feature and other small functions.
    replace *_feature_num function with macro.
    remove writeq/readq.
v4: fix SPDX license issue
    rename files to dfl.[ch], fix typo and add more comments.
    remove static feature_info tables for FME and Port.
    remove check on next_afu link list as only FIU has next_afu ptr.
    remove unused macro in header file.
    add more comments for functions.
---
 drivers/fpga/Kconfig  |  16 +
 drivers/fpga/Makefile |   3 +
 drivers/fpga/dfl.c    | 787 ++++++++++++++++++++++++++++++++++++++++++++++++++
 drivers/fpga/dfl.h    | 345 ++++++++++++++++++++++
 4 files changed, 1151 insertions(+)
 create mode 100644 drivers/fpga/dfl.c
 create mode 100644 drivers/fpga/dfl.h

diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
index f47ef84..01ad31f 100644
--- a/drivers/fpga/Kconfig
+++ b/drivers/fpga/Kconfig

@@ -124,4 +124,20 @@ config OF_FPGA_REGION
 	  Support for loading FPGA images by applying a Device Tree
 	  overlay.
 
+config FPGA_DFL
+	tristate "FPGA Device Feature List (DFL) support"
+	select FPGA_BRIDGE
+	select FPGA_REGION
+	help
+	  Device Feature List (DFL) defines a feature list structure that
+	  creates a link list of feature headers within the MMIO space
+	  to provide an extensible way of adding features for FPGA.
+	  Driver can walk through the feature headers to enumerate feature
+	  devices (e.g FPGA Management Engine, Port and Accelerator
+	  Function Unit) and their private features for target FPGA devices.
+
+	  Select this option to enable common support for Field-Programmable
+	  Gate Array (FPGA) solutions which implement Device Feature List.
+	  It provides enumeration APIs, and feature device infrastructure.
+
 endif # FPGA

diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index 3cb276a..c4c62b9 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile

@@ -27,3 +27,6 @@ obj-$(CONFIG_XILINX_PR_DECOUPLER)	+= xilinx-pr-decoupler.o
 # High Level Interfaces
 obj-$(CONFIG_FPGA_REGION)		+= fpga-region.o
 obj-$(CONFIG_OF_FPGA_REGION)		+= of-fpga-region.o
+
+# FPGA Device Feature List Support
+obj-$(CONFIG_FPGA_DFL)			+= dfl.o

diff --git a/drivers/fpga/dfl.c b/drivers/fpga/dfl.c
new file mode 100644
index 0000000..f50694e
--- /dev/null
+++ b/drivers/fpga/dfl.c

@@ -0,0 +1,787 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Driver for FPGA Device Feature List (DFL) Support
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Zhang Yi <yi.z.zhang@intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ */
+#include <linux/module.h>
+
+#include "dfl.h"
+
+static DEFINE_MUTEX(fpga_id_mutex);
+
+enum fpga_id_type {
+	FME_ID,		/* fme id allocation and mapping */
+	PORT_ID,	/* port id allocation and mapping */
+	FPGA_ID_MAX,
+};
+
+/* it is protected by fpga_id_mutex */
+static struct idr fpga_ids[FPGA_ID_MAX];
+
+static void fpga_ids_init(void)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(fpga_ids); i++)
+		idr_init(fpga_ids + i);
+}
+
+static void fpga_ids_destroy(void)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(fpga_ids); i++)
+		idr_destroy(fpga_ids + i);
+}
+
+static int alloc_fpga_id(enum fpga_id_type type, struct device *dev)
+{
+	int id;
+
+	WARN_ON(type >= FPGA_ID_MAX);
+	mutex_lock(&fpga_id_mutex);
+	id = idr_alloc(fpga_ids + type, dev, 0, 0, GFP_KERNEL);
+	mutex_unlock(&fpga_id_mutex);
+
+	return id;
+}
+
+static void free_fpga_id(enum fpga_id_type type, int id)
+{
+	WARN_ON(type >= FPGA_ID_MAX);
+	mutex_lock(&fpga_id_mutex);
+	idr_remove(fpga_ids + type, id);
+	mutex_unlock(&fpga_id_mutex);
+}
+
+static enum fpga_id_type feature_dev_id_type(struct platform_device *pdev)
+{
+	if (!strcmp(pdev->name, FPGA_FEATURE_DEV_FME))
+		return FME_ID;
+
+	if (!strcmp(pdev->name, FPGA_FEATURE_DEV_PORT))
+		return PORT_ID;
+
+	WARN_ON(1);
+
+	return FPGA_ID_MAX;
+}
+
+/**
+ * struct build_feature_devs_info - info collected during feature dev build.
+ *
+ * @dev: device to enumerate.
+ * @cdev: the container device for all feature devices.
+ * @feature_dev: current feature device.
+ * @ioaddr: header register region address of feature device in enumeration.
+ * @sub_features: a sub features link list for feature device in enumeration.
+ * @feature_num: number of sub features for feature device in enumeration.
+ */
+struct build_feature_devs_info {
+	struct device *dev;
+	struct fpga_cdev *cdev;
+	struct platform_device *feature_dev;
+	void __iomem *ioaddr;
+	struct list_head sub_features;
+	int feature_num;
+};
+
+/**
+ * struct feature_info - sub feature info collected during feature dev build.
+ *
+ * @fid: id of this sub feature.
+ * @mmio_res: mmio resource of this sub feature.
+ * @ioaddr: mapped base address of mmio resource.
+ * @node: node in sub_features link list.
+ */
+struct feature_info {
+	u64 fid;
+	struct resource mmio_res;
+	void __iomem *ioaddr;
+	struct list_head node;
+};
+
+static void fpga_cdev_add_port_dev(struct fpga_cdev *cdev,
+				   struct platform_device *port_pdev)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&port_pdev->dev);
+
+	mutex_lock(&cdev->lock);
+	list_add(&pdata->node, &cdev->port_dev_list);
+	get_device(&pdata->dev->dev);
+	mutex_unlock(&cdev->lock);
+}
+
+/*
+ * register current feature device, it is called when we need to switch to
+ * another feature parsing or we have parsed all features on given device
+ * feature list.
+ */
+static int build_info_commit_dev(struct build_feature_devs_info *binfo)
+{
+	struct platform_device *fdev = binfo->feature_dev;
+	struct feature_platform_data *pdata;
+	struct feature_info *finfo, *p;
+	int ret, index = 0;
+
+	if (!fdev)
+		return 0;
+
+	/*
+	 * we do not need to care for the memory which is associated with
+	 * the platform device. After calling platform_device_unregister(),
+	 * it will be automatically freed by device's release() callback,
+	 * platform_device_release().
+	 */
+	pdata = kzalloc(feature_platform_data_size(binfo->feature_num),
+			GFP_KERNEL);
+	if (pdata) {
+		pdata->dev = fdev;
+		pdata->num = binfo->feature_num;
+		mutex_init(&pdata->lock);
+	} else {
+		return -ENOMEM;
+	}
+
+	/*
+	 * the count should be initialized to 0 to make sure
+	 *__fpga_port_enable() following __fpga_port_disable()
+	 * works properly for port device.
+	 * and it should always be 0 for fme device.
+	 */
+	WARN_ON(pdata->disable_count);
+
+	fdev->dev.platform_data = pdata;
+
+	/* each sub feature has one MMIO resource */
+	fdev->num_resources = binfo->feature_num;
+	fdev->resource = kcalloc(binfo->feature_num, sizeof(*fdev->resource),
+				 GFP_KERNEL);
+	if (!fdev->resource)
+		return -ENOMEM;
+
+	/* fill features and resource information for feature dev */
+	list_for_each_entry_safe(finfo, p, &binfo->sub_features, node) {
+		struct feature *feature = &pdata->features[index];
+
+		/* save resource information for each feature */
+		feature->id = finfo->fid;
+		feature->resource_index = index;
+		feature->ioaddr = finfo->ioaddr;
+		fdev->resource[index++] = finfo->mmio_res;
+
+		list_del(&finfo->node);
+		kfree(finfo);
+	}
+
+	ret = platform_device_add(binfo->feature_dev);
+	if (!ret) {
+		if (feature_dev_id_type(binfo->feature_dev) == PORT_ID)
+			fpga_cdev_add_port_dev(binfo->cdev, binfo->feature_dev);
+		else
+			binfo->cdev->fme_dev =
+					get_device(&binfo->feature_dev->dev);
+		/*
+		 * reset it to avoid build_info_free() freeing their resource.
+		 *
+		 * The resource of successfully registered feature devices
+		 * will be freed by platform_device_unregister(). See the
+		 * comments in build_info_create_dev().
+		 */
+		binfo->feature_dev = NULL;
+	}
+
+	return ret;
+}
+
+static int
+build_info_create_dev(struct build_feature_devs_info *binfo,
+		      enum fpga_id_type type, const char *name,
+		      void __iomem *ioaddr)
+{
+	struct platform_device *fdev;
+	int ret;
+
+	/* we will create a new device, commit current device first */
+	ret = build_info_commit_dev(binfo);
+	if (ret)
+		return ret;
+
+	/*
+	 * we use -ENODEV as the initialization indicator which indicates
+	 * whether the id need to be reclaimed
+	 */
+	fdev = platform_device_alloc(name, -ENODEV);
+	if (!fdev)
+		return -ENOMEM;
+
+	binfo->feature_dev = fdev;
+	binfo->feature_num = 0;
+	binfo->ioaddr = ioaddr;
+	INIT_LIST_HEAD(&binfo->sub_features);
+
+	fdev->id = alloc_fpga_id(type, &fdev->dev);
+	if (fdev->id < 0)
+		return fdev->id;
+
+	fdev->dev.parent = &binfo->cdev->region.dev;
+
+	return 0;
+}
+
+static void build_info_free(struct build_feature_devs_info *binfo)
+{
+	struct feature_info *finfo, *p;
+
+	/*
+	 * it is a valid id, free it. See comments in
+	 * build_info_create_dev()
+	 */
+	if (binfo->feature_dev && binfo->feature_dev->id >= 0) {
+		free_fpga_id(feature_dev_id_type(binfo->feature_dev),
+			     binfo->feature_dev->id);
+
+		list_for_each_entry_safe(finfo, p, &binfo->sub_features, node) {
+			list_del(&finfo->node);
+			kfree(finfo);
+		}
+	}
+
+	platform_device_put(binfo->feature_dev);
+
+	devm_kfree(binfo->dev, binfo);
+}
+
+static inline u32 feature_size(void __iomem *start)
+{
+	u64 v = readq(start + DFH);
+	u32 ofst = FIELD_GET(DFH_NEXT_HDR_OFST, v);
+	/* workaround for private features with invalid size, use 4K instead */
+	return ofst ? ofst : 4096;
+}
+
+static u64 feature_id(void __iomem *start)
+{
+	u64 v = readq(start + DFH);
+	u16 id = FIELD_GET(DFH_ID, v);
+	u8 type = FIELD_GET(DFH_TYPE, v);
+
+	if (type == DFH_TYPE_FIU)
+		return FEATURE_ID_FIU_HEADER;
+	else if (type == DFH_TYPE_PRIVATE)
+		return id;
+	else if (type == DFH_TYPE_AFU)
+		return FEATURE_ID_AFU;
+
+	WARN_ON(1);
+	return 0;
+}
+
+/*
+ * when create sub feature instances, for private features, it doesn't need
+ * to provide resource size and feature id as they could be read from DFH
+ * register. For afu sub feature, its register region only contains user
+ * defined registers, so never trust any information from it, just use the
+ * resource size information provided by its parent FIU.
+ */
+static int
+create_feature_instance(struct build_feature_devs_info *binfo,
+			struct fpga_enum_dfl *dfl, resource_size_t ofst,
+			resource_size_t size, u64 fid)
+{
+	struct feature_info *finfo;
+
+	/* read feature size and id if inputs are invalid */
+	size = size ? size : feature_size(dfl->ioaddr + ofst);
+	fid = fid ? fid : feature_id(dfl->ioaddr + ofst);
+
+	if (dfl->len - ofst < size)
+		return -EINVAL;
+
+	finfo = kzalloc(sizeof(*finfo), GFP_KERNEL);
+	if (!finfo)
+		return -ENOMEM;
+
+	finfo->fid = fid;
+	finfo->mmio_res.start = dfl->start + ofst;
+	finfo->mmio_res.end = finfo->mmio_res.start + size - 1;
+	finfo->mmio_res.flags = IORESOURCE_MEM;
+	finfo->ioaddr = dfl->ioaddr + ofst;
+
+	list_add_tail(&finfo->node, &binfo->sub_features);
+	binfo->feature_num++;
+
+	return 0;
+}
+
+static int parse_feature_fme(struct build_feature_devs_info *binfo,
+			     struct fpga_enum_dfl *dfl, resource_size_t ofst)
+{
+	int ret;
+
+	ret = build_info_create_dev(binfo, FME_ID, FPGA_FEATURE_DEV_FME,
+				    dfl->ioaddr + ofst);
+	if (ret)
+		return ret;
+
+	return create_feature_instance(binfo, dfl, ofst, 0, 0);
+}
+
+static int parse_feature_port(struct build_feature_devs_info *binfo,
+			      struct fpga_enum_dfl *dfl,
+			      resource_size_t ofst)
+{
+	int ret;
+
+	ret = build_info_create_dev(binfo, PORT_ID, FPGA_FEATURE_DEV_PORT,
+				    dfl->ioaddr + ofst);
+	if (ret)
+		return ret;
+
+	return create_feature_instance(binfo, dfl, ofst, 0, 0);
+}
+
+static int parse_feature_port_afu(struct build_feature_devs_info *binfo,
+				  struct fpga_enum_dfl *dfl,
+				  resource_size_t ofst)
+{
+	u64 v = readq(binfo->ioaddr + PORT_HDR_CAP);
+	u32 size = FIELD_GET(PORT_CAP_MMIO_SIZE, v) << 10;
+
+	WARN_ON(!size);
+
+	return create_feature_instance(binfo, dfl, ofst, size, FEATURE_ID_AFU);
+}
+
+static int parse_feature_afu(struct build_feature_devs_info *binfo,
+			     struct fpga_enum_dfl *dfl,
+			     resource_size_t ofst)
+{
+	if (!binfo->feature_dev) {
+		dev_err(binfo->dev, "this AFU does not belong to any FIU.\n");
+		return -EINVAL;
+	}
+
+	switch (feature_dev_id_type(binfo->feature_dev)) {
+	case PORT_ID:
+		return parse_feature_port_afu(binfo, dfl, ofst);
+	default:
+		dev_info(binfo->dev, "AFU belonging to FIU %s is not supported yet.\n",
+			 binfo->feature_dev->name);
+	}
+
+	return 0;
+}
+
+static int parse_feature_fiu(struct build_feature_devs_info *binfo,
+			     struct fpga_enum_dfl *dfl,
+			     resource_size_t ofst)
+{
+	u32 id, offset;
+	u64 v;
+	int ret = 0;
+
+	v = readq(dfl->ioaddr + ofst + DFH);
+	id = FIELD_GET(DFH_ID, v);
+
+	switch (id) {
+	case DFH_ID_FIU_FME:
+		ret = parse_feature_fme(binfo, dfl, ofst);
+		break;
+	case DFH_ID_FIU_PORT:
+		ret = parse_feature_port(binfo, dfl, ofst);
+		break;
+	default:
+		dev_info(binfo->dev, "FIU TYPE %d is not supported yet.\n",
+			 id);
+	}
+
+	if (ret)
+		return ret;
+
+	/* Find and parse FIU's child AFU via its NEXT_AFU register */
+	v = readq(dfl->ioaddr + ofst + NEXT_AFU);
+
+	offset = FIELD_GET(NEXT_AFU_NEXT_DFH_OFST, v);
+	if (offset)
+		return parse_feature_afu(binfo, dfl, ofst + offset);
+
+	dev_dbg(binfo->dev, "No AFUs detected on FIU %d\n", id);
+
+	return ret;
+}
+
+static int parse_feature_private(struct build_feature_devs_info *binfo,
+				 struct fpga_enum_dfl *dfl,
+				 resource_size_t ofst)
+{
+	if (!binfo->feature_dev) {
+		dev_err(binfo->dev, "the private feature %llx does not belong to any AFU.\n",
+			(unsigned long long)feature_id(dfl->ioaddr + ofst));
+		return -EINVAL;
+	}
+
+	return create_feature_instance(binfo, dfl, ofst, 0, 0);
+}
+
+/**
+ * parse_feature - parse a feature on given device feature list
+ *
+ * @binfo: build feature devices information.
+ * @dfl: device feature list to parse
+ * @ofst: offset to feature header on this device feature list
+ */
+static int parse_feature(struct build_feature_devs_info *binfo,
+			 struct fpga_enum_dfl *dfl, resource_size_t ofst)
+{
+	u64 v;
+	u32 type;
+
+	v = readq(dfl->ioaddr + ofst + DFH);
+	type = FIELD_GET(DFH_TYPE, v);
+
+	switch (type) {
+	case DFH_TYPE_AFU:
+		return parse_feature_afu(binfo, dfl, ofst);
+	case DFH_TYPE_PRIVATE:
+		return parse_feature_private(binfo, dfl, ofst);
+	case DFH_TYPE_FIU:
+		return parse_feature_fiu(binfo, dfl, ofst);
+	default:
+		dev_info(binfo->dev,
+			 "Feature Type %x is not supported.\n", type);
+	}
+
+	return 0;
+}
+
+static int parse_feature_list(struct build_feature_devs_info *binfo,
+			      struct fpga_enum_dfl *dfl)
+{
+	void __iomem *start = dfl->ioaddr;
+	void __iomem *end = dfl->ioaddr + dfl->len;
+	int ret = 0;
+	u32 ofst = 0;
+	u64 v;
+
+	/* walk through the device feature list via DFH's next DFH pointer. */
+	for (; start < end; start += ofst) {
+		if (end - start < DFH_SIZE) {
+			dev_err(binfo->dev, "The region is too small to contain a feature.\n");
+			return -EINVAL;
+		}
+
+		ret = parse_feature(binfo, dfl, start - dfl->ioaddr);
+		if (ret)
+			return ret;
+
+		v = readq(start + DFH);
+		ofst = FIELD_GET(DFH_NEXT_HDR_OFST, v);
+
+		/* stop parsing if EOL(End of List) is set or offset is 0 */
+		if ((v & DFH_EOL) || !ofst)
+			break;
+	}
+
+	/* commit current feature device when reach the end of list */
+	return build_info_commit_dev(binfo);
+}
+
+struct fpga_enum_info *fpga_enum_info_alloc(struct device *dev)
+{
+	struct fpga_enum_info *info;
+
+	get_device(dev);
+
+	info = devm_kzalloc(dev, sizeof(*info), GFP_KERNEL);
+	if (!info) {
+		put_device(dev);
+		return NULL;
+	}
+
+	info->dev = dev;
+	INIT_LIST_HEAD(&info->dfls);
+
+	return info;
+}
+EXPORT_SYMBOL_GPL(fpga_enum_info_alloc);
+
+void fpga_enum_info_free(struct fpga_enum_info *info)
+{
+	struct fpga_enum_dfl *tmp, *dfl;
+	struct device *dev;
+
+	if (!info)
+		return;
+
+	dev = info->dev;
+
+	/* remove all device feature lists in the list. */
+	list_for_each_entry_safe(dfl, tmp, &info->dfls, node) {
+		list_del(&dfl->node);
+		devm_kfree(dev, dfl);
+	}
+
+	devm_kfree(dev, info);
+	put_device(dev);
+}
+EXPORT_SYMBOL_GPL(fpga_enum_info_free);
+
+/**
+ * fpga_enum_info_add_dfl - add info for a device feature list to fpga_enum_info
+ *
+ * @info: ptr to fpga_enum_info
+ * @start: mmio resource address of the device feature list.
+ * @len: mmio resource length of the device feature list.
+ * @ioaddr: mapped mmio resource address of the device feature list.
+ *
+ * One FPGA device may have 1 or more Device Feature Lists (DFLs), use this
+ * function to add information of each DFL to common data structure for next
+ * step enumeration.
+ *
+ * Return: 0 on success, negative error code otherwise.
+ */
+int fpga_enum_info_add_dfl(struct fpga_enum_info *info, resource_size_t start,
+			   resource_size_t len, void __iomem *ioaddr)
+{
+	struct fpga_enum_dfl *dfl;
+
+	dfl = devm_kzalloc(info->dev, sizeof(*dfl), GFP_KERNEL);
+	if (!dfl)
+		return -ENOMEM;
+
+	dfl->start = start;
+	dfl->len = len;
+	dfl->ioaddr = ioaddr;
+
+	list_add_tail(&dfl->node, &info->dfls);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(fpga_enum_info_add_dfl);
+
+static int remove_feature_dev(struct device *dev, void *data)
+{
+	struct platform_device *pdev = to_platform_device(dev);
+	enum fpga_id_type type = feature_dev_id_type(pdev);
+	int id = pdev->id;
+
+	platform_device_unregister(pdev);
+
+	free_fpga_id(type, id);
+
+	return 0;
+}
+
+static void remove_feature_devs(struct fpga_cdev *cdev)
+{
+	device_for_each_child(&cdev->region.dev, NULL, remove_feature_dev);
+}
+
+/**
+ * fpga_enumerate_feature_devs - enumerate feature devices
+ * @info: information for enumeration.
+ *
+ * This function creates a container device (base FPGA region), enumerates
+ * feature devices based on the enumeration info and creates platform devices
+ * under the container device.
+ *
+ * Return: fpga_cdev struct on success, -errno on failure
+ */
+struct fpga_cdev *fpga_enumerate_feature_devs(struct fpga_enum_info *info)
+{
+	struct build_feature_devs_info *binfo;
+	struct fpga_cdev *cdev;
+	struct fpga_enum_dfl *dfl;
+	int ret = 0;
+
+	if (!info->dev)
+		return ERR_PTR(-ENODEV);
+
+	cdev = devm_kzalloc(info->dev, sizeof(*cdev), GFP_KERNEL);
+	if (!cdev)
+		return ERR_PTR(-ENOMEM);
+
+	cdev->parent = info->dev;
+	mutex_init(&cdev->lock);
+	INIT_LIST_HEAD(&cdev->port_dev_list);
+	cdev->region.parent = info->dev;
+
+	ret = fpga_region_register(&cdev->region);
+	if (ret)
+		goto free_cdev_exit;
+
+	/* create and init build info for enumeration */
+	binfo = devm_kzalloc(info->dev, sizeof(*binfo), GFP_KERNEL);
+	if (!binfo) {
+		ret = -ENOMEM;
+		goto unregister_region_exit;
+	}
+
+	binfo->dev = info->dev;
+	binfo->cdev = cdev;
+
+	/*
+	 * start enumeration for all feature devices based on Device Feature
+	 * Lists.
+	 */
+	list_for_each_entry(dfl, &info->dfls, node) {
+		ret = parse_feature_list(binfo, dfl);
+		if (ret) {
+			remove_feature_devs(cdev);
+			build_info_free(binfo);
+			goto unregister_region_exit;
+		}
+	}
+
+	build_info_free(binfo);
+
+	return cdev;
+
+unregister_region_exit:
+	fpga_region_unregister(&cdev->region);
+free_cdev_exit:
+	devm_kfree(cdev->parent, cdev);
+	return ERR_PTR(ret);
+}
+EXPORT_SYMBOL_GPL(fpga_enumerate_feature_devs);
+
+/**
+ * fpga_remove_feature_devs - remove all feature devices
+ * @cdev: fpga container device.
+ *
+ * Remove the container device and all feature devices under given container
+ * devices.
+ */
+void fpga_remove_feature_devs(struct fpga_cdev *cdev)
+{
+	struct feature_platform_data *pdata, *ptmp;
+
+	remove_feature_devs(cdev);
+
+	mutex_lock(&cdev->lock);
+	if (cdev->fme_dev) {
+		/* the fme should be unregistered. */
+		WARN_ON(device_is_registered(cdev->fme_dev));
+		put_device(cdev->fme_dev);
+	}
+
+	list_for_each_entry_safe(pdata, ptmp, &cdev->port_dev_list, node) {
+		struct platform_device *port_dev = pdata->dev;
+
+		/* the port should be unregistered. */
+		WARN_ON(device_is_registered(&port_dev->dev));
+		list_del(&pdata->node);
+		put_device(&port_dev->dev);
+	}
+	mutex_unlock(&cdev->lock);
+
+	fpga_region_unregister(&cdev->region);
+	devm_kfree(cdev->parent, cdev);
+}
+EXPORT_SYMBOL_GPL(fpga_remove_feature_devs);
+
+int fpga_port_id(struct platform_device *pdev)
+{
+	void __iomem *base = get_feature_ioaddr_by_id(&pdev->dev,
+						      PORT_FEATURE_ID_HEADER);
+
+	return FIELD_GET(PORT_CAP_PORT_NUM, readq(base + PORT_HDR_CAP));
+}
+EXPORT_SYMBOL_GPL(fpga_port_id);
+
+/**
+ * __fpga_port_enable - enable a port
+ * @pdev: port platform device.
+ *
+ * Enable Port by clear the port soft reset bit, which is set by default.
+ * The User AFU is unable to respond to any MMIO access while in reset.
+ * __fpga_port_enable function should only be used after __fpga_port_disable
+ * function.
+ */
+void __fpga_port_enable(struct platform_device *pdev)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	void __iomem *base;
+	u64 v;
+
+	WARN_ON(!pdata->disable_count);
+
+	if (--pdata->disable_count != 0)
+		return;
+
+	base = get_feature_ioaddr_by_id(&pdev->dev, PORT_FEATURE_ID_HEADER);
+
+	/* Clear port soft reset */
+	v = readq(base + PORT_HDR_CTRL);
+	v &= ~PORT_CTRL_SFTRST;
+	writeq(v, base + PORT_HDR_CTRL);
+}
+EXPORT_SYMBOL_GPL(__fpga_port_enable);
+
+#define RST_POLL_INVL 10 /* us */
+#define RST_POLL_TIMEOUT 1000 /* us */
+
+/**
+ * __fpga_port_disable - disable a port
+ * @pdev: port platform device.
+ *
+ * Disable Port by setting the port soft reset bit, it puts the port into
+ * reset.
+ */
+int __fpga_port_disable(struct platform_device *pdev)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	void __iomem *base;
+	u64 v;
+
+	if (pdata->disable_count++ != 0)
+		return 0;
+
+	base = get_feature_ioaddr_by_id(&pdev->dev, PORT_FEATURE_ID_HEADER);
+
+	/* Set port soft reset */
+	v = readq(base + PORT_HDR_CTRL);
+	v |= PORT_CTRL_SFTRST;
+	writeq(v, base + PORT_HDR_CTRL);
+
+	/*
+	 * HW sets ack bit to 1 when all outstanding requests have been drained
+	 * on this port and minimum soft reset pulse width has elapsed.
+	 * Driver polls port_soft_reset_ack to determine if reset done by HW.
+	 */
+	if (readq_poll_timeout(base + PORT_HDR_CTRL, v, v & PORT_CTRL_SFTRST,
+			       RST_POLL_INVL, RST_POLL_TIMEOUT)) {
+		dev_err(&pdev->dev, "timeout, fail to reset device\n");
+		return -ETIMEDOUT;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(__fpga_port_disable);
+
+static int __init dfl_fpga_init(void)
+{
+	fpga_ids_init();
+
+	return 0;
+}
+
+static void __exit dfl_fpga_exit(void)
+{
+	fpga_ids_destroy();
+}
+
+module_init(dfl_fpga_init);
+module_exit(dfl_fpga_exit);
+
+MODULE_DESCRIPTION("FPGA Device Feature List (DFL) Support");
+MODULE_AUTHOR("Intel Corporation");
+MODULE_LICENSE("GPL v2");

diff --git a/drivers/fpga/dfl.h b/drivers/fpga/dfl.h
new file mode 100644
index 0000000..22dcf73
--- /dev/null
+++ b/drivers/fpga/dfl.h

@@ -0,0 +1,345 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Driver Header File for FPGA Device Feature List (DFL) Support
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Zhang Yi <yi.z.zhang@intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ */
+
+#ifndef __FPGA_DFL_H
+#define __FPGA_DFL_H
+
+#include <linux/bitfield.h>
+#include <linux/delay.h>
+#include <linux/fs.h>
+#include <linux/iopoll.h>
+#include <linux/io-64-nonatomic-lo-hi.h>
+#include <linux/platform_device.h>
+#include <linux/slab.h>
+#include <linux/uuid.h>
+#include <linux/fpga/fpga-region.h>
+
+/* maximum supported number of ports */
+#define MAX_FPGA_PORT_NUM 4
+/* plus one for fme device */
+#define MAX_FEATURE_DEV_NUM	(MAX_FPGA_PORT_NUM + 1)
+
+/* Reserved 0x0 for Header Group Register and 0xff for AFU */
+#define FEATURE_ID_FIU_HEADER		0x0
+#define FEATURE_ID_AFU			0xff
+
+#define FME_FEATURE_ID_HEADER		FEATURE_ID_FIU_HEADER
+#define FME_FEATURE_ID_THERMAL_MGMT	0x1
+#define FME_FEATURE_ID_POWER_MGMT	0x2
+#define FME_FEATURE_ID_GLOBAL_IPERF	0x3
+#define FME_FEATURE_ID_GLOBAL_ERR	0x4
+#define FME_FEATURE_ID_PR_MGMT		0x5
+#define FME_FEATURE_ID_HSSI		0x6
+#define FME_FEATURE_ID_GLOBAL_DPERF	0x7
+
+#define PORT_FEATURE_ID_HEADER		FEATURE_ID_FIU_HEADER
+#define PORT_FEATURE_ID_AFU		FEATURE_ID_AFU
+#define PORT_FEATURE_ID_ERROR		0x10
+#define PORT_FEATURE_ID_UMSG		0x11
+#define PORT_FEATURE_ID_UINT		0x12
+#define PORT_FEATURE_ID_STP		0x13
+
+/*
+ * Device Feature Header Register Set
+ *
+ * For FIUs, they all have DFH + GUID + NEXT_AFU as common header registers.
+ * For AFUs, they have DFH + GUID as common header registers.
+ * For private features, they only have DFH register as common header.
+ */
+#define DFH			0x0
+#define GUID_L			0x8
+#define GUID_H			0x10
+#define NEXT_AFU		0x18
+
+#define DFH_SIZE		0x8
+
+/* Device Feature Header Register Bitfield */
+#define DFH_ID			GENMASK_ULL(11, 0)	/* Feature ID */
+#define DFH_ID_FIU_FME		0
+#define DFH_ID_FIU_PORT		1
+#define DFH_REVISION		GENMASK_ULL(15, 12)	/* Feature revision */
+#define DFH_NEXT_HDR_OFST	GENMASK_ULL(39, 16)	/* Offset to next DFH */
+#define DFH_EOL			BIT(40)			/* End of list */
+#define DFH_TYPE		GENMASK_ULL(63, 60)	/* Feature type */
+#define DFH_TYPE_AFU		1
+#define DFH_TYPE_PRIVATE	3
+#define DFH_TYPE_FIU		4
+
+/* Next AFU Register Bitfield */
+#define NEXT_AFU_NEXT_DFH_OFST	GENMASK_ULL(23, 0)	/* Offset to next AFU */
+
+/* FME Header Register Set */
+#define FME_HDR_DFH		DFH
+#define FME_HDR_GUID_L		GUID_L
+#define FME_HDR_GUID_H		GUID_H
+#define FME_HDR_NEXT_AFU	NEXT_AFU
+#define FME_HDR_CAP		0x30
+#define FME_HDR_PORT_OFST(n)	(0x38 + ((n) * 0x8))
+#define FME_HDR_BITSTREAM_ID	0x60
+#define FME_HDR_BITSTREAM_MD	0x68
+
+/* FME Fab Capability Register Bitfield */
+#define FME_CAP_FABRIC_VERID	GENMASK_ULL(7, 0)	/* Fabric version ID */
+#define FME_CAP_SOCKET_ID	BIT(8)			/* Socket ID */
+#define FME_CAP_PCIE0_LINK_AVL	BIT(12)			/* PCIE0 Link */
+#define FME_CAP_PCIE1_LINK_AVL	BIT(13)			/* PCIE1 Link */
+#define FME_CAP_COHR_LINK_AVL	BIT(14)			/* Coherent Link */
+#define FME_CAP_IOMMU_AVL	BIT(16)			/* IOMMU available */
+#define FME_CAP_NUM_PORTS	GENMASK_ULL(19, 17)	/* Number of ports */
+#define FME_CAP_ADDR_WIDTH	GENMASK_ULL(29, 24)	/* Address bus width */
+#define FME_CAP_CACHE_SIZE	GENMASK_ULL(43, 32)	/* cache size in KB */
+#define FME_CAP_CACHE_ASSOC	GENMASK_ULL(47, 44)	/* Associativity */
+
+/* FME Port Offset Register Bitfield */
+/* Offset to port device feature header */
+#define FME_PORT_OFST_DFH_OFST	GENMASK_ULL(23, 0)
+/* PCI Bar ID for this port */
+#define FME_PORT_OFST_BAR_ID	GENMASK_ULL(34, 32)
+/* AFU MMIO access permission. 1 - VF, 0 - PF. */
+#define FME_PORT_OFST_ACC_CTRL	BIT(55)
+#define FME_PORT_OFST_ACC_PF	0
+#define FME_PORT_OFST_ACC_VF	1
+#define FME_PORT_OFST_IMP	BIT(60)
+
+/* PORT Header Register Set */
+#define PORT_HDR_DFH		DFH
+#define PORT_HDR_GUID_L		GUID_L
+#define PORT_HDR_GUID_H		GUID_H
+#define PORT_HDR_NEXT_AFU	NEXT_AFU
+#define PORT_HDR_CAP		0x30
+#define PORT_HDR_CTRL		0x38
+
+/* Port Capability Register Bitfield */
+#define PORT_CAP_PORT_NUM	GENMASK_ULL(1, 0)	/* ID of this port */
+#define PORT_CAP_MMIO_SIZE	GENMASK_ULL(23, 8)	/* MMIO size in KB */
+#define PORT_CAP_SUPP_INT_NUM	GENMASK_ULL(35, 32)	/* Interrupts num */
+
+/* Port Control Register Bitfield */
+#define PORT_CTRL_SFTRST	BIT(0)			/* Port soft reset */
+/* Latency tolerance reporting. '1' >= 40us, '0' < 40us.*/
+#define PORT_CTRL_LATENCY	BIT(2)
+#define PORT_CTRL_SFTRST_ACK	BIT(4)			/* HW ack for reset */
+
+/**
+ * struct feature - sub feature of the feature devices
+ *
+ * @id:	sub feature id.
+ * @resource_index: each sub feature has one mmio resource for its registers.
+ *		    this index is used to find its mmio resource from the
+ *		    feature dev (platform device)'s reources.
+ * @ioaddr: mapped mmio resource address.
+ */
+struct feature {
+	u64 id;
+	int resource_index;
+	void __iomem *ioaddr;
+};
+
+/**
+ * struct feature_platform_data - platform data for feature devices
+ *
+ * @node: node to link feature devs to container device's port_dev_list.
+ * @lock: mutex to protect platform data.
+ * @dev: ptr to platform device linked with this platform data.
+ * @disable_count: count for port disable.
+ * @num: number for sub features.
+ * @features: sub features of this feature dev.
+ */
+struct feature_platform_data {
+	struct list_head node;
+	struct mutex lock;
+	struct platform_device *dev;
+	unsigned int disable_count;
+
+	int num;
+	struct feature features[0];
+};
+
+#define FPGA_FEATURE_DEV_FME		"dfl-fme"
+#define FPGA_FEATURE_DEV_PORT		"dfl-port"
+
+static inline int feature_platform_data_size(const int num)
+{
+	return sizeof(struct feature_platform_data) +
+		num * sizeof(struct feature);
+}
+
+int fpga_port_id(struct platform_device *pdev);
+
+static inline int fpga_port_check_id(struct platform_device *pdev,
+				     void *pport_id)
+{
+	return fpga_port_id(pdev) == *(int *)pport_id;
+}
+
+void __fpga_port_enable(struct platform_device *pdev);
+int __fpga_port_disable(struct platform_device *pdev);
+
+static inline void fpga_port_enable(struct platform_device *pdev)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+
+	mutex_lock(&pdata->lock);
+	__fpga_port_enable(pdev);
+	mutex_unlock(&pdata->lock);
+}
+
+static inline int fpga_port_disable(struct platform_device *pdev)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	int ret;
+
+	mutex_lock(&pdata->lock);
+	ret = __fpga_port_disable(pdev);
+	mutex_unlock(&pdata->lock);
+
+	return ret;
+}
+
+/*
+ * This function resets the FPGA Port and its accelerator (AFU) by function
+ * __fpga_port_disable and __fpga_port_enable (set port soft reset bit and
+ * then clear it). Userspace can do Port reset at any time, e.g during DMA
+ * or Partial Reconfiguration. But it should never cause any system level
+ * issue, only functional failure (e.g DMA or PR operation failure) and be
+ * recoverable from the failure.
+ *
+ * Note: the accelerator (AFU) is not accessible when its port is in reset
+ * (disabled). Any attempts on MMIO access to AFU while in reset, will
+ * result errors reported via port error reporting sub feature (if present).
+ */
+static inline int __fpga_port_reset(struct platform_device *pdev)
+{
+	int ret;
+
+	ret = __fpga_port_disable(pdev);
+	if (ret)
+		return ret;
+
+	__fpga_port_enable(pdev);
+
+	return 0;
+}
+
+static inline int fpga_port_reset(struct platform_device *pdev)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	int ret;
+
+	mutex_lock(&pdata->lock);
+	ret = __fpga_port_reset(pdev);
+	mutex_unlock(&pdata->lock);
+
+	return ret;
+}
+
+#define fpga_dev_for_each_feature(pdata, feature)			    \
+	for ((feature) = (pdata)->features;				    \
+	   (feature) < (pdata)->features + (pdata)->num; (feature)++)
+
+static inline struct feature *get_feature_by_id(struct device *dev, u64 id)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(dev);
+	struct feature *feature;
+
+	fpga_dev_for_each_feature(pdata, feature)
+		if (feature->id == id)
+			return feature;
+
+	return NULL;
+}
+
+static inline void __iomem *get_feature_ioaddr_by_id(struct device *dev, u64 id)
+{
+	struct feature *feature = get_feature_by_id(dev, id);
+
+	if (feature && feature->ioaddr)
+		return feature->ioaddr;
+
+	WARN_ON(1);
+	return NULL;
+}
+
+static inline bool feature_is_fme(void __iomem *base)
+{
+	u64 v = readq(base + DFH);
+
+	return (FIELD_GET(DFH_TYPE, v) == DFH_TYPE_FIU) &&
+		(FIELD_GET(DFH_ID, v) == DFH_ID_FIU_FME);
+}
+
+static inline bool feature_is_port(void __iomem *base)
+{
+	u64 v = readq(base + DFH);
+
+	return (FIELD_GET(DFH_TYPE, v) == DFH_TYPE_FIU) &&
+		(FIELD_GET(DFH_ID, v) == DFH_ID_FIU_PORT);
+}
+
+/**
+ * struct fpga_enum_info - FPGA enumeration information
+ *
+ * @dev: parent device.
+ * @dfls: list of device feature lists.
+ */
+struct fpga_enum_info {
+	struct device *dev;
+	struct list_head dfls;
+};
+
+/**
+ * struct fpga_enum_dfl - FPGA enumeration device feature list information
+ *
+ * @start: base address of this device feature list.
+ * @len: size of this device feature list.
+ * @ioaddr: mapped base address of this device feature list.
+ * @node: node in list of device feature lists.
+ */
+struct fpga_enum_dfl {
+	resource_size_t start;
+	resource_size_t len;
+
+	void __iomem *ioaddr;
+
+	struct list_head node;
+};
+
+struct fpga_enum_info *fpga_enum_info_alloc(struct device *dev);
+int fpga_enum_info_add_dfl(struct fpga_enum_info *info, resource_size_t start,
+			   resource_size_t len, void __iomem *ioaddr);
+void fpga_enum_info_free(struct fpga_enum_info *info);
+
+/**
+ * struct fpga_cdev - fpga container device
+ *
+ * @parent: parent device of this container device.
+ * @region: base fpga region.
+ * @fme_dev: FME feature device under this container device.
+ * @lock: mutex lock to protect the port device list.
+ * @port_dev_list: list of all port feature devices under this container device.
+ */
+struct fpga_cdev {
+	struct device *parent;
+
+	struct fpga_region region;
+
+	struct device *fme_dev;
+
+	struct mutex lock; /* to protect the port device list */
+	struct list_head port_dev_list;
+};
+
+struct fpga_cdev *fpga_enumerate_feature_devs(struct fpga_enum_info *info);
+void fpga_remove_feature_devs(struct fpga_cdev *cdev);
+
+#endif /* __FPGA_DFL_H */

-- 
2.7.4

[PATCH v4 01/24] docs: fpga: add a document for FPGA Device Feature List (DFL) Framework Overview

From: Wu Hao <hidden>
Date: 2018-02-13 09:56:09

Add a document for FPGA Device Feature List (DFL) Framework Overview.

Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Xiao Guangrong <redacted>
Signed-off-by: Wu Hao <redacted>
---
v2: added FME fpga-mgr/bridge/region platform driver to driver organization.
    updated open discussion per current implementation.
    fixed some typos.
v3: use FPGA base region as container device instead of fpga-dev class.
    split common enumeration code from pcie driver to functions exposed by
    device feature list framework.
    update FME performance reporting which supports both integrated (iperf/)
    and discrete (dperf/) FPGA solutions.
v4: rename this doc to Device Feature List (DFL) Framework Overview (dfl.txt)
    add Device Feature List introduction and re-organize the content.
    add description for port reset, bitstream_id/metadata and etc.
---
 Documentation/fpga/dfl.txt | 382 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 382 insertions(+)
 create mode 100644 Documentation/fpga/dfl.txt

diff --git a/Documentation/fpga/dfl.txt b/Documentation/fpga/dfl.txt
new file mode 100644
index 0000000..28e4e22
--- /dev/null
+++ b/Documentation/fpga/dfl.txt

@@ -0,0 +1,382 @@
+===============================================================================
+              FPGA Device Feature List (DFL) Framework Overview
+-------------------------------------------------------------------------------
+                Enno Luebbers <enno.luebbers@intel.com>
+                Xiao Guangrong <guangrong.xiao@linux.intel.com>
+                Wu Hao <hao.wu@intel.com>
+
+The Device Feature List (DFL) FPGA framework (and drivers according to this
+this framework) hides the very details of low layer hardwares and provides
+unified interfaces for userspace applications to configure, enumerate, open
+and access FPGA accelerators on platforms implemented the DFL in the device
+memory, and enables system level management functions such as FPGA
+reconfiguration, power management and virtualization.
+
+Device Feature List (DFL) Overview
+==================================
+Device Feature List (DFL) defines a link list of feature headers within the
+device MMIO space to provide an extensible way of adding features. Software can
+walk through these predefined data structures to enumerate FPGA features:
+FPGA Interface Unit (FIU), Accelerated Function Unit (AFU) and Private Features,
+as illustrated below:
+
+    Header            Header            Header            Header
+ +----------+  +-->+----------+  +-->+----------+  +-->+----------+
+ |   Type   |  |   |  Type    |  |   |  Type    |  |   |  Type    |
+ |   FIU    |  |   | Private  |  |   | Private  |  |   | Private  |
+ +----------+  |   | Feature  |  |   | Feature  |  |   | Feature  |
+ | Next_DFH |--+   +----------+  |   +----------+  |   +----------+
+ +----------+      | Next_DFH |--+   | Next_DFH |--+   | Next_DFH |--> NULL
+ |    ID    |      +----------+      +----------+      +----------+
+ +----------+      |    ID    |      |    ID    |      |    ID    |
+ | Next_AFU |--+   +----------+      +----------+      +----------+
+ +----------+  |   | Feature  |      | Feature  |      | Feature  |
+ | Feature  |  |   | Region   |      | Region   |      | Region   |
+ | Region   |  |   +----------+      +----------+      +----------+
+ +----------+  |      Header
+               +-->+----------+
+                   |   Type   |
+                   |   AFU    |
+                   +----------+
+                   |   GUID   |
+                   +----------+
+                   |  Feature |
+                   |  Region  |
+                   +----------+
+
+FPGA Interface Unit (FIU) represents a standalone functional unit for the
+interface to FPGA, e.g the FPGA Management Engine (FME) and Port. (more
+descriptions on FME and Port in later sections). Accelerated Function Unit (AFU)
+represents a FPGA programmable region, and usually connects to a FIU as its
+child as illustrated above. Private Features are sub features of the FIUs/AFUs,
+they could be various function blocks with different ID, but they are linked to
+one list via the Next Device Feature Header (DFH) pointer as children. The
+registers for actual functions are grouped as feature regions which always
+follow the common header registers. The feature region located after the FIU
+header, is named as header register set for given FIU type. e.g FME Header
+Register Set.
+
+This Device Feature List provides a way of linking features together, it's
+convenient for software to locate each feature by walking through this list,
+and can be implemented in register regions of any FPGA device.
+
+FIU - FME (FPGA Management Engine)
+==================================
+The FPGA Management Engine performs power and thermal management, error
+reporting, reconfiguration, performance reporting for integrated and discrete
+solution, and other infrastructure functions. Each FPGA device only has one FME.
+
+User-space applications can acquire exclusive access to the FME using open(),
+and release it using close().
+
+The following functions are exposed through ioctls:
+
+ Get driver API version (FPGA_GET_API_VERSION)
+ Check for extensions (FPGA_CHECK_EXTENSION)
+ Assign port to PF (*FPGA_FME_PORT_ASSIGN)
+ Release port from PF (*FPGA_FME_PORT_RELEASE)
+ Program bitstream (FPGA_FME_PORT_PR)
+
+*FPGA_FME_PORT_ASSIGN/RELEASE are only used for FPGA virtualization. Please
+refer to later section "FPGA Virtualization - PCIe" for more details.
+
+More functions are exposed through sysfs
+(/sys/class/fpga_region/regionX/dfl-fme.n/):
+
+ Read bitstream ID (bitstream_id)
+     bitstream_id indicates version of the blue bitstream (static FPGA region).
+
+ Read bitstream metadata (bitstream_metadata)
+     bistream_metadata includes more detailed information of the blue bitstream
+     (static FPGA region), e.g synthesis date and seed.
+
+ Read number of ports (ports_num)
+     one FPGA device may have more than 1 port, this sysfs interface indicates
+     how many ports the FPGA device has.
+
+ Read socket ID (socket_id)
+     socket_id is only used in integrated solution, to indicate which socket
+     the FPGA device belongs to.
+
+ Read performance counters (iperf/ and dperf/)
+ Power management (power_mgmt/)
+ Thermal management (thermal_mgmt/)
+ Error reporting (errors/)
+
+FIU - PORT
+==========
+A port represents the interface between the static FPGA fabric (the "blue
+bitstream") and a partially reconfigurable region containing an AFU (the "green
+bitstream"). It controls the communication from SW to the accelerator and
+exposes features such as reset and debug. Each FPGA device may have more than
+1 port.
+
+AFU
+===
+An AFU is attached to a port FIU and exposes a fixed length MMIO region to be
+used for accelerator-specific control registers.
+
+User-space applications can acquire exclusive access to an AFU attached to a
+port by using open() on the port device node, and release it using close().
+
+The following functions are exposed through ioctls:
+
+ Get driver API version (FPGA_GET_API_VERSION)
+ Check for extensions (FPGA_CHECK_EXTENSION)
+ Get port info (FPGA_PORT_GET_INFO)
+ Get MMIO region info (FPGA_PORT_GET_REGION_INFO)
+ Map DMA buffer (FPGA_PORT_DMA_MAP)
+ Unmap DMA buffer (FPGA_PORT_DMA_UNMAP)
+ Reset AFU (*FPGA_PORT_RESET)
+ Enable UMsg (FPGA_PORT_UMSG_ENABLE)
+ Disable UMsg (FPGA_PORT_UMSG_DISABLE)
+ Set UMsg mode (FPGA_PORT_UMSG_SET_MODE)
+ Set UMsg base address (FPGA_PORT_UMSG_SET_BASE_ADDR)
+
+*FPGA_PORT_RESET: reset the FPGA Port and its AFU. Userspace can do Port reset
+at any time, e.g during DMA or Partial Reconfiguration. But it should never
+cause any system level issue, only functional failure (e.g DMA or PR operation
+failure) and be recoverable from the failure.
+
+User-space applications can also mmap() accelerator MMIO regions.
+
+More functions are exposed through sysfs:
+(/sys/class/fpga_region/<regionX>/<dfl-port.m>/):
+
+ Read Accelerator GUID (afu_id)
+ Error reporting (errors/)
+
+DFL Framework Overview
+======================
+
+         +----------+    +--------+ +--------+ +--------+
+         |   FME    |    |  AFU   | |  AFU   | |  AFU   |
+         |  Module  |    | Module | | Module | | Module |
+         +----------+    +--------+ +--------+ +--------+
+                 +-----------------------+
+                 | FPGA Container Device |    Device Feature List
+                 |  (FPGA Base Region)   |         Framework
+                 +-----------------------+
+--------------------------------------------------------------------
+               +----------------------------+
+               |   FPGA Bus Device Module   |
+               | (e.g PCIE/Platform Device) |
+               +----------------------------+
+                 +------------------------+
+                 |  FPGA Hardware Device  |
+                 +------------------------+
+
+DFL Framework in kernel provides common interfaces to create container device
+(FPGA base region), discover feature devices and their private features from the
+given Device Feature Lists, and create platform devices for feature devices
+(e.g FME, Port and AFU) with related resources under the container device. It
+also abstracts operations for the private features and exposes common ops to
+feature device drivers.
+
+The FPGA Bus Device could be different devices e.g PCIe device, platform device
+and etc. Its driver is always loaded first once the device is detected on its
+own bus. This driver plays an infrastructural role in the driver architecture.
+It locates the DFLs in the device memory, handles them and related resources
+to common interfaces from DFL framework for enumeration. (Please refer to
+drivers/fpga/dfl.c for detailed enumeration APIs).
+
+The FPGA Management Engine (FME) driver is a platform driver which is loaded
+automatically after FME platform device creation from the PCIE driver. It
+provides the key features for FPGA management, including:
+
+	a) Power and thermal management, error reporting, performance reporting
+	   and other infrastructure functions. Users can access these functions
+	   via sysfs interfaces exposed by FME driver.
+	b) Partial Reconfiguration. The FME driver creates FPGA manager, FPGA
+	   bridges and FPGA regions during PR sub feature initialization; Once
+	   it receives an FPGA_FME_PORT_PR ioctl from user, it invokes the
+	   common interface function from FPGA Region to complete the partial
+	   reconfiguration of the bitstream to the given port.
+	c) Port management for virtualization. The FME driver introduces two
+	   ioctls, FPGA_FME_PORT_RELEASE (releases given port from PF) and
+	   FPGA_FME_PORT_ASSIGN (assigns the port back to PF). Once the port is
+	   released from the PF, it can be assigned to the VF through the SRIOV
+	   interfaces provided by PCIE driver. (Refer to "FPGA virtualization"
+	   for more details).
+
+Similar to the the FME driver, the FPGA Accelerated Function Unit (AFU) driver
+is probed once the AFU platform device is created. The main function of this
+module is to provide an interface for userspace applications to access the
+individual accelerators, including basic reset control on port, AFU MMIO region
+export, dma buffer mapping service, UMsg notification, and remote debug
+functions (see above).
+
+After feature platform devices creation, matched platform drivers will be loaded
+automatically to handle different functionalities. Please refer to next sections
+for detailed information on functional units which has been already implemented
+under this DFL framework.
+
+Partial Reconfiguration
+=======================
+As mentioned above, accelerators can be reconfigured through partial
+reconfiguration of a green bitstream file (GBS). The green bitstream must have
+been generated for the exact blue bitstream and targeted reconfigurable region
+(port) of the FPGA; otherwise, the reconfiguration operation will fail and
+possibly cause system instability. This compatibility can be checked by
+comparing the interface ID noted in the GBS header against the interface ID
+exposed by the FME through sysfs (see above). This check is usually done by
+user-space before calling the reconfiguration IOCTL.
+
+FPGA virtualization - PCIe
+==========================
+This section describes the virtualization support on DFL based FPGA device to
+enable accessing an accelerator from applications running in a virtual machine
+(VM). This section only describes the PCIe based FPGA device with SRIOV support.
+
+Features supported by the particular FPGA device are exposed through Device
+Feature Lists, as illustrated below:
+
+  +-------------------------------+  +-------------+
+  |              PF               |  |     VF      |
+  +-------------------------------+  +-------------+
+      ^            ^         ^              ^
+      |            |         |              |
++-----|------------|---------|--------------|-------+
+|     |            |         |              |       |
+|  +-----+     +-------+ +-------+      +-------+   |
+|  | FME |     | Port0 | | Port1 |      | Port2 |   |
+|  +-----+     +-------+ +-------+      +-------+   |
+|                  ^         ^              ^       |
+|                  |         |              |       |
+|              +-------+ +------+       +-------+   |
+|              |  AFU  | |  AFU |       |  AFU  |   |
+|              +-------+ +------+       +-------+   |
+|                                                   |
+|                 FPGA PCIe Device                  |
++---------------------------------------------------+
+
+FME is always accessed through the physical function (PF).
+
+Ports (and related AFUs) are accessed via PF by default, but could be exposed
+through virtual function (VF) devices via PCIe SRIOV. Each VF only contains
+1 Port and 1 AFU for isolation. Users could assign individual VFs (accelerators)
+created via PCIe SRIOV interface, to virtual machines.
+
+The driver organization in virtualization case is illustrated below:
+
+  +-------++------++------+             |
+  | FME   || FME  || FME  |             |
+  | FPGA  || FPGA || FPGA |             |
+  |Manager||Bridge||Region|             |
+  +-------++------++------+             |
+  +-----------------------+  +--------+ |             +--------+
+  |          FME          |  |  AFU   | |             |  AFU   |
+  |         Module        |  | Module | |             | Module |
+  +-----------------------+  +--------+ |             +--------+
+        +-----------------------+       |       +-----------------------+
+        | FPGA Container Device |       |       | FPGA Container Device |
+        |  (FPGA Base Region)   |       |       |  (FPGA Base Region)   |
+        +-----------------------+       |       +-----------------------+
+          +------------------+          |         +------------------+
+          | FPGA PCIE Module |          | Virtual | FPGA PCIE Module |
+          +------------------+   Host   | Machine +------------------+
+ -------------------------------------- | ------------------------------
+           +---------------+            |          +---------------+
+           | PCI PF Device |            |          | PCI VF Device |
+           +---------------+            |          +---------------+
+
+FPGA PCIe device driver is always loaded first once a FPGA PCIe PF or VF device
+is detected. It:
+
+	a) finish enumeration on both FPGA PCIe PF and VF device using common
+	   interfaces from DFL framework.
+	b) supports SRIOV.
+
+The FME device driver plays a management role in this driver architecture, it
+provides ioctls to release Port from PF and assign Port to PF. After release
+a port from PF, then it's safe to expose this port through a VF via PCIe SRIOV
+sysfs interface.
+
+To enable accessing an accelerator from applications running in a VM, the
+respective AFU's port needs to be assigned to a VF using the following steps:
+
+	a) The PF owns all AFU ports by default. Any port that needs to be
+	   reassigned to a VF must first be released through the
+	   FPGA_FME_PORT_RELEASE ioctl on the FME device.
+
+	b) Once N ports are released from PF, then user can use command below
+	   to enable SRIOV and VFs. Each VF owns only one Port with AFU.
+
+	   echo N > $PCI_DEVICE_PATH/sriov_numvfs
+
+	c) Pass through the VFs to VMs
+
+	d) The AFU under VF is accessible from applications in VM (using the
+	   same driver inside the VF).
+
+Note that an FME can't be assigned to a VF, thus PR and other management
+functions are only available via the PF.
+
+Device enumeration
+==================
+This section introduces how applications enumerate the fpga device from
+the sysfs hierarchy under /sys/class/fpga_region.
+
+In the example below, two DFL based FPGA devices are installed in the host. Each
+fpga device has one FME and two ports (AFUs).
+
+FPGA regions are created under /sys/class/fpga_region/
+
+	/sys/class/fpga_region/region0
+	/sys/class/fpga_region/region1
+	/sys/class/fpga_region/region2
+	...
+
+Application needs to search each regionX folder, if feature device is found,
+(e.g "dfl-port.n" or "dfl-fme.m" is found), then it's the base
+fpga region which represents the FPGA device.
+
+Each base region has one FME and two ports (AFUs) as child devices:
+
+	/sys/class/fpga_region/region0/dfl-fme.0
+	/sys/class/fpga_region/region0/dfl-port.0
+	/sys/class/fpga_region/region0/dfl-port.1
+	...
+
+	/sys/class/fpga_region/region3/dfl-fme.1
+	/sys/class/fpga_region/region3/dfl-port.2
+	/sys/class/fpga_region/region3/dfl-port.3
+	...
+
+In general, the FME/AFU sysfs interfaces are named as follows:
+
+	/sys/class/fpga_region/<regionX>/<dfl-fme.n>/
+	/sys/class/fpga_region/<regionX>/<dfl-port.m>/
+
+with 'n' consecutively numbering all FMEs and 'm' consecutively numbering all
+ports.
+
+The device nodes used for ioctl() or mmap() can be referenced through:
+
+	/sys/class/fpga_region/<regionX>/<dfl-fme.n>/dev
+	/sys/class/fpga_region/<regionX>/<dfl-port.n>/dev
+
+Add new FIUs support
+====================
+It's possible that developers made some new function blocks (FIUs) under this
+DFL framework, then new platform device driver needs to be developed for the
+new feature dev (FIU) following the same way as existing feature dev drivers
+(e.g FME and Port/AFU platform device driver). Besides that, it requires
+modification on DFL framework enumeration code too, for new FIU type detection
+and related platform devices creation.
+
+Add new private features support
+================================
+In some cases, we may need to add some new private features to existing FIUs
+(e.g FME or Port). Developers don't need to touch enumeration code in DFL
+framework, as each private feature will be parsed automatically, and related
+mmio resources can be found under FIU platform device created by DFL framework.
+Developer only needs to provide a sub feature driver with matched feature id.
+FME Partial Reconfiguration Sub Feature driver (see drivers/fpga/dfl-fme-pr.c)
+could be a reference.
+
+Open discussion
+===============
+FME driver exports one ioctl (FPGA_FME_PORT_PR) for partial reconfiguration to
+user now. In the future, if unified user interfaces for reconfiguration are
+added, FME driver should switch to them from ioctl interface.

-- 
2.7.4

Re: [PATCH v4 03/24] fpga: mgr: add status for fpga-manager

From: Alan Tull <atull@kernel.org>
Date: 2018-02-14 15:56:11

On Tue, Feb 13, 2018 at 3:24 AM, Wu Hao [off-list ref] wrote:

Hi Hao,

This patch adds status to fpga-manager data structure,

Please update the header to better describe the current changes, this
patch no longer adds status to the fpga mgr data struct.

to allow
driver to store full/partial reconfiguration errors and other
status information, and adds one status callback to fpga_manager_ops
to allow fpga_manager to collect latest status when failures are
detected.

Other than that, looks good to me!  Thanks for taking my feedback.
Alan

The following sysfs file is created:
* /sys/class/fpga_manager/<fpga>/status
  Return status of fpga manager, including reconfiguration errors.

Signed-off-by: Wu Hao <redacted>

Acked-by: Alan Tull <atull@kernel.org>

Re: [PATCH v4 06/24] fpga: dfl: adds fpga_cdev_find_port

From: Alan Tull <atull@kernel.org>
Date: 2018-02-14 16:25:31

On Tue, Feb 13, 2018 at 3:24 AM, Wu Hao [off-list ref] wrote:

Hi Hao,

For feature devices, we need a method to find the port dedicated
to the device. This patch adds a function fpga_cdev_find_port
for this purpose. e.g FPGA Management Engine (FME) Partial
Reconfiguration sub feature, it uses this function to find
dedicated port on the device for PR function implementation.

OK, that is very clear now, thanks!  Ack with the minor doc changes below.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
Signed-off-by: Wu Hao <redacted>

Acked-by: Alan Tull <redacted>

quoted hunk

---
v3: s/fpga_for_each_port/fpga_cdev_find_port/
    move fpga_cdev_find_port to fpga-dfl module.
v4: improve description in commit message.
    add comments to remind user to put_device after use this function.
---
 drivers/fpga/dfl.c | 34 ++++++++++++++++++++++++++++++++++
 drivers/fpga/dfl.h | 20 ++++++++++++++++++++
 2 files changed, 54 insertions(+)

diff --git a/drivers/fpga/dfl.c b/drivers/fpga/dfl.c
index dcfe5ab..38dc819 100644
--- a/drivers/fpga/dfl.c
+++ b/drivers/fpga/dfl.c

@@ -783,6 +783,40 @@ void fpga_remove_feature_devs(struct fpga_cdev *cdev)
 }
 EXPORT_SYMBOL_GPL(fpga_remove_feature_devs);

+/**
+ * __fpga_cdev_find_port - find a port under given container device
+ *
+ * @cdev: container device
+ * @data: data passed to match function
+ * @match: match function used to find specific port from the port device list
+ *
+ * Find a port device under container device. This function needs to be
+ * invoked with lock held.
+ *
+ * This function returns NULL if the device doesn't match and non-zero if it
+ * does.

Good but this would be more brief and more specific and more in line
with recommended format in Documentation/kernel-doc-nano-HOWTO.txt:

 * Return: pointer to port's platform device if successful, NULL otherwise.

If this function returns non-zero and a reference to the platform
+ * device of port can be obtained.

This if..then type sentence is not complete (missing the 'then' part).
I see where you might have gotten inspiration for it from core.c.
Probably you could just delete the sentence and all would be clear.

+ *
+ * NOTE: you will need to drop the device reference with put_device() after use.
+ */

Good

Thanks,
Alan

quoted hunk

+struct platform_device *
+__fpga_cdev_find_port(struct fpga_cdev *cdev, void *data,
+                     int (*match)(struct platform_device *, void *))
+{
+       struct feature_platform_data *pdata;
+       struct platform_device *port_dev;
+
+       list_for_each_entry(pdata, &cdev->port_dev_list, node) {
+               port_dev = pdata->dev;
+
+               if (match(port_dev, data) && get_device(&port_dev->dev))
+                       return port_dev;
+       }
+
+       return NULL;
+}
+EXPORT_SYMBOL_GPL(__fpga_cdev_find_port);
+
 int fpga_port_id(struct platform_device *pdev)
 {
        void __iomem *base = get_feature_ioaddr_by_id(&pdev->dev,

diff --git a/drivers/fpga/dfl.h b/drivers/fpga/dfl.h
index 9b19399..d6ecda1 100644
--- a/drivers/fpga/dfl.h
+++ b/drivers/fpga/dfl.h

@@ -356,4 +356,24 @@ struct fpga_cdev {
 struct fpga_cdev *fpga_enumerate_feature_devs(struct fpga_enum_info *info);
 void fpga_remove_feature_devs(struct fpga_cdev *cdev);

+/*
+ * need to drop the device reference with put_device() after use port platform
+ * device returned by __fpga_cdev_find_port and fpga_cdev_find_port functions.
+ */
+struct platform_device *
+__fpga_cdev_find_port(struct fpga_cdev *cdev, void *data,
+                     int (*match)(struct platform_device *, void *));
+
+static inline struct platform_device *
+fpga_cdev_find_port(struct fpga_cdev *cdev, void *data,
+                   int (*match)(struct platform_device *, void *))
+{
+       struct platform_device *pdev;
+
+       mutex_lock(&cdev->lock);
+       pdev = __fpga_cdev_find_port(cdev, data, match);
+       mutex_unlock(&cdev->lock);
+
+       return pdev;
+}
 #endif /* __FPGA_DFL_H */
--

2.7.4

Re: [PATCH v4 11/24] fpga: dfl: fme: add header sub feature support

From: Alan Tull <atull@kernel.org>
Date: 2018-02-14 16:37:09

On Tue, Feb 13, 2018 at 3:24 AM, Wu Hao [off-list ref] wrote:

Hi Hao,

Looks good.

From: Kang Luwei <redacted>

The Header Register set is always present for FPGA Management Engine (FME),
this patch implements init and uinit function for header sub feature and
introduce several read-only sysfs interfaces for the capability and status.

Sysfs interfaces:
* /sys/class/fpga_region/<regionX>/<dfl-fme.x>/ports_num
  Read-only. Number of ports implemented

* /sys/class/fpga_region/<regionX>/<dfl-fme.x>/bitstream_id
  Read-only. Blue Bitstream (static FPGA region) identifier number. It contains
  the detailed version and other information of this static FPGA region.

* /sys/class/fpga_region/<regionX>/<dfl-fme.x>/bitstream_metadata
  Read-only. Blue Bitstream (static FPGA region) meta data. It contains the
  synthesis date, seed and other information of this static FPGA region.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Kang Luwei <redacted>
Signed-off-by: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
Signed-off-by: Wu Hao <redacted>

Acked-by: Alan Tull <redacted>

Thanks,
Alan

Re: [PATCH v4 06/24] fpga: dfl: adds fpga_cdev_find_port

From: Moritz Fischer <mdf@kernel.org>
Date: 2018-02-14 21:00:05

Hi Hao,

On Tue, Feb 13, 2018 at 05:24:35PM +0800, Wu Hao wrote:

For feature devices, we need a method to find the port dedicated
to the device. This patch adds a function fpga_cdev_find_port
for this purpose. e.g FPGA Management Engine (FME) Partial
Reconfiguration sub feature, it uses this function to find
dedicated port on the device for PR function implementation.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
Signed-off-by: Wu Hao <redacted>

Acked-by: Moritz Fischer <redacted>

quoted hunk

---
v3: s/fpga_for_each_port/fpga_cdev_find_port/
    move fpga_cdev_find_port to fpga-dfl module.
v4: improve description in commit message.
    add comments to remind user to put_device after use this function.
---
 drivers/fpga/dfl.c | 34 ++++++++++++++++++++++++++++++++++
 drivers/fpga/dfl.h | 20 ++++++++++++++++++++
 2 files changed, 54 insertions(+)

diff --git a/drivers/fpga/dfl.c b/drivers/fpga/dfl.c
index dcfe5ab..38dc819 100644
--- a/drivers/fpga/dfl.c
+++ b/drivers/fpga/dfl.c

@@ -783,6 +783,40 @@ void fpga_remove_feature_devs(struct fpga_cdev *cdev)
 }
 EXPORT_SYMBOL_GPL(fpga_remove_feature_devs);
 
+/**
+ * __fpga_cdev_find_port - find a port under given container device
+ *
+ * @cdev: container device
+ * @data: data passed to match function
+ * @match: match function used to find specific port from the port device list
+ *
+ * Find a port device under container device. This function needs to be
+ * invoked with lock held.
+ *
+ * This function returns NULL if the device doesn't match and non-zero if it
+ * does. If this function returns non-zero and a reference to the platform
+ * device of port can be obtained.
+ *
+ * NOTE: you will need to drop the device reference with put_device() after use.
+ */
+struct platform_device *
+__fpga_cdev_find_port(struct fpga_cdev *cdev, void *data,
+		      int (*match)(struct platform_device *, void *))
+{
+	struct feature_platform_data *pdata;
+	struct platform_device *port_dev;
+
+	list_for_each_entry(pdata, &cdev->port_dev_list, node) {
+		port_dev = pdata->dev;
+
+		if (match(port_dev, data) && get_device(&port_dev->dev))
+			return port_dev;
+	}
+
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(__fpga_cdev_find_port);
+
 int fpga_port_id(struct platform_device *pdev)
 {
 	void __iomem *base = get_feature_ioaddr_by_id(&pdev->dev,

diff --git a/drivers/fpga/dfl.h b/drivers/fpga/dfl.h
index 9b19399..d6ecda1 100644
--- a/drivers/fpga/dfl.h
+++ b/drivers/fpga/dfl.h

@@ -356,4 +356,24 @@ struct fpga_cdev {
 struct fpga_cdev *fpga_enumerate_feature_devs(struct fpga_enum_info *info);
 void fpga_remove_feature_devs(struct fpga_cdev *cdev);
 
+/*
+ * need to drop the device reference with put_device() after use port platform
+ * device returned by __fpga_cdev_find_port and fpga_cdev_find_port functions.
+ */
+struct platform_device *
+__fpga_cdev_find_port(struct fpga_cdev *cdev, void *data,
+		      int (*match)(struct platform_device *, void *));
+
+static inline struct platform_device *
+fpga_cdev_find_port(struct fpga_cdev *cdev, void *data,
+		    int (*match)(struct platform_device *, void *))
+{
+	struct platform_device *pdev;
+
+	mutex_lock(&cdev->lock);
+	pdev = __fpga_cdev_find_port(cdev, data, match);
+	mutex_unlock(&cdev->lock);
+
+	return pdev;
+}
 #endif /* __FPGA_DFL_H */

-- 
2.7.4

Thanks,
Moritz

Re: [PATCH v4 03/24] fpga: mgr: add status for fpga-manager

From: Moritz Fischer <mdf@kernel.org>
Date: 2018-02-14 21:00:49

Hi Hao,

with Alan's Feedback

On Tue, Feb 13, 2018 at 05:24:32PM +0800, Wu Hao wrote:

This patch adds status to fpga-manager data structure, to allow
driver to store full/partial reconfiguration errors and other
status information, and adds one status callback to fpga_manager_ops
to allow fpga_manager to collect latest status when failures are
detected.

The following sysfs file is created:
* /sys/class/fpga_manager/<fpga>/status
  Return status of fpga manager, including reconfiguration errors.

Signed-off-by: Wu Hao <redacted>

Acked-by: Moritz Fischer <mdf@kernel.org>

quoted hunk

---
v3: add one line description for status
    add status callback function to fpga_manager_ops
    update fpga-mgr status if any failure or during initialization
    s/INCOMPATIBLE_BS_ERR/INCOMPATIBLE_IMAGE_ERR/
v4: simply code per suggestion from Alan.
    add supported status strings (and descriptions) in sysfs document.
    remove unused error code (SECURE_LOAD_ERR).
---
 Documentation/ABI/testing/sysfs-class-fpga-manager | 24 +++++++++++++++++++
 drivers/fpga/fpga-mgr.c                            | 28 ++++++++++++++++++++++
 include/linux/fpga/fpga-mgr.h                      |  9 +++++++
 3 files changed, 61 insertions(+)

diff --git a/Documentation/ABI/testing/sysfs-class-fpga-manager b/Documentation/ABI/testing/sysfs-class-fpga-manager
index 23056c5..60db277 100644
--- a/Documentation/ABI/testing/sysfs-class-fpga-manager
+++ b/Documentation/ABI/testing/sysfs-class-fpga-manager

@@ -35,3 +35,27 @@ Description:	Read fpga manager state as a string.
 		* write complete	= Doing post programming steps
 		* write complete error	= Error while doing post programming
 		* operating		= FPGA is programmed and operating
+
+What:		/sys/class/fpga_manager/<fpga>/status
+Date:		February 2018
+KernelVersion:	4.16
+Contact:	Wu Hao <hao.wu@intel.com>
+Description:	Read fpga manager status as a string.
+		If FPGA programming operation fails, it could be caused by crc
+		error or incompatible bitstream image. The intent of this
+		interface is to provide more detailed information for FPGA
+		programming errors to userspace. This is a list of strings for
+		the supported status.
+
+		* reconfig operation error 	- invalid operations detected by
+						  reconfiguration hardware.
+						  e.g start reconfiguration
+						  with errors not cleared
+		* reconfig CRC error		- CRC error detected by
+						  reconfiguration hardware.
+		* reconfig incompatible image	- reconfiguration image is
+						  incompatible with hardware
+		* reconfig IP protocol error	- protocol errors detected by
+						  reconfiguration hardware
+		* reconfig fifo overflow error	- FIFO overflow detected by
+						  reconfiguration hardware

diff --git a/drivers/fpga/fpga-mgr.c b/drivers/fpga/fpga-mgr.c
index 245e7a6..50b7396 100644
--- a/drivers/fpga/fpga-mgr.c
+++ b/drivers/fpga/fpga-mgr.c

@@ -397,12 +397,40 @@ static ssize_t state_show(struct device *dev,
 	return sprintf(buf, "%s\n", state_str[mgr->state]);
 }
 
+static ssize_t status_show(struct device *dev,
+			   struct device_attribute *attr, char *buf)
+{
+	struct fpga_manager *mgr = to_fpga_manager(dev);
+	u64 status;
+	int len = 0;
+
+	if (!mgr->mops->status)
+		return -ENOENT;
+
+	status = mgr->mops->status(mgr);
+
+	if (status & FPGA_MGR_STATUS_OPERATION_ERR)
+		len += sprintf(buf + len, "reconfig operation error\n");
+	if (status & FPGA_MGR_STATUS_CRC_ERR)
+		len += sprintf(buf + len, "reconfig CRC error\n");
+	if (status & FPGA_MGR_STATUS_INCOMPATIBLE_IMAGE_ERR)
+		len += sprintf(buf + len, "reconfig incompatible image\n");
+	if (status & FPGA_MGR_STATUS_IP_PROTOCOL_ERR)
+		len += sprintf(buf + len, "reconfig IP protocol error\n");
+	if (status & FPGA_MGR_STATUS_FIFO_OVERFLOW_ERR)
+		len += sprintf(buf + len, "reconfig fifo overflow error\n");
+
+	return len;
+}
+
 static DEVICE_ATTR_RO(name);
 static DEVICE_ATTR_RO(state);
+static DEVICE_ATTR_RO(status);
 
 static struct attribute *fpga_mgr_attrs[] = {
 	&dev_attr_name.attr,
 	&dev_attr_state.attr,
+	&dev_attr_status.attr,
 	NULL,
 };
 ATTRIBUTE_GROUPS(fpga_mgr);

diff --git a/include/linux/fpga/fpga-mgr.h b/include/linux/fpga/fpga-mgr.h
index 6e98b66..6c79183 100644
--- a/include/linux/fpga/fpga-mgr.h
+++ b/include/linux/fpga/fpga-mgr.h

@@ -112,6 +112,7 @@ struct fpga_image_info {
  * struct fpga_manager_ops - ops for low level fpga manager drivers
  * @initial_header_size: Maximum number of bytes that should be passed into write_init
  * @state: returns an enum value of the FPGA's state
+ * @status: returns status of the FPGA, including reconfiguration error code
  * @write_init: prepare the FPGA to receive confuration data
  * @write: write count bytes of configuration data to the FPGA
  * @write_sg: write the scatter list of configuration data to the FPGA

@@ -126,6 +127,7 @@ struct fpga_image_info {
 struct fpga_manager_ops {
 	size_t initial_header_size;
 	enum fpga_mgr_states (*state)(struct fpga_manager *mgr);
+	u64 (*status)(struct fpga_manager *mgr);
 	int (*write_init)(struct fpga_manager *mgr,
 			  struct fpga_image_info *info,
 			  const char *buf, size_t count);

@@ -137,6 +139,13 @@ struct fpga_manager_ops {
 	const struct attribute_group **groups;
 };
 
+/* FPGA manager status: Partial/Full Reconfiguration errors */
+#define FPGA_MGR_STATUS_OPERATION_ERR		BIT(0)
+#define FPGA_MGR_STATUS_CRC_ERR			BIT(1)
+#define FPGA_MGR_STATUS_INCOMPATIBLE_IMAGE_ERR	BIT(2)
+#define FPGA_MGR_STATUS_IP_PROTOCOL_ERR		BIT(3)
+#define FPGA_MGR_STATUS_FIFO_OVERFLOW_ERR	BIT(4)
+
 /**
  * struct fpga_manager - fpga manager structure
  * @name: name of low level fpga manager

-- 
2.7.4

Re: [PATCH v4 07/24] fpga: dfl: add feature device infrastructure

From: Moritz Fischer <mdf@kernel.org>
Date: 2018-02-14 21:08:19

HI Hao,

On Tue, Feb 13, 2018 at 05:24:36PM +0800, Wu Hao wrote:

quoted hunk

From: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>

This patch abstracts the common operations of the sub features, and defines
the feature_ops data structure, including init, uinit and ioctl function
pointers. And this patch adds some common helper functions for FME and AFU
drivers, e.g feature_dev_use_begin/end which are used to ensure exclusive
usage of the feature device file.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Kang Luwei <redacted>
Signed-off-by: Zhang Yi <redacted>
Signed-off-by: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
Signed-off-by: Wu Hao <redacted>
---
v2: rebased
v3: use const for feature_ops.
    replace pci related function.
v4: rebase and add more comments in code.
---
 drivers/fpga/dfl.c | 59 +++++++++++++++++++++++++++++++++++++
 drivers/fpga/dfl.h | 85 +++++++++++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 143 insertions(+), 1 deletion(-)

diff --git a/drivers/fpga/dfl.c b/drivers/fpga/dfl.c
index 38dc819..c0aad87 100644
--- a/drivers/fpga/dfl.c
+++ b/drivers/fpga/dfl.c

@@ -74,6 +74,65 @@ static enum fpga_id_type feature_dev_id_type(struct platform_device *pdev)
 	return FPGA_ID_MAX;
 }
 
+void fpga_dev_feature_uinit(struct platform_device *pdev)
+{
+	struct feature *feature;
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);

See comment below w.r.t ordering declarations. Not a must for sure.

+
+	fpga_dev_for_each_feature(pdata, feature)
+		if (feature->ops) {
+			feature->ops->uinit(pdev, feature);
+			feature->ops = NULL;
+		}
+}
+EXPORT_SYMBOL_GPL(fpga_dev_feature_uinit);
+
+static int
+feature_instance_init(struct platform_device *pdev,
+		      struct feature_platform_data *pdata,
+		      struct feature *feature, struct feature_driver *drv)
+{
+	int ret;
+
+	WARN_ON(!feature->ioaddr);

Not sure I understand correctly, is the !feature->ioaddr a use-case that
happens? If not just return early.

+
+	ret = drv->ops->init(pdev, feature);
+	if (ret)
+		return ret;
+
+	feature->ops = drv->ops;
+
+	return ret;
+}
+
+int fpga_dev_feature_init(struct platform_device *pdev,
+			  struct feature_driver *feature_drvs)
+{
+	struct feature *feature;
+	struct feature_driver *drv = feature_drvs;
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	int ret;

We don't have clear guidelines here, but some subsystems want reverse
X-Mas tree declarations.

quoted hunk

+
+	while (drv->ops) {
+		fpga_dev_for_each_feature(pdata, feature) {
+			/* match feature and drv using id */
+			if (feature->id == drv->id) {
+				ret = feature_instance_init(pdev, pdata,
+							    feature, drv);
+				if (ret)
+					goto exit;
+			}
+		}
+		drv++;
+	}
+
+	return 0;
+exit:
+	fpga_dev_feature_uinit(pdev);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(fpga_dev_feature_init);
+
 struct fpga_chardev_info {
 	const char *name;
 	dev_t devt;

diff --git a/drivers/fpga/dfl.h b/drivers/fpga/dfl.h
index d6ecda1..ede1e95 100644
--- a/drivers/fpga/dfl.h
+++ b/drivers/fpga/dfl.h

@@ -132,6 +132,17 @@
 #define PORT_CTRL_SFTRST_ACK	BIT(4)			/* HW ack for reset */
 
 /**
+ * struct feature_driver - sub feature's driver
+ *
+ * @id:	sub feature id.
+ * @ops: ops of this sub feature.
+ */
+struct feature_driver {
+	u64 id;
+	const struct feature_ops *ops;
+};
+
+/**
  * struct feature - sub feature of the feature devices
  *
  * @id:	sub feature id.

@@ -139,13 +150,17 @@
  *		    this index is used to find its mmio resource from the
  *		    feature dev (platform device)'s reources.
  * @ioaddr: mapped mmio resource address.
+ * @ops: ops of this sub feature.
  */
 struct feature {
 	u64 id;
 	int resource_index;
 	void __iomem *ioaddr;
+	const struct feature_ops *ops;
 };
 
+#define DEV_STATUS_IN_USE	0
+
 /**
  * struct feature_platform_data - platform data for feature devices
  *

@@ -155,6 +170,8 @@ struct feature {
  * @dev: ptr to platform device linked with this platform data.
  * @disable_count: count for port disable.
  * @num: number for sub features.
+ * @dev_status: dev status (e.g DEV_STATUS_IN_USE).
+ * @private: ptr to feature dev private data.
  * @features: sub features of this feature dev.
  */
 struct feature_platform_data {

@@ -163,11 +180,44 @@ struct feature_platform_data {
 	struct cdev cdev;
 	struct platform_device *dev;
 	unsigned int disable_count;
-
+	unsigned long dev_status;
+	void *private;
 	int num;
 	struct feature features[0];
 };
 
+static inline int feature_dev_use_begin(struct feature_platform_data *pdata)
+{
+	/* Test and set IN_USE flags to ensure file is exclusively used */
+	if (test_and_set_bit_lock(DEV_STATUS_IN_USE, &pdata->dev_status))
+		return -EBUSY;
+
+	return 0;
+}
+
+static inline void feature_dev_use_end(struct feature_platform_data *pdata)
+{
+	clear_bit_unlock(DEV_STATUS_IN_USE, &pdata->dev_status);
+}
+
+static inline void
+fpga_pdata_set_private(struct feature_platform_data *pdata, void *private)
+{
+	pdata->private = private;
+}
+
+static inline void *fpga_pdata_get_private(struct feature_platform_data *pdata)
+{
+	return pdata->private;
+}
+
+struct feature_ops {
+	int (*init)(struct platform_device *pdev, struct feature *feature);
+	void (*uinit)(struct platform_device *pdev, struct feature *feature);
+	long (*ioctl)(struct platform_device *pdev, struct feature *feature,
+		      unsigned int cmd, unsigned long arg);
+};
+
 #define FPGA_FEATURE_DEV_FME		"dfl-fme"
 #define FPGA_FEATURE_DEV_PORT		"dfl-port"

@@ -177,6 +227,10 @@ static inline int feature_platform_data_size(const int num)
 		num * sizeof(struct feature);
 }
 
+void fpga_dev_feature_uinit(struct platform_device *pdev);
+int fpga_dev_feature_init(struct platform_device *pdev,
+			  struct feature_driver *feature_drvs);
+
 enum fpga_devt_type {
 	FPGA_DEVT_FME,
 	FPGA_DEVT_PORT,

@@ -257,6 +311,15 @@ static inline int fpga_port_reset(struct platform_device *pdev)
 	return ret;
 }
 
+static inline
+struct platform_device *fpga_inode_to_feature_dev(struct inode *inode)
+{
+	struct feature_platform_data *pdata;
+
+	pdata = container_of(inode->i_cdev, struct feature_platform_data, cdev);
+	return pdata->dev;
+}
+
 #define fpga_dev_for_each_feature(pdata, feature)			    \
 	for ((feature) = (pdata)->features;				    \
 	   (feature) < (pdata)->features + (pdata)->num; (feature)++)

@@ -284,6 +347,17 @@ static inline void __iomem *get_feature_ioaddr_by_id(struct device *dev, u64 id)
 	return NULL;
 }
 
+static inline bool is_feature_present(struct device *dev, u64 id)
+{
+	return !!get_feature_ioaddr_by_id(dev, id);
+}
+
+static inline struct device *
+fpga_pdata_to_parent(struct feature_platform_data *pdata)
+{
+	return pdata->dev->dev.parent->parent;
+}
+
 static inline bool feature_is_fme(void __iomem *base)
 {
 	u64 v = readq(base + DFH);

@@ -376,4 +450,13 @@ fpga_cdev_find_port(struct fpga_cdev *cdev, void *data,
 
 	return pdev;
 }
+
+static inline struct fpga_cdev *
+fpga_pdata_to_fpga_cdev(struct feature_platform_data *pdata)
+{
+	struct device *dev = pdata->dev->dev.parent;
+	struct fpga_region *region = to_fpga_region(dev);
+
+	return container_of(region, struct fpga_cdev, region);
+}
 #endif /* __FPGA_DFL_H */

-- 
2.7.4

Thanks,
Moritz

Re: [PATCH v4 07/24] fpga: dfl: add feature device infrastructure

From: Alan Tull <atull@kernel.org>
Date: 2018-02-14 21:14:37

On Wed, Feb 14, 2018 at 3:03 PM, Moritz Fischer [off-list ref] wrote:

Hi Moritz,

HI Hao,

On Tue, Feb 13, 2018 at 05:24:36PM +0800, Wu Hao wrote:

quoted

From: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>

This patch abstracts the common operations of the sub features, and defines
the feature_ops data structure, including init, uinit and ioctl function
pointers. And this patch adds some common helper functions for FME and AFU
drivers, e.g feature_dev_use_begin/end which are used to ensure exclusive
usage of the feature device file.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Kang Luwei <redacted>
Signed-off-by: Zhang Yi <redacted>
Signed-off-by: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
Signed-off-by: Wu Hao <redacted>
---
v2: rebased
v3: use const for feature_ops.
    replace pci related function.
v4: rebase and add more comments in code.
---
 drivers/fpga/dfl.c | 59 +++++++++++++++++++++++++++++++++++++
 drivers/fpga/dfl.h | 85 +++++++++++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 143 insertions(+), 1 deletion(-)

diff --git a/drivers/fpga/dfl.c b/drivers/fpga/dfl.c
index 38dc819..c0aad87 100644
--- a/drivers/fpga/dfl.c
+++ b/drivers/fpga/dfl.c

@@ -74,6 +74,65 @@ static enum fpga_id_type feature_dev_id_type(struct platform_device *pdev)
      return FPGA_ID_MAX;
 }

+void fpga_dev_feature_uinit(struct platform_device *pdev)
+{
+     struct feature *feature;
+     struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);

See comment below w.r.t ordering declarations. Not a must for sure.

quoted

+
+     fpga_dev_for_each_feature(pdata, feature)
+             if (feature->ops) {
+                     feature->ops->uinit(pdev, feature);
+                     feature->ops = NULL;
+             }
+}
+EXPORT_SYMBOL_GPL(fpga_dev_feature_uinit);
+
+static int
+feature_instance_init(struct platform_device *pdev,
+                   struct feature_platform_data *pdata,
+                   struct feature *feature, struct feature_driver *drv)
+{
+     int ret;
+
+     WARN_ON(!feature->ioaddr);

Not sure I understand correctly, is the !feature->ioaddr a use-case that
happens? If not just return early.

quoted

+
+     ret = drv->ops->init(pdev, feature);
+     if (ret)
+             return ret;
+
+     feature->ops = drv->ops;
+
+     return ret;
+}
+
+int fpga_dev_feature_init(struct platform_device *pdev,
+                       struct feature_driver *feature_drvs)
+{
+     struct feature *feature;
+     struct feature_driver *drv = feature_drvs;
+     struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+     int ret;

We don't have clear guidelines here, but some subsystems want reverse
X-Mas tree declarations.

Sounds good!  I agree.

Alan

RE: [PATCH v4 03/24] fpga: mgr: add status for fpga-manager

From: Wu, Hao <hidden>
Date: 2018-02-15 09:42:39

On Tue, Feb 13, 2018 at 3:24 AM, Wu Hao [off-list ref] wrote:

Hi Hao,

quoted

This patch adds status to fpga-manager data structure,

Please update the header to better describe the current changes, this
patch no longer adds status to the fpga mgr data struct.

Sorry, will fix it. Thanks a lot!

Hao

quoted

to allow
driver to store full/partial reconfiguration errors and other
status information, and adds one status callback to fpga_manager_ops
to allow fpga_manager to collect latest status when failures are
detected.

Other than that, looks good to me!  Thanks for taking my feedback.
Alan

quoted

The following sysfs file is created:
* /sys/class/fpga_manager/<fpga>/status
  Return status of fpga manager, including reconfiguration errors.

Signed-off-by: Wu Hao <redacted>

Acked-by: Alan Tull <atull@kernel.org>

RE: [PATCH v4 06/24] fpga: dfl: adds fpga_cdev_find_port

From: Wu, Hao <hidden>
Date: 2018-02-15 09:46:48

On Tue, Feb 13, 2018 at 3:24 AM, Wu Hao [off-list ref] wrote:

Hi Hao,

quoted

For feature devices, we need a method to find the port dedicated
to the device. This patch adds a function fpga_cdev_find_port
for this purpose. e.g FPGA Management Engine (FME) Partial
Reconfiguration sub feature, it uses this function to find
dedicated port on the device for PR function implementation.

OK, that is very clear now, thanks!  Ack with the minor doc changes below.

quoted

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Xiao Guangrong <redacted>
Signed-off-by: Wu Hao <redacted>

Acked-by: Alan Tull <atull@kernel.org>

quoted

---
v3: s/fpga_for_each_port/fpga_cdev_find_port/
    move fpga_cdev_find_port to fpga-dfl module.
v4: improve description in commit message.
    add comments to remind user to put_device after use this function.
---
 drivers/fpga/dfl.c | 34 ++++++++++++++++++++++++++++++++++
 drivers/fpga/dfl.h | 20 ++++++++++++++++++++
 2 files changed, 54 insertions(+)

diff --git a/drivers/fpga/dfl.c b/drivers/fpga/dfl.c
index dcfe5ab..38dc819 100644
--- a/drivers/fpga/dfl.c
+++ b/drivers/fpga/dfl.c

@@ -783,6 +783,40 @@ void fpga_remove_feature_devs(struct fpga_cdev

*cdev)

quoted

 }
 EXPORT_SYMBOL_GPL(fpga_remove_feature_devs);

+/**
+ * __fpga_cdev_find_port - find a port under given container device
+ *
+ * @cdev: container device
+ * @data: data passed to match function
+ * @match: match function used to find specific port from the port device list
+ *
+ * Find a port device under container device. This function needs to be
+ * invoked with lock held.
+ *
+ * This function returns NULL if the device doesn't match and non-zero if it
+ * does.

Good but this would be more brief and more specific and more in line
with recommended format in Documentation/kernel-doc-nano-HOWTO.txt:

 * Return: pointer to port's platform device if successful, NULL otherwise.

quoted

If this function returns non-zero and a reference to the platform
+ * device of port can be obtained.

This if..then type sentence is not complete (missing the 'then' part).
I see where you might have gotten inspiration for it from core.c.
Probably you could just delete the sentence and all would be clear.

Thanks for the review and the suggestion on this, will update the
comments to fix it.

Hao

quoted

+ *
+ * NOTE: you will need to drop the device reference with put_device() after

use.

quoted

+ */

Good

Thanks,
Alan

quoted

+struct platform_device *
+__fpga_cdev_find_port(struct fpga_cdev *cdev, void *data,
+                     int (*match)(struct platform_device *, void *))
+{
+       struct feature_platform_data *pdata;
+       struct platform_device *port_dev;
+
+       list_for_each_entry(pdata, &cdev->port_dev_list, node) {
+               port_dev = pdata->dev;
+
+               if (match(port_dev, data) && get_device(&port_dev->dev))
+                       return port_dev;
+       }
+
+       return NULL;
+}
+EXPORT_SYMBOL_GPL(__fpga_cdev_find_port);
+
 int fpga_port_id(struct platform_device *pdev)
 {
        void __iomem *base = get_feature_ioaddr_by_id(&pdev->dev,

diff --git a/drivers/fpga/dfl.h b/drivers/fpga/dfl.h
index 9b19399..d6ecda1 100644
--- a/drivers/fpga/dfl.h
+++ b/drivers/fpga/dfl.h

@@ -356,4 +356,24 @@ struct fpga_cdev {
 struct fpga_cdev *fpga_enumerate_feature_devs(struct fpga_enum_info

*info);

quoted

 void fpga_remove_feature_devs(struct fpga_cdev *cdev);

+/*
+ * need to drop the device reference with put_device() after use port

platform

quoted

+ * device returned by __fpga_cdev_find_port and fpga_cdev_find_port

functions.

quoted

+ */
+struct platform_device *
+__fpga_cdev_find_port(struct fpga_cdev *cdev, void *data,
+                     int (*match)(struct platform_device *, void *));
+
+static inline struct platform_device *
+fpga_cdev_find_port(struct fpga_cdev *cdev, void *data,
+                   int (*match)(struct platform_device *, void *))
+{
+       struct platform_device *pdev;
+
+       mutex_lock(&cdev->lock);
+       pdev = __fpga_cdev_find_port(cdev, data, match);
+       mutex_unlock(&cdev->lock);
+
+       return pdev;
+}
 #endif /* __FPGA_DFL_H */
--
2.7.4

RE: [PATCH v4 07/24] fpga: dfl: add feature device infrastructure

From: Wu, Hao <hidden>
Date: 2018-02-15 10:05:28

On Wed, Feb 14, 2018 at 3:03 PM, Moritz Fischer [off-list ref] wrote:

Hi Moritz,

quoted

HI Hao,

Hi Alan and Moritz

Thanks a lot for the code review and comments.

quoted

On Tue, Feb 13, 2018 at 05:24:36PM +0800, Wu Hao wrote:

quoted

From: Xiao Guangrong <redacted>

This patch abstracts the common operations of the sub features, and defines
the feature_ops data structure, including init, uinit and ioctl function
pointers. And this patch adds some common helper functions for FME and

AFU

quoted

drivers, e.g feature_dev_use_begin/end which are used to ensure exclusive
usage of the feature device file.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Kang Luwei <redacted>
Signed-off-by: Zhang Yi <redacted>
Signed-off-by: Xiao Guangrong <redacted>
Signed-off-by: Wu Hao <redacted>
---
v2: rebased
v3: use const for feature_ops.
    replace pci related function.
v4: rebase and add more comments in code.
---
 drivers/fpga/dfl.c | 59 +++++++++++++++++++++++++++++++++++++
 drivers/fpga/dfl.h | 85

+++++++++++++++++++++++++++++++++++++++++++++++++++++-

quoted

 2 files changed, 143 insertions(+), 1 deletion(-)

diff --git a/drivers/fpga/dfl.c b/drivers/fpga/dfl.c
index 38dc819..c0aad87 100644
--- a/drivers/fpga/dfl.c
+++ b/drivers/fpga/dfl.c

@@ -74,6 +74,65 @@ static enum fpga_id_type feature_dev_id_type(struct

platform_device *pdev)

quoted

      return FPGA_ID_MAX;
 }

+void fpga_dev_feature_uinit(struct platform_device *pdev)
+{
+     struct feature *feature;
+     struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);

See comment below w.r.t ordering declarations. Not a must for sure.

quoted

+
+     fpga_dev_for_each_feature(pdata, feature)
+             if (feature->ops) {
+                     feature->ops->uinit(pdev, feature);
+                     feature->ops = NULL;
+             }
+}
+EXPORT_SYMBOL_GPL(fpga_dev_feature_uinit);
+
+static int
+feature_instance_init(struct platform_device *pdev,
+                   struct feature_platform_data *pdata,
+                   struct feature *feature, struct feature_driver *drv)
+{
+     int ret;
+
+     WARN_ON(!feature->ioaddr);

Not sure I understand correctly, is the !feature->ioaddr a use-case that
happens? If not just return early.

Actually this should never happen (init a feature without mapped mmio
resource address). If this warning is seen, that means there should be
critical issues somewhere in driver enumeration code. But sure, I can just
use if () return instead. : )

quoted

+
+     ret = drv->ops->init(pdev, feature);
+     if (ret)
+             return ret;
+
+     feature->ops = drv->ops;
+
+     return ret;
+}
+
+int fpga_dev_feature_init(struct platform_device *pdev,
+                       struct feature_driver *feature_drvs)
+{
+     struct feature *feature;
+     struct feature_driver *drv = feature_drvs;
+     struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+     int ret;

We don't have clear guidelines here, but some subsystems want reverse
X-Mas tree declarations.

Sounds good!  I agree.

Do you mean we should reverse fpga_xxx definitions? If yes, then I can update
the code to use fpga_dfl_xxx or dfl_xxx instead. : )

Thanks
Hao

Alan

Re: [PATCH v4 07/24] fpga: dfl: add feature device infrastructure

From: Moritz Fischer <mdf@kernel.org>
Date: 2018-02-15 19:53:59

On Thu, Feb 15, 2018 at 10:05:20AM +0000, Wu, Hao wrote:

quoted

On Wed, Feb 14, 2018 at 3:03 PM, Moritz Fischer [off-list ref] wrote:

Hi Moritz,

quoted

HI Hao,

Hi Alan and Moritz

Thanks a lot for the code review and comments.

quoted

On Tue, Feb 13, 2018 at 05:24:36PM +0800, Wu Hao wrote:

quoted

From: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>

This patch abstracts the common operations of the sub features, and defines
the feature_ops data structure, including init, uinit and ioctl function
pointers. And this patch adds some common helper functions for FME and

AFU

quoted

drivers, e.g feature_dev_use_begin/end which are used to ensure exclusive
usage of the feature device file.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Kang Luwei <redacted>
Signed-off-by: Zhang Yi <redacted>
Signed-off-by: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
Signed-off-by: Wu Hao <redacted>
---
v2: rebased
v3: use const for feature_ops.
    replace pci related function.
v4: rebase and add more comments in code.
---
 drivers/fpga/dfl.c | 59 +++++++++++++++++++++++++++++++++++++
 drivers/fpga/dfl.h | 85

+++++++++++++++++++++++++++++++++++++++++++++++++++++-

quoted

 2 files changed, 143 insertions(+), 1 deletion(-)

diff --git a/drivers/fpga/dfl.c b/drivers/fpga/dfl.c
index 38dc819..c0aad87 100644
--- a/drivers/fpga/dfl.c
+++ b/drivers/fpga/dfl.c

@@ -74,6 +74,65 @@ static enum fpga_id_type feature_dev_id_type(struct

platform_device *pdev)

quoted

      return FPGA_ID_MAX;
 }

+void fpga_dev_feature_uinit(struct platform_device *pdev)
+{
+     struct feature *feature;
+     struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);

See comment below w.r.t ordering declarations. Not a must for sure.

quoted

+
+     fpga_dev_for_each_feature(pdata, feature)
+             if (feature->ops) {
+                     feature->ops->uinit(pdev, feature);
+                     feature->ops = NULL;
+             }
+}
+EXPORT_SYMBOL_GPL(fpga_dev_feature_uinit);
+
+static int
+feature_instance_init(struct platform_device *pdev,
+                   struct feature_platform_data *pdata,
+                   struct feature *feature, struct feature_driver *drv)
+{
+     int ret;
+
+     WARN_ON(!feature->ioaddr);

Not sure I understand correctly, is the !feature->ioaddr a use-case that
happens? If not just return early.

Actually this should never happen (init a feature without mapped mmio
resource address). If this warning is seen, that means there should be
critical issues somewhere in driver enumeration code. But sure, I can just
use if () return instead. : )

quoted

+
+     ret = drv->ops->init(pdev, feature);
+     if (ret)
+             return ret;
+
+     feature->ops = drv->ops;
+
+     return ret;
+}
+
+int fpga_dev_feature_init(struct platform_device *pdev,
+                       struct feature_driver *feature_drvs)
+{
+     struct feature *feature;
+     struct feature_driver *drv = feature_drvs;
+     struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+     int ret;

We don't have clear guidelines here, but some subsystems want reverse
X-Mas tree declarations.

Sounds good!  I agree.

Do you mean we should reverse fpga_xxx definitions? If yes, then I can update
the code to use fpga_dfl_xxx or dfl_xxx instead. : )

More a stylistic thing, in the sense that you'd have the longest line
first:

+     struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+     struct feature_driver *drv = feature_drvs;
+     struct feature *feature;
+     int ret;

Instead of:

+     struct feature *feature;
+     struct feature_driver *drv = feature_drvs;
+     struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+     int ret;

as I said not a big deal, some subsystems want you to do this, I don't
think we made that a strict rule so far, but it makes it visually more
pleasing ;-)

Moritz

RE: [PATCH v4 07/24] fpga: dfl: add feature device infrastructure

From: Wu, Hao <hidden>
Date: 2018-02-18 02:15:26

On Thu, Feb 15, 2018 at 10:05:20AM +0000, Wu, Hao wrote:

quoted

On Wed, Feb 14, 2018 at 3:03 PM, Moritz Fischer [off-list ref] wrote:

Hi Moritz,

quoted

HI Hao,

Hi Alan and Moritz

Thanks a lot for the code review and comments.

quoted

On Tue, Feb 13, 2018 at 05:24:36PM +0800, Wu Hao wrote:

quoted

From: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>

This patch abstracts the common operations of the sub features, and

defines

quoted

the feature_ops data structure, including init, uinit and ioctl function
pointers. And this patch adds some common helper functions for FME and

AFU

quoted

drivers, e.g feature_dev_use_begin/end which are used to ensure

exclusive

quoted

usage of the feature device file.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Kang Luwei <redacted>
Signed-off-by: Zhang Yi <redacted>
Signed-off-by: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
Signed-off-by: Wu Hao <redacted>
---
v2: rebased
v3: use const for feature_ops.
    replace pci related function.
v4: rebase and add more comments in code.
---
 drivers/fpga/dfl.c | 59 +++++++++++++++++++++++++++++++++++++
 drivers/fpga/dfl.h | 85

+++++++++++++++++++++++++++++++++++++++++++++++++++++-

quoted

 2 files changed, 143 insertions(+), 1 deletion(-)

diff --git a/drivers/fpga/dfl.c b/drivers/fpga/dfl.c
index 38dc819..c0aad87 100644
--- a/drivers/fpga/dfl.c
+++ b/drivers/fpga/dfl.c

@@ -74,6 +74,65 @@ static enum fpga_id_type

feature_dev_id_type(struct

quoted

platform_device *pdev)

quoted

      return FPGA_ID_MAX;
 }

+void fpga_dev_feature_uinit(struct platform_device *pdev)
+{
+     struct feature *feature;
+     struct feature_platform_data *pdata = dev_get_platdata(&pdev-

dev);

quoted

See comment below w.r.t ordering declarations. Not a must for sure.

quoted

+
+     fpga_dev_for_each_feature(pdata, feature)
+             if (feature->ops) {
+                     feature->ops->uinit(pdev, feature);
+                     feature->ops = NULL;
+             }
+}
+EXPORT_SYMBOL_GPL(fpga_dev_feature_uinit);
+
+static int
+feature_instance_init(struct platform_device *pdev,
+                   struct feature_platform_data *pdata,
+                   struct feature *feature, struct feature_driver *drv)
+{
+     int ret;
+
+     WARN_ON(!feature->ioaddr);

Not sure I understand correctly, is the !feature->ioaddr a use-case that
happens? If not just return early.

Actually this should never happen (init a feature without mapped mmio
resource address). If this warning is seen, that means there should be
critical issues somewhere in driver enumeration code. But sure, I can just
use if () return instead. : )

quoted

+
+     ret = drv->ops->init(pdev, feature);
+     if (ret)
+             return ret;
+
+     feature->ops = drv->ops;
+
+     return ret;
+}
+
+int fpga_dev_feature_init(struct platform_device *pdev,
+                       struct feature_driver *feature_drvs)
+{
+     struct feature *feature;
+     struct feature_driver *drv = feature_drvs;
+     struct feature_platform_data *pdata = dev_get_platdata(&pdev-

dev);

quoted

+     int ret;

We don't have clear guidelines here, but some subsystems want reverse
X-Mas tree declarations.

Sounds good!  I agree.

Do you mean we should reverse fpga_xxx definitions? If yes, then I can update
the code to use fpga_dfl_xxx or dfl_xxx instead. : )

More a stylistic thing, in the sense that you'd have the longest line
first:

+     struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+     struct feature_driver *drv = feature_drvs;
+     struct feature *feature;
+     int ret;

Instead of:

+     struct feature *feature;
+     struct feature_driver *drv = feature_drvs;
+     struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+     int ret;

as I said not a big deal, some subsystems want you to do this, I don't
think we made that a strict rule so far, but it makes it visually more
pleasing ;-)

Oh.. I see. Thanks for the suggestion, I will update the patch for this.

Hao

Moritz

Re: [PATCH v4 01/24] docs: fpga: add a document for FPGA Device Feature List (DFL) Framework Overview

From: Alan Tull <atull@kernel.org>
Date: 2018-02-26 22:49:39

On Tue, Feb 13, 2018 at 3:24 AM, Wu Hao [off-list ref] wrote:

Hi Hao,

The new overview and diagrams are a great improvement, thanks!

Some comments below.

quoted hunk

Add a document for FPGA Device Feature List (DFL) Framework Overview.

Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Xiao Guangrong <redacted>
Signed-off-by: Wu Hao <redacted>
---
v2: added FME fpga-mgr/bridge/region platform driver to driver organization.
    updated open discussion per current implementation.
    fixed some typos.
v3: use FPGA base region as container device instead of fpga-dev class.
    split common enumeration code from pcie driver to functions exposed by
    device feature list framework.
    update FME performance reporting which supports both integrated (iperf/)
    and discrete (dperf/) FPGA solutions.
v4: rename this doc to Device Feature List (DFL) Framework Overview (dfl.txt)
    add Device Feature List introduction and re-organize the content.
    add description for port reset, bitstream_id/metadata and etc.
---
 Documentation/fpga/dfl.txt | 382 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 382 insertions(+)
 create mode 100644 Documentation/fpga/dfl.txt

diff --git a/Documentation/fpga/dfl.txt b/Documentation/fpga/dfl.txt
new file mode 100644
index 0000000..28e4e22
--- /dev/null
+++ b/Documentation/fpga/dfl.txt

@@ -0,0 +1,382 @@
+===============================================================================
+              FPGA Device Feature List (DFL) Framework Overview
+-------------------------------------------------------------------------------
+                Enno Luebbers <enno.luebbers@intel.com>
+                Xiao Guangrong <guangrong.xiao@linux.intel.com>
+                Wu Hao <hao.wu@intel.com>
+
+The Device Feature List (DFL) FPGA framework (and drivers according to this
+this framework) hides the very details of low layer hardwares and provides
+unified interfaces for userspace applications to configure, enumerate, open
+and access FPGA accelerators on platforms implemented the DFL in the device
+memory, and enables system level management functions such as FPGA
+reconfiguration, power management and virtualization.

This is good, but could be clearer if you break this up into > 1 sentence.

Power management and virtualization are coming in the future, aren't
implemented here.  The upstream will push back on documentation of
features that aren't included in a patchset, since that can all
change.   The cover letter can mention the direction you intend to go
in the future.  The future patchsets can document whatever
functionality they add.  Noted below a few times also.

+
+Device Feature List (DFL) Overview
+==================================
+Device Feature List (DFL) defines a link list of feature headers within the

*linked

+device MMIO space to provide an extensible way of adding features. Software can
+walk through these predefined data structures to enumerate FPGA features:
+FPGA Interface Unit (FIU), Accelerated Function Unit (AFU) and Private Features,
+as illustrated below:
+
+    Header            Header            Header            Header
+ +----------+  +-->+----------+  +-->+----------+  +-->+----------+
+ |   Type   |  |   |  Type    |  |   |  Type    |  |   |  Type    |
+ |   FIU    |  |   | Private  |  |   | Private  |  |   | Private  |
+ +----------+  |   | Feature  |  |   | Feature  |  |   | Feature  |
+ | Next_DFH |--+   +----------+  |   +----------+  |   +----------+
+ +----------+      | Next_DFH |--+   | Next_DFH |--+   | Next_DFH |--> NULL
+ |    ID    |      +----------+      +----------+      +----------+
+ +----------+      |    ID    |      |    ID    |      |    ID    |
+ | Next_AFU |--+   +----------+      +----------+      +----------+
+ +----------+  |   | Feature  |      | Feature  |      | Feature  |
+ | Feature  |  |   | Region   |      | Region   |      | Region   |
+ | Region   |  |   +----------+      +----------+      +----------+
+ +----------+  |      Header
+               +-->+----------+
+                   |   Type   |
+                   |   AFU    |
+                   +----------+
+                   |   GUID   |
+                   +----------+
+                   |  Feature |
+                   |  Region  |
+                   +----------+
+

OK, yes, this is much more clear.

+FPGA Interface Unit (FIU) represents a standalone functional unit for the
+interface to FPGA, e.g the FPGA Management Engine (FME) and Port. (more

Nit, don't need the period after Port.

+descriptions on FME and Port in later sections). Accelerated Function Unit (AFU)
+represents a FPGA programmable region, and usually connects to a FIU as its

The AFU is always connected to a port (which is like an FPGA bridge), right?

+child as illustrated above. Private Features are sub features of the FIUs/AFUs,
+they could be various function blocks with different ID, but they are linked to
+one list via the Next Device Feature Header (DFH) pointer as children. The
+registers for actual functions are grouped as feature regions which always
+follow the common header registers. The feature region located after the FIU
+header, is named as header register set for given FIU type. e.g FME Header
+Register Set.

Is there anything else in the Feature Region besides the register set?
 Why not just call it the Feature Register Set or Header Register Set
depending on whether it's in a FIU header or a private feature header?

+
+This Device Feature List provides a way of linking features together, it's
+convenient for software to locate each feature by walking through this list,
+and can be implemented in register regions of any FPGA device.
+
+FIU - FME (FPGA Management Engine)
+==================================
+The FPGA Management Engine performs power and thermal management, error
+reporting, reconfiguration, performance reporting for integrated and discrete
+solution, and other infrastructure functions. Each FPGA device only has one FME.

Currently only reconfiguration is supported.

By integrated, you are referring to FPGAs in SoC's?  Is there a reason
to mention "for integrated and discrete solution" here?

+
+User-space applications can acquire exclusive access to the FME using open(),
+and release it using close().
+
+The following functions are exposed through ioctls:
+
+ Get driver API version (FPGA_GET_API_VERSION)
+ Check for extensions (FPGA_CHECK_EXTENSION)
+ Assign port to PF (*FPGA_FME_PORT_ASSIGN)
+ Release port from PF (*FPGA_FME_PORT_RELEASE)
+ Program bitstream (FPGA_FME_PORT_PR)
+
+*FPGA_FME_PORT_ASSIGN/RELEASE are only used for FPGA virtualization. Please
+refer to later section "FPGA Virtualization - PCIe" for more details.
+
+More functions are exposed through sysfs
+(/sys/class/fpga_region/regionX/dfl-fme.n/):
+
+ Read bitstream ID (bitstream_id)
+     bitstream_id indicates version of the blue bitstream (static FPGA region).
+
+ Read bitstream metadata (bitstream_metadata)
+     bistream_metadata includes more detailed information of the blue bitstream
+     (static FPGA region), e.g synthesis date and seed.
+
+ Read number of ports (ports_num)
+     one FPGA device may have more than 1 port, this sysfs interface indicates
+     how many ports the FPGA device has.
+
+ Read socket ID (socket_id)
+     socket_id is only used in integrated solution, to indicate which socket
+     the FPGA device belongs to.

Integrated?  Do you mean discrete FPGAs?  Confused here, sorry.  I
don't see this implemented in the patchset, so I'm not clear what this
will look like.  Should add this later when it is implemented.  Of
course, since you've mentioned this here, we could discuss it.

+
+ Read performance counters (iperf/ and dperf/)
+ Power management (power_mgmt/)
+ Thermal management (thermal_mgmt/)
+ Error reporting (errors/)

These aren't implemented here, shouldn't be documented here.

+
+FIU - PORT
+==========
+A port represents the interface between the static FPGA fabric (the "blue
+bitstream") and a partially reconfigurable region containing an AFU (the "green
+bitstream"). It controls the communication from SW to the accelerator and
+exposes features such as reset and debug. Each FPGA device may have more than
+1 port.

I suggest adding that there is always one port per PR region (or AFU).

+
+AFU
+===
+An AFU is attached to a port FIU and exposes a fixed length MMIO region to be
+used for accelerator-specific control registers.
+
+User-space applications can acquire exclusive access to an AFU attached to a
+port by using open() on the port device node, and release it using close().
+
+The following functions are exposed through ioctls:
+
+ Get driver API version (FPGA_GET_API_VERSION)
+ Check for extensions (FPGA_CHECK_EXTENSION)
+ Get port info (FPGA_PORT_GET_INFO)
+ Get MMIO region info (FPGA_PORT_GET_REGION_INFO)
+ Map DMA buffer (FPGA_PORT_DMA_MAP)
+ Unmap DMA buffer (FPGA_PORT_DMA_UNMAP)
+ Reset AFU (*FPGA_PORT_RESET)
+ Enable UMsg (FPGA_PORT_UMSG_ENABLE)
+ Disable UMsg (FPGA_PORT_UMSG_DISABLE)
+ Set UMsg mode (FPGA_PORT_UMSG_SET_MODE)
+ Set UMsg base address (FPGA_PORT_UMSG_SET_BASE_ADDR)

I think the last four aren't implemented in this patchset.  Generally
it's best if documentation covers what is currently implemented.  With
lists like this, you can add to the list as your future patchsets add
the functionality.

+
+*FPGA_PORT_RESET: reset the FPGA Port and its AFU. Userspace can do Port reset
+at any time, e.g during DMA or Partial Reconfiguration. But it should never
+cause any system level issue, only functional failure (e.g DMA or PR operation
+failure) and be recoverable from the failure.

Thanks for noting this.

+
+User-space applications can also mmap() accelerator MMIO regions.
+
+More functions are exposed through sysfs:
+(/sys/class/fpga_region/<regionX>/<dfl-port.m>/):
+
+ Read Accelerator GUID (afu_id)
+ Error reporting (errors/)
+
+DFL Framework Overview
+======================
+
+         +----------+    +--------+ +--------+ +--------+
+         |   FME    |    |  AFU   | |  AFU   | |  AFU   |
+         |  Module  |   | Module | | Module | | Module |
+         +----------+    +--------+ +--------+ +--------+
+                 +-----------------------+
+                 | FPGA Container Device |    Device Feature List
+                 |  (FPGA Base Region)   |         Framework
+                 +-----------------------+
+--------------------------------------------------------------------
+               +----------------------------+
+               |   FPGA Bus Device Module   |
+               | (e.g PCIE/Platform Device) |
+               +----------------------------+

This name almost sounds like it's adding a FPGA Bus.  How about DFL
Device Module?

+                 +------------------------+
+                 |  FPGA Hardware Device  |
+                 +------------------------+
+
+DFL Framework in kernel provides common interfaces to create container device
+(FPGA base region), discover feature devices and their private features from the
+given Device Feature Lists, and create platform devices for feature devices
+(e.g FME, Port and AFU) with related resources under the container device. It
+also abstracts operations for the private features and exposes common ops to
+feature device drivers.
+
+The FPGA Bus Device could be different devices e.g PCIe device, platform device
+and etc. Its driver is always loaded first once the device is detected on its
+own bus. This driver plays an infrastructural role in the driver architecture.
+It locates the DFLs in the device memory, handles them and related resources
+to common interfaces from DFL framework for enumeration. (Please refer to
+drivers/fpga/dfl.c for detailed enumeration APIs).
+
+The FPGA Management Engine (FME) driver is a platform driver which is loaded
+automatically after FME platform device creation from the PCIE driver.

Perhaps "...from the DFL device module" would be more genereric.

It
+provides the key features for FPGA management, including:
+
+       a) Power and thermal management, error reporting, performance reporting
+          and other infrastructure functions. Users can access these functions
+          via sysfs interfaces exposed by FME driver.

You should add this when it's implemented.

+       b) Partial Reconfiguration. The FME driver creates FPGA manager, FPGA
+          bridges and FPGA regions during PR sub feature initialization; Once
+          it receives an FPGA_FME_PORT_PR ioctl from user, it invokes the
+          common interface function from FPGA Region to complete the partial
+          reconfiguration of the bitstream to the given port.
+       c) Port management for virtualization. The FME driver introduces two
+          ioctls, FPGA_FME_PORT_RELEASE (releases given port from PF) and
+          FPGA_FME_PORT_ASSIGN (assigns the port back to PF). Once the port is
+          released from the PF, it can be assigned to the VF through the SRIOV
+          interfaces provided by PCIE driver. (Refer to "FPGA virtualization"
+          for more details).
+
+Similar to the the FME driver, the FPGA Accelerated Function Unit (AFU) driver
+is probed once the AFU platform device is created. The main function of this
+module is to provide an interface for userspace applications to access the
+individual accelerators, including basic reset control on port, AFU MMIO region
+export, dma buffer mapping service, UMsg notification, and remote debug
+functions (see above).
+
+After feature platform devices creation, matched platform drivers will be loaded
+automatically to handle different functionalities. Please refer to next sections
+for detailed information on functional units which has been already implemented
+under this DFL framework.
+
+Partial Reconfiguration
+=======================
+As mentioned above, accelerators can be reconfigured through partial
+reconfiguration of a green bitstream file (GBS). The green bitstream must have
+been generated for the exact blue bitstream and targeted reconfigurable region
+(port) of the FPGA; otherwise, the reconfiguration operation will fail and
+possibly cause system instability. This compatibility can be checked by
+comparing the interface ID noted in the GBS header against the interface ID
+exposed by the FME through sysfs (see above).

I think the method of checking compatibility changed and isn't
interface_id anymore.

This check is usually done by
+user-space before calling the reconfiguration IOCTL.
+
+FPGA virtualization - PCIe
+==========================

This section can be added when virtualization support is added.

+This section describes the virtualization support on DFL based FPGA device to
+enable accessing an accelerator from applications running in a virtual machine
+(VM). This section only describes the PCIe based FPGA device with SRIOV support.
+
+Features supported by the particular FPGA device are exposed through Device
+Feature Lists, as illustrated below:
+
+  +-------------------------------+  +-------------+
+  |              PF               |  |     VF      |
+  +-------------------------------+  +-------------+
+      ^            ^         ^              ^
+      |            |         |              |
++-----|------------|---------|--------------|-------+
+|     |            |         |              |       |
+|  +-----+     +-------+ +-------+      +-------+   |
+|  | FME |     | Port0 | | Port1 |      | Port2 |   |
+|  +-----+     +-------+ +-------+      +-------+   |
+|                  ^         ^              ^       |
+|                  |         |              |       |
+|              +-------+ +------+       +-------+   |
+|              |  AFU  | |  AFU |       |  AFU  |   |
+|              +-------+ +------+       +-------+   |
+|                                                   |
+|                 FPGA PCIe Device                  |
++---------------------------------------------------+
+
+FME is always accessed through the physical function (PF).
+
+Ports (and related AFUs) are accessed via PF by default, but could be exposed
+through virtual function (VF) devices via PCIe SRIOV. Each VF only contains
+1 Port and 1 AFU for isolation. Users could assign individual VFs (accelerators)
+created via PCIe SRIOV interface, to virtual machines.
+
+The driver organization in virtualization case is illustrated below:
+
+  +-------++------++------+             |
+  | FME   || FME  || FME  |             |
+  | FPGA  || FPGA || FPGA |             |
+  |Manager||Bridge||Region|             |
+  +-------++------++------+             |
+  +-----------------------+  +--------+ |             +--------+
+  |          FME          |  |  AFU   | |             |  AFU   |
+  |         Module        |  | Module | |             | Module |
+  +-----------------------+  +--------+ |             +--------+
+        +-----------------------+       |       +-----------------------+
+        | FPGA Container Device |       |       | FPGA Container Device |
+        |  (FPGA Base Region)   |       |       |  (FPGA Base Region)   |
+        +-----------------------+       |       +-----------------------+
+          +------------------+          |         +------------------+
+          | FPGA PCIE Module |          | Virtual | FPGA PCIE Module |
+          +------------------+   Host   | Machine +------------------+
+ -------------------------------------- | ------------------------------
+           +---------------+            |          +---------------+
+           | PCI PF Device |            |          | PCI VF Device |
+           +---------------+            |          +---------------+
+
+FPGA PCIe device driver is always loaded first once a FPGA PCIe PF or VF device
+is detected. It:
+
+       a) finish enumeration on both FPGA PCIe PF and VF device using common
+          interfaces from DFL framework.
+       b) supports SRIOV.
+
+The FME device driver plays a management role in this driver architecture, it
+provides ioctls to release Port from PF and assign Port to PF. After release
+a port from PF, then it's safe to expose this port through a VF via PCIe SRIOV
+sysfs interface.
+
+To enable accessing an accelerator from applications running in a VM, the
+respective AFU's port needs to be assigned to a VF using the following steps:
+
+       a) The PF owns all AFU ports by default. Any port that needs to be
+          reassigned to a VF must first be released through the
+          FPGA_FME_PORT_RELEASE ioctl on the FME device.
+
+       b) Once N ports are released from PF, then user can use command below
+          to enable SRIOV and VFs. Each VF owns only one Port with AFU.
+
+          echo N > $PCI_DEVICE_PATH/sriov_numvfs
+
+       c) Pass through the VFs to VMs
+
+       d) The AFU under VF is accessible from applications in VM (using the
+          same driver inside the VF).
+
+Note that an FME can't be assigned to a VF, thus PR and other management
+functions are only available via the PF.
+
+Device enumeration
+==================
+This section introduces how applications enumerate the fpga device from
+the sysfs hierarchy under /sys/class/fpga_region.
+
+In the example below, two DFL based FPGA devices are installed in the host. Each
+fpga device has one FME and two ports (AFUs).
+
+FPGA regions are created under /sys/class/fpga_region/
+
+       /sys/class/fpga_region/region0
+       /sys/class/fpga_region/region1
+       /sys/class/fpga_region/region2
+       ...
+
+Application needs to search each regionX folder, if feature device is found,
+(e.g "dfl-port.n" or "dfl-fme.m" is found), then it's the base
+fpga region which represents the FPGA device.
+
+Each base region has one FME and two ports (AFUs) as child devices:
+
+       /sys/class/fpga_region/region0/dfl-fme.0
+       /sys/class/fpga_region/region0/dfl-port.0
+       /sys/class/fpga_region/region0/dfl-port.1
+       ...
+
+       /sys/class/fpga_region/region3/dfl-fme.1
+       /sys/class/fpga_region/region3/dfl-port.2
+       /sys/class/fpga_region/region3/dfl-port.3
+       ...
+
+In general, the FME/AFU sysfs interfaces are named as follows:
+
+       /sys/class/fpga_region/<regionX>/<dfl-fme.n>/
+       /sys/class/fpga_region/<regionX>/<dfl-port.m>/
+
+with 'n' consecutively numbering all FMEs and 'm' consecutively numbering all
+ports.
+
+The device nodes used for ioctl() or mmap() can be referenced through:
+
+       /sys/class/fpga_region/<regionX>/<dfl-fme.n>/dev
+       /sys/class/fpga_region/<regionX>/<dfl-port.n>/dev
+
+Add new FIUs support
+====================
+It's possible that developers made some new function blocks (FIUs) under this
+DFL framework, then new platform device driver needs to be developed for the
+new feature dev (FIU) following the same way as existing feature dev drivers
+(e.g FME and Port/AFU platform device driver). Besides that, it requires
+modification on DFL framework enumeration code too, for new FIU type detection
+and related platform devices creation.
+
+Add new private features support
+================================
+In some cases, we may need to add some new private features to existing FIUs
+(e.g FME or Port). Developers don't need to touch enumeration code in DFL
+framework, as each private feature will be parsed automatically, and related
+mmio resources can be found under FIU platform device created by DFL framework.
+Developer only needs to provide a sub feature driver with matched feature id.
+FME Partial Reconfiguration Sub Feature driver (see drivers/fpga/dfl-fme-pr.c)
+could be a reference.
+
+Open discussion
+===============
+FME driver exports one ioctl (FPGA_FME_PORT_PR) for partial reconfiguration to
+user now. In the future, if unified user interfaces for reconfiguration are
+added, FME driver should switch to them from ioctl interface.
--
2.7.4

Re: [PATCH v4 01/24] docs: fpga: add a document for FPGA Device Feature List (DFL) Framework Overview

From: Wu Hao <hidden>
Date: 2018-02-27 02:21:54

On Mon, Feb 26, 2018 at 04:48:55PM -0600, Alan Tull wrote:

On Tue, Feb 13, 2018 at 3:24 AM, Wu Hao [off-list ref] wrote:

Hi Hao,

The new overview and diagrams are a great improvement, thanks!

Some comments below.

Hi Alan

Thanks a lot for the comments.

quoted

Add a document for FPGA Device Feature List (DFL) Framework Overview.

Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Xiao Guangrong <redacted>
Signed-off-by: Wu Hao <redacted>
---
v2: added FME fpga-mgr/bridge/region platform driver to driver organization.
    updated open discussion per current implementation.
    fixed some typos.
v3: use FPGA base region as container device instead of fpga-dev class.
    split common enumeration code from pcie driver to functions exposed by
    device feature list framework.
    update FME performance reporting which supports both integrated (iperf/)
    and discrete (dperf/) FPGA solutions.
v4: rename this doc to Device Feature List (DFL) Framework Overview (dfl.txt)
    add Device Feature List introduction and re-organize the content.
    add description for port reset, bitstream_id/metadata and etc.
---
 Documentation/fpga/dfl.txt | 382 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 382 insertions(+)
 create mode 100644 Documentation/fpga/dfl.txt

diff --git a/Documentation/fpga/dfl.txt b/Documentation/fpga/dfl.txt
new file mode 100644
index 0000000..28e4e22
--- /dev/null
+++ b/Documentation/fpga/dfl.txt

@@ -0,0 +1,382 @@
+===============================================================================
+              FPGA Device Feature List (DFL) Framework Overview
+-------------------------------------------------------------------------------
+                Enno Luebbers <enno.luebbers@intel.com>
+                Xiao Guangrong <guangrong.xiao@linux.intel.com>
+                Wu Hao <hao.wu@intel.com>
+
+The Device Feature List (DFL) FPGA framework (and drivers according to this
+this framework) hides the very details of low layer hardwares and provides
+unified interfaces for userspace applications to configure, enumerate, open
+and access FPGA accelerators on platforms implemented the DFL in the device
+memory, and enables system level management functions such as FPGA
+reconfiguration, power management and virtualization.

This is good, but could be clearer if you break this up into > 1 sentence.

Sure.

Power management and virtualization are coming in the future, aren't
implemented here.  The upstream will push back on documentation of
features that aren't included in a patchset, since that can all
change.   The cover letter can mention the direction you intend to go
in the future.  The future patchsets can document whatever
functionality they add.  Noted below a few times also.

OK, then I will remove the features that aren't included in this patchset,
and add them later with implementation.

quoted

+
+Device Feature List (DFL) Overview
+==================================
+Device Feature List (DFL) defines a link list of feature headers within the

*linked

will fix this.

quoted

+device MMIO space to provide an extensible way of adding features. Software can
+walk through these predefined data structures to enumerate FPGA features:
+FPGA Interface Unit (FIU), Accelerated Function Unit (AFU) and Private Features,
+as illustrated below:
+
+    Header            Header            Header            Header
+ +----------+  +-->+----------+  +-->+----------+  +-->+----------+
+ |   Type   |  |   |  Type    |  |   |  Type    |  |   |  Type    |
+ |   FIU    |  |   | Private  |  |   | Private  |  |   | Private  |
+ +----------+  |   | Feature  |  |   | Feature  |  |   | Feature  |
+ | Next_DFH |--+   +----------+  |   +----------+  |   +----------+
+ +----------+      | Next_DFH |--+   | Next_DFH |--+   | Next_DFH |--> NULL
+ |    ID    |      +----------+      +----------+      +----------+
+ +----------+      |    ID    |      |    ID    |      |    ID    |
+ | Next_AFU |--+   +----------+      +----------+      +----------+
+ +----------+  |   | Feature  |      | Feature  |      | Feature  |
+ | Feature  |  |   | Region   |      | Region   |      | Region   |
+ | Region   |  |   +----------+      +----------+      +----------+
+ +----------+  |      Header
+               +-->+----------+
+                   |   Type   |
+                   |   AFU    |
+                   +----------+
+                   |   GUID   |
+                   +----------+
+                   |  Feature |
+                   |  Region  |
+                   +----------+
+

OK, yes, this is much more clear.

quoted

+FPGA Interface Unit (FIU) represents a standalone functional unit for the
+interface to FPGA, e.g the FPGA Management Engine (FME) and Port. (more

Nit, don't need the period after Port.

will fix this.

quoted

+descriptions on FME and Port in later sections). Accelerated Function Unit (AFU)
+represents a FPGA programmable region, and usually connects to a FIU as its

The AFU is always connected to a port (which is like an FPGA bridge), right?

Yes. Only FME and Port is defined now, and AFU is always connected to a port.

quoted

+child as illustrated above. Private Features are sub features of the FIUs/AFUs,
+they could be various function blocks with different ID, but they are linked to
+one list via the Next Device Feature Header (DFH) pointer as children. The
+registers for actual functions are grouped as feature regions which always
+follow the common header registers. The feature region located after the FIU
+header, is named as header register set for given FIU type. e.g FME Header
+Register Set.

Is there anything else in the Feature Region besides the register set?

No.

 Why not just call it the Feature Register Set or Header Register Set
depending on whether it's in a FIU header or a private feature header?

Agree. Feature Region is mentioned in spec, but let's use Feature Register Set
or Header Register Set instead, as it seems to be more clear. Will update the
diagram as well.

quoted

+
+This Device Feature List provides a way of linking features together, it's
+convenient for software to locate each feature by walking through this list,
+and can be implemented in register regions of any FPGA device.
+
+FIU - FME (FPGA Management Engine)
+==================================
+The FPGA Management Engine performs power and thermal management, error
+reporting, reconfiguration, performance reporting for integrated and discrete
+solution, and other infrastructure functions. Each FPGA device only has one FME.

Currently only reconfiguration is supported.

By integrated, you are referring to FPGAs in SoC's?  Is there a reason
to mention "for integrated and discrete solution" here?

No, it's not FPGAs in SoC's, but server platform with in-package FPGA.
There are 2 different private features for performance reporting, one is for
the in-package FPGA on server, the other one is for discrete PCIe FPGA
acceleration cards.

To be accurate, will use "server integrated" instead to avoid confusion.
But anyway, these will be removed for now as they are not implemented in this
patchset.

quoted

+
+User-space applications can acquire exclusive access to the FME using open(),
+and release it using close().
+
+The following functions are exposed through ioctls:
+
+ Get driver API version (FPGA_GET_API_VERSION)
+ Check for extensions (FPGA_CHECK_EXTENSION)
+ Assign port to PF (*FPGA_FME_PORT_ASSIGN)
+ Release port from PF (*FPGA_FME_PORT_RELEASE)
+ Program bitstream (FPGA_FME_PORT_PR)
+
+*FPGA_FME_PORT_ASSIGN/RELEASE are only used for FPGA virtualization. Please
+refer to later section "FPGA Virtualization - PCIe" for more details.
+
+More functions are exposed through sysfs
+(/sys/class/fpga_region/regionX/dfl-fme.n/):
+
+ Read bitstream ID (bitstream_id)
+     bitstream_id indicates version of the blue bitstream (static FPGA region).
+
+ Read bitstream metadata (bitstream_metadata)
+     bistream_metadata includes more detailed information of the blue bitstream
+     (static FPGA region), e.g synthesis date and seed.
+
+ Read number of ports (ports_num)
+     one FPGA device may have more than 1 port, this sysfs interface indicates
+     how many ports the FPGA device has.
+
+ Read socket ID (socket_id)
+     socket_id is only used in integrated solution, to indicate which socket
+     the FPGA device belongs to.

Integrated?  Do you mean discrete FPGAs?  Confused here, sorry.  I
don't see this implemented in the patchset, so I'm not clear what this
will look like.  Should add this later when it is implemented.  Of
course, since you've mentioned this here, we could discuss it.

Sorry for the confusion. Will remove it and add it later with implementation.
It's the in-package FPGA on server platform, e.g Xeon + FPGA. socket_id is a
read-only value from the hardware register, it indicates which socket the FPGA
belongs to. e.g 1 server may have 2 sockets, each socket has one integrated FPGA
device. Userspace could use this information to distinguish the FPGA devices in
the same server platform and choose resources (e.g CPU, memory) closer to the
dedicated FPGA for lower latency and better performance.

quoted

+
+ Read performance counters (iperf/ and dperf/)
+ Power management (power_mgmt/)
+ Thermal management (thermal_mgmt/)
+ Error reporting (errors/)

These aren't implemented here, shouldn't be documented here.

Sure.

quoted

+
+FIU - PORT
+==========
+A port represents the interface between the static FPGA fabric (the "blue
+bitstream") and a partially reconfigurable region containing an AFU (the "green
+bitstream"). It controls the communication from SW to the accelerator and
+exposes features such as reset and debug. Each FPGA device may have more than
+1 port.

I suggest adding that there is always one port per PR region (or AFU).

Sure.

quoted

+
+AFU
+===
+An AFU is attached to a port FIU and exposes a fixed length MMIO region to be
+used for accelerator-specific control registers.
+
+User-space applications can acquire exclusive access to an AFU attached to a
+port by using open() on the port device node, and release it using close().
+
+The following functions are exposed through ioctls:
+
+ Get driver API version (FPGA_GET_API_VERSION)
+ Check for extensions (FPGA_CHECK_EXTENSION)
+ Get port info (FPGA_PORT_GET_INFO)
+ Get MMIO region info (FPGA_PORT_GET_REGION_INFO)
+ Map DMA buffer (FPGA_PORT_DMA_MAP)
+ Unmap DMA buffer (FPGA_PORT_DMA_UNMAP)
+ Reset AFU (*FPGA_PORT_RESET)
+ Enable UMsg (FPGA_PORT_UMSG_ENABLE)
+ Disable UMsg (FPGA_PORT_UMSG_DISABLE)
+ Set UMsg mode (FPGA_PORT_UMSG_SET_MODE)
+ Set UMsg base address (FPGA_PORT_UMSG_SET_BASE_ADDR)

I think the last four aren't implemented in this patchset.  Generally
it's best if documentation covers what is currently implemented.  With
lists like this, you can add to the list as your future patchsets add
the functionality.

Sure. will remove the last 4 items for now.

quoted

+
+*FPGA_PORT_RESET: reset the FPGA Port and its AFU. Userspace can do Port reset
+at any time, e.g during DMA or Partial Reconfiguration. But it should never
+cause any system level issue, only functional failure (e.g DMA or PR operation
+failure) and be recoverable from the failure.

Thanks for noting this.

quoted

+
+User-space applications can also mmap() accelerator MMIO regions.
+
+More functions are exposed through sysfs:
+(/sys/class/fpga_region/<regionX>/<dfl-port.m>/):
+
+ Read Accelerator GUID (afu_id)
+ Error reporting (errors/)
+
+DFL Framework Overview
+======================
+
+         +----------+    +--------+ +--------+ +--------+
+         |   FME    |    |  AFU   | |  AFU   | |  AFU   |
+         |  Module  |   | Module | | Module | | Module |
+         +----------+    +--------+ +--------+ +--------+
+                 +-----------------------+
+                 | FPGA Container Device |    Device Feature List
+                 |  (FPGA Base Region)   |         Framework
+                 +-----------------------+
+--------------------------------------------------------------------
+               +----------------------------+
+               |   FPGA Bus Device Module   |
+               | (e.g PCIE/Platform Device) |
+               +----------------------------+

This name almost sounds like it's adding a FPGA Bus.  How about DFL
Device Module?

Sounds good. will update it.

quoted

+                 +------------------------+
+                 |  FPGA Hardware Device  |
+                 +------------------------+
+
+DFL Framework in kernel provides common interfaces to create container device
+(FPGA base region), discover feature devices and their private features from the
+given Device Feature Lists, and create platform devices for feature devices
+(e.g FME, Port and AFU) with related resources under the container device. It
+also abstracts operations for the private features and exposes common ops to
+feature device drivers.
+
+The FPGA Bus Device could be different devices e.g PCIe device, platform device
+and etc. Its driver is always loaded first once the device is detected on its
+own bus. This driver plays an infrastructural role in the driver architecture.
+It locates the DFLs in the device memory, handles them and related resources
+to common interfaces from DFL framework for enumeration. (Please refer to
+drivers/fpga/dfl.c for detailed enumeration APIs).
+
+The FPGA Management Engine (FME) driver is a platform driver which is loaded
+automatically after FME platform device creation from the PCIE driver.

Perhaps "...from the DFL device module" would be more genereric.

Yes, agree. will fix it.

quoted

It
+provides the key features for FPGA management, including:
+
+       a) Power and thermal management, error reporting, performance reporting
+          and other infrastructure functions. Users can access these functions
+          via sysfs interfaces exposed by FME driver.

You should add this when it's implemented.

Sure.

quoted

+       b) Partial Reconfiguration. The FME driver creates FPGA manager, FPGA
+          bridges and FPGA regions during PR sub feature initialization; Once
+          it receives an FPGA_FME_PORT_PR ioctl from user, it invokes the
+          common interface function from FPGA Region to complete the partial
+          reconfiguration of the bitstream to the given port.
+       c) Port management for virtualization. The FME driver introduces two
+          ioctls, FPGA_FME_PORT_RELEASE (releases given port from PF) and
+          FPGA_FME_PORT_ASSIGN (assigns the port back to PF). Once the port is
+          released from the PF, it can be assigned to the VF through the SRIOV
+          interfaces provided by PCIE driver. (Refer to "FPGA virtualization"
+          for more details).
+
+Similar to the the FME driver, the FPGA Accelerated Function Unit (AFU) driver
+is probed once the AFU platform device is created. The main function of this
+module is to provide an interface for userspace applications to access the
+individual accelerators, including basic reset control on port, AFU MMIO region
+export, dma buffer mapping service, UMsg notification, and remote debug
+functions (see above).
+
+After feature platform devices creation, matched platform drivers will be loaded
+automatically to handle different functionalities. Please refer to next sections
+for detailed information on functional units which has been already implemented
+under this DFL framework.
+
+Partial Reconfiguration
+=======================
+As mentioned above, accelerators can be reconfigured through partial
+reconfiguration of a green bitstream file (GBS). The green bitstream must have
+been generated for the exact blue bitstream and targeted reconfigurable region
+(port) of the FPGA; otherwise, the reconfiguration operation will fail and
+possibly cause system instability. This compatibility can be checked by
+comparing the interface ID noted in the GBS header against the interface ID
+exposed by the FME through sysfs (see above).

I think the method of checking compatibility changed and isn't
interface_id anymore.

Sorry, will update it.

quoted

This check is usually done by
+user-space before calling the reconfiguration IOCTL.
+
+FPGA virtualization - PCIe
+==========================

This section can be added when virtualization support is added.

Sure.

Thanks
Hao

quoted

+This section describes the virtualization support on DFL based FPGA device to
+enable accessing an accelerator from applications running in a virtual machine
+(VM). This section only describes the PCIe based FPGA device with SRIOV support.
+
+Features supported by the particular FPGA device are exposed through Device
+Feature Lists, as illustrated below:
+
+  +-------------------------------+  +-------------+
+  |              PF               |  |     VF      |
+  +-------------------------------+  +-------------+
+      ^            ^         ^              ^
+      |            |         |              |
++-----|------------|---------|--------------|-------+
+|     |            |         |              |       |
+|  +-----+     +-------+ +-------+      +-------+   |
+|  | FME |     | Port0 | | Port1 |      | Port2 |   |
+|  +-----+     +-------+ +-------+      +-------+   |
+|                  ^         ^              ^       |
+|                  |         |              |       |
+|              +-------+ +------+       +-------+   |
+|              |  AFU  | |  AFU |       |  AFU  |   |
+|              +-------+ +------+       +-------+   |
+|                                                   |
+|                 FPGA PCIe Device                  |
++---------------------------------------------------+
+
+FME is always accessed through the physical function (PF).
+
+Ports (and related AFUs) are accessed via PF by default, but could be exposed
+through virtual function (VF) devices via PCIe SRIOV. Each VF only contains
+1 Port and 1 AFU for isolation. Users could assign individual VFs (accelerators)
+created via PCIe SRIOV interface, to virtual machines.
+
+The driver organization in virtualization case is illustrated below:
+
+  +-------++------++------+             |
+  | FME   || FME  || FME  |             |
+  | FPGA  || FPGA || FPGA |             |
+  |Manager||Bridge||Region|             |
+  +-------++------++------+             |
+  +-----------------------+  +--------+ |             +--------+
+  |          FME          |  |  AFU   | |             |  AFU   |
+  |         Module        |  | Module | |             | Module |
+  +-----------------------+  +--------+ |             +--------+
+        +-----------------------+       |       +-----------------------+
+        | FPGA Container Device |       |       | FPGA Container Device |
+        |  (FPGA Base Region)   |       |       |  (FPGA Base Region)   |
+        +-----------------------+       |       +-----------------------+
+          +------------------+          |         +------------------+
+          | FPGA PCIE Module |          | Virtual | FPGA PCIE Module |
+          +------------------+   Host   | Machine +------------------+
+ -------------------------------------- | ------------------------------
+           +---------------+            |          +---------------+
+           | PCI PF Device |            |          | PCI VF Device |
+           +---------------+            |          +---------------+
+
+FPGA PCIe device driver is always loaded first once a FPGA PCIe PF or VF device
+is detected. It:
+
+       a) finish enumeration on both FPGA PCIe PF and VF device using common
+          interfaces from DFL framework.
+       b) supports SRIOV.
+
+The FME device driver plays a management role in this driver architecture, it
+provides ioctls to release Port from PF and assign Port to PF. After release
+a port from PF, then it's safe to expose this port through a VF via PCIe SRIOV
+sysfs interface.
+
+To enable accessing an accelerator from applications running in a VM, the
+respective AFU's port needs to be assigned to a VF using the following steps:
+
+       a) The PF owns all AFU ports by default. Any port that needs to be
+          reassigned to a VF must first be released through the
+          FPGA_FME_PORT_RELEASE ioctl on the FME device.
+
+       b) Once N ports are released from PF, then user can use command below
+          to enable SRIOV and VFs. Each VF owns only one Port with AFU.
+
+          echo N > $PCI_DEVICE_PATH/sriov_numvfs
+
+       c) Pass through the VFs to VMs
+
+       d) The AFU under VF is accessible from applications in VM (using the
+          same driver inside the VF).
+
+Note that an FME can't be assigned to a VF, thus PR and other management
+functions are only available via the PF.
+
+Device enumeration
+==================
+This section introduces how applications enumerate the fpga device from
+the sysfs hierarchy under /sys/class/fpga_region.
+
+In the example below, two DFL based FPGA devices are installed in the host. Each
+fpga device has one FME and two ports (AFUs).
+
+FPGA regions are created under /sys/class/fpga_region/
+
+       /sys/class/fpga_region/region0
+       /sys/class/fpga_region/region1
+       /sys/class/fpga_region/region2
+       ...
+
+Application needs to search each regionX folder, if feature device is found,
+(e.g "dfl-port.n" or "dfl-fme.m" is found), then it's the base
+fpga region which represents the FPGA device.
+
+Each base region has one FME and two ports (AFUs) as child devices:
+
+       /sys/class/fpga_region/region0/dfl-fme.0
+       /sys/class/fpga_region/region0/dfl-port.0
+       /sys/class/fpga_region/region0/dfl-port.1
+       ...
+
+       /sys/class/fpga_region/region3/dfl-fme.1
+       /sys/class/fpga_region/region3/dfl-port.2
+       /sys/class/fpga_region/region3/dfl-port.3
+       ...
+
+In general, the FME/AFU sysfs interfaces are named as follows:
+
+       /sys/class/fpga_region/<regionX>/<dfl-fme.n>/
+       /sys/class/fpga_region/<regionX>/<dfl-port.m>/
+
+with 'n' consecutively numbering all FMEs and 'm' consecutively numbering all
+ports.
+
+The device nodes used for ioctl() or mmap() can be referenced through:
+
+       /sys/class/fpga_region/<regionX>/<dfl-fme.n>/dev
+       /sys/class/fpga_region/<regionX>/<dfl-port.n>/dev
+
+Add new FIUs support
+====================
+It's possible that developers made some new function blocks (FIUs) under this
+DFL framework, then new platform device driver needs to be developed for the
+new feature dev (FIU) following the same way as existing feature dev drivers
+(e.g FME and Port/AFU platform device driver). Besides that, it requires
+modification on DFL framework enumeration code too, for new FIU type detection
+and related platform devices creation.
+
+Add new private features support
+================================
+In some cases, we may need to add some new private features to existing FIUs
+(e.g FME or Port). Developers don't need to touch enumeration code in DFL
+framework, as each private feature will be parsed automatically, and related
+mmio resources can be found under FIU platform device created by DFL framework.
+Developer only needs to provide a sub feature driver with matched feature id.
+FME Partial Reconfiguration Sub Feature driver (see drivers/fpga/dfl-fme-pr.c)
+could be a reference.
+
+Open discussion
+===============
+FME driver exports one ioctl (FPGA_FME_PORT_PR) for partial reconfiguration to
+user now. In the future, if unified user interfaces for reconfiguration are
+added, FME driver should switch to them from ioctl interface.
--
2.7.4

Re: [PATCH v4 13/24] fpga: region: add compat_id support

From: Alan Tull <atull@kernel.org>
Date: 2018-02-28 22:55:58

On Tue, Feb 13, 2018 at 3:24 AM, Wu Hao [off-list ref] wrote:

Hi Hao,

quoted hunk

This patch introduces a compat_id member and sysfs interface for each
fpga-region, e.g userspace applications could read the compat_id
from the sysfs interface for compatibility checking before PR.

Signed-off-by: Wu Hao <redacted>
---
 Documentation/ABI/testing/sysfs-class-fpga-region |  5 +++++
 drivers/fpga/fpga-region.c                        | 19 +++++++++++++++++++
 include/linux/fpga/fpga-region.h                  | 13 +++++++++++++
 3 files changed, 37 insertions(+)
 create mode 100644 Documentation/ABI/testing/sysfs-class-fpga-region

diff --git a/Documentation/ABI/testing/sysfs-class-fpga-region b/Documentation/ABI/testing/sysfs-class-fpga-region
new file mode 100644
index 0000000..419d930
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-class-fpga-region

@@ -0,0 +1,5 @@
+What:          /sys/class/fpga_region/<region>/compat_id
+Date:          February 2018
+KernelVersion: 4.16
+Contact:       Wu Hao <hao.wu@intel.com>
+Description:   FPGA region id for compatibility check.

diff --git a/drivers/fpga/fpga-region.c b/drivers/fpga/fpga-region.c
index 660a91b..babec96 100644
--- a/drivers/fpga/fpga-region.c
+++ b/drivers/fpga/fpga-region.c

@@ -162,6 +162,24 @@ int fpga_region_program_fpga(struct fpga_region *region)
 }
 EXPORT_SYMBOL_GPL(fpga_region_program_fpga);

+static ssize_t compat_id_show(struct device *dev,
+                             struct device_attribute *attr, char *buf)
+{
+       struct fpga_region *region = to_fpga_region(dev);

This looks good, but not all users of FPGA are going to use compat_id.
How would you feel about making it a pointer in struct fpga_region?
With compat_id as a pointer, could check for non-null compat_id
pointer and return an error if it wasn't initialized.

quoted hunk

+
+       return sprintf(buf, "%016llx%016llx\n",
+                      (unsigned long long)region->compat_id.id_h,
+                      (unsigned long long)region->compat_id.id_l);
+}
+
+static DEVICE_ATTR_RO(compat_id);
+
+static struct attribute *fpga_region_attrs[] = {
+       &dev_attr_compat_id.attr,
+       NULL,
+};
+ATTRIBUTE_GROUPS(fpga_region);
+
 int fpga_region_register(struct fpga_region *region)
 {
        struct device *dev = region->parent;

@@ -226,6 +244,7 @@ static int __init fpga_region_init(void)
        if (IS_ERR(fpga_region_class))
                return PTR_ERR(fpga_region_class);

+       fpga_region_class->dev_groups = fpga_region_groups;
        fpga_region_class->dev_release = fpga_region_dev_release;

        return 0;

diff --git a/include/linux/fpga/fpga-region.h b/include/linux/fpga/fpga-region.h
index 423c87e..bf97dcc 100644
--- a/include/linux/fpga/fpga-region.h
+++ b/include/linux/fpga/fpga-region.h

@@ -6,6 +6,17 @@
 #include <linux/fpga/fpga-bridge.h>

 /**
+ * struct fpga_region_compat_id - FPGA Region id for compatibility check
+ *
+ * @id_h: high 64bit of the compat_id
+ * @id_l: low 64bit of the compat_id
+ */
+struct fpga_region_compat_id {
+       u64 id_h;
+       u64 id_l;

I guess each user will choose how to define these bits.

quoted hunk

+};
+
+/**
  * struct fpga_region - FPGA Region structure
  * @dev: FPGA Region device
  * @parent: parent device

@@ -13,6 +24,7 @@
  * @bridge_list: list of FPGA bridges specified in region
  * @mgr: FPGA manager
  * @info: FPGA image info
+ * @compat_id: FPGA region id for compatibility check.
  * @priv: private data
  * @get_bridges: optional function to get bridges to a list
  * @groups: optional attribute groups.
@@ -24,6 +36,7 @@ struct fpga_region {
        struct list_head bridge_list;
        struct fpga_manager *mgr;
        struct fpga_image_info *info;
+       struct fpga_region_compat_id compat_id;

Here it would be a pointer instead.

Alan

        void *priv;
        int (*get_bridges)(struct fpga_region *region);
        const struct attribute_group **groups;
--
2.7.4

Re: [PATCH v4 15/24] fpga: dfl-fme-pr: add compat_id support for dfl-fme-region platform device.

From: Alan Tull <atull@kernel.org>
Date: 2018-02-28 23:07:40

On Tue, Feb 13, 2018 at 3:24 AM, Wu Hao [off-list ref] wrote:

Hi Hao,

quoted hunk

This patch adds compat_id support when driver creates the platform
device for dfl-fme-region. It allows dfl-fme-region platform driver
to create fpga-region with correct compat_id.

Signed-off-by: Wu Hao <redacted>
---
 drivers/fpga/dfl-fme-pr.c | 20 ++++++++++++++++++++
 drivers/fpga/dfl-fme-pr.h |  3 +++
 2 files changed, 23 insertions(+)

diff --git a/drivers/fpga/dfl-fme-pr.c b/drivers/fpga/dfl-fme-pr.c
index 526e90b..c17170b 100644
--- a/drivers/fpga/dfl-fme-pr.c
+++ b/drivers/fpga/dfl-fme-pr.c

@@ -314,6 +314,25 @@ static void fpga_fme_destroy_bridges(struct feature_platform_data *pdata)
 }

 /**
+ * fpga_fme_get_region_compat_id - read region compat_id from hardware
+ *
+ * @dev: fme device.
+ */
+static struct fpga_region_compat_id
+fpga_fme_get_region_compat_id(struct device *dev)
+{
+       struct fpga_region_compat_id compat_id;
+       void __iomem *fme_pr;
+
+       fme_pr = get_feature_ioaddr_by_id(dev, FME_FEATURE_ID_PR_MGMT);
+
+       compat_id.id_l = readq(fme_pr + FME_PR_INTFC_ID_L);
+       compat_id.id_h = readq(fme_pr + FME_PR_INTFC_ID_H);

I tried building for 32 bit ARM and ran into readq and writeq not
being defined.  v2 had proper '#indef readq/writeq', so looks like we
need that after all.

Alan

Re: [PATCH v4 15/24] fpga: dfl-fme-pr: add compat_id support for dfl-fme-region platform device.

From: Wu Hao <hidden>
Date: 2018-03-01 05:59:23

On Wed, Feb 28, 2018 at 05:06:57PM -0600, Alan Tull wrote:

On Tue, Feb 13, 2018 at 3:24 AM, Wu Hao [off-list ref] wrote:

Hi Hao,

quoted

This patch adds compat_id support when driver creates the platform
device for dfl-fme-region. It allows dfl-fme-region platform driver
to create fpga-region with correct compat_id.

Signed-off-by: Wu Hao <redacted>
---
 drivers/fpga/dfl-fme-pr.c | 20 ++++++++++++++++++++
 drivers/fpga/dfl-fme-pr.h |  3 +++
 2 files changed, 23 insertions(+)

diff --git a/drivers/fpga/dfl-fme-pr.c b/drivers/fpga/dfl-fme-pr.c
index 526e90b..c17170b 100644
--- a/drivers/fpga/dfl-fme-pr.c
+++ b/drivers/fpga/dfl-fme-pr.c

@@ -314,6 +314,25 @@ static void fpga_fme_destroy_bridges(struct feature_platform_data *pdata)
 }

 /**
+ * fpga_fme_get_region_compat_id - read region compat_id from hardware
+ *
+ * @dev: fme device.
+ */
+static struct fpga_region_compat_id
+fpga_fme_get_region_compat_id(struct device *dev)
+{
+       struct fpga_region_compat_id compat_id;
+       void __iomem *fme_pr;
+
+       fme_pr = get_feature_ioaddr_by_id(dev, FME_FEATURE_ID_PR_MGMT);
+
+       compat_id.id_l = readq(fme_pr + FME_PR_INTFC_ID_L);
+       compat_id.id_h = readq(fme_pr + FME_PR_INTFC_ID_H);

I tried building for 32 bit ARM and ran into readq and writeq not
being defined.  v2 had proper '#indef readq/writeq', so looks like we
need that after all.

Sorry, it's missing a header file for dfl-fme-pr.h, as some files don't include
dfl.h at this version. That header file has definitions for readq/writeq.

+#include <linux/io-64-nonatomic-lo-hi.h>

will fix it.

Thanks
Hao

Alan

Re: [PATCH v4 13/24] fpga: region: add compat_id support

From: Wu Hao <hidden>
Date: 2018-03-01 06:27:34

On Wed, Feb 28, 2018 at 04:55:15PM -0600, Alan Tull wrote:

On Tue, Feb 13, 2018 at 3:24 AM, Wu Hao [off-list ref] wrote:

Hi Hao,

Hi Alan,

Thanks for the review.

quoted

This patch introduces a compat_id member and sysfs interface for each
fpga-region, e.g userspace applications could read the compat_id
from the sysfs interface for compatibility checking before PR.

Signed-off-by: Wu Hao <redacted>
---
 Documentation/ABI/testing/sysfs-class-fpga-region |  5 +++++
 drivers/fpga/fpga-region.c                        | 19 +++++++++++++++++++
 include/linux/fpga/fpga-region.h                  | 13 +++++++++++++
 3 files changed, 37 insertions(+)
 create mode 100644 Documentation/ABI/testing/sysfs-class-fpga-region

diff --git a/Documentation/ABI/testing/sysfs-class-fpga-region b/Documentation/ABI/testing/sysfs-class-fpga-region
new file mode 100644
index 0000000..419d930
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-class-fpga-region

@@ -0,0 +1,5 @@
+What:          /sys/class/fpga_region/<region>/compat_id
+Date:          February 2018
+KernelVersion: 4.16
+Contact:       Wu Hao <hao.wu@intel.com>
+Description:   FPGA region id for compatibility check.

diff --git a/drivers/fpga/fpga-region.c b/drivers/fpga/fpga-region.c
index 660a91b..babec96 100644
--- a/drivers/fpga/fpga-region.c
+++ b/drivers/fpga/fpga-region.c

@@ -162,6 +162,24 @@ int fpga_region_program_fpga(struct fpga_region *region)
 }
 EXPORT_SYMBOL_GPL(fpga_region_program_fpga);

+static ssize_t compat_id_show(struct device *dev,
+                             struct device_attribute *attr, char *buf)
+{
+       struct fpga_region *region = to_fpga_region(dev);

This looks good, but not all users of FPGA are going to use compat_id.
How would you feel about making it a pointer in struct fpga_region?
With compat_id as a pointer, could check for non-null compat_id
pointer and return an error if it wasn't initialized.

It sounds good to me.

if (!region->compat_id)
	return -ENOENT;

quoted

+
+       return sprintf(buf, "%016llx%016llx\n",
+                      (unsigned long long)region->compat_id.id_h,
+                      (unsigned long long)region->compat_id.id_l);
+}
+
+static DEVICE_ATTR_RO(compat_id);
+
+static struct attribute *fpga_region_attrs[] = {
+       &dev_attr_compat_id.attr,
+       NULL,
+};
+ATTRIBUTE_GROUPS(fpga_region);
+
 int fpga_region_register(struct fpga_region *region)
 {
        struct device *dev = region->parent;

@@ -226,6 +244,7 @@ static int __init fpga_region_init(void)
        if (IS_ERR(fpga_region_class))
                return PTR_ERR(fpga_region_class);

+       fpga_region_class->dev_groups = fpga_region_groups;
        fpga_region_class->dev_release = fpga_region_dev_release;

        return 0;

diff --git a/include/linux/fpga/fpga-region.h b/include/linux/fpga/fpga-region.h
index 423c87e..bf97dcc 100644
--- a/include/linux/fpga/fpga-region.h
+++ b/include/linux/fpga/fpga-region.h

@@ -6,6 +6,17 @@
 #include <linux/fpga/fpga-bridge.h>

 /**
+ * struct fpga_region_compat_id - FPGA Region id for compatibility check
+ *
+ * @id_h: high 64bit of the compat_id
+ * @id_l: low 64bit of the compat_id
+ */
+struct fpga_region_compat_id {
+       u64 id_h;
+       u64 id_l;

I guess each user will choose how to define these bits.

Yes.

quoted

+};
+
+/**
  * struct fpga_region - FPGA Region structure
  * @dev: FPGA Region device
  * @parent: parent device

@@ -13,6 +24,7 @@
  * @bridge_list: list of FPGA bridges specified in region
  * @mgr: FPGA manager
  * @info: FPGA image info
+ * @compat_id: FPGA region id for compatibility check.
  * @priv: private data
  * @get_bridges: optional function to get bridges to a list
  * @groups: optional attribute groups.
@@ -24,6 +36,7 @@ struct fpga_region {
        struct list_head bridge_list;
        struct fpga_manager *mgr;
        struct fpga_image_info *info;
+       struct fpga_region_compat_id compat_id;

Here it would be a pointer instead.

Yes. Will update this patch.

Thanks
Hao

Alan

quoted

        void *priv;
        int (*get_bridges)(struct fpga_region *region);
        const struct attribute_group **groups;
--
2.7.4

Re: [PATCH v4 15/24] fpga: dfl-fme-pr: add compat_id support for dfl-fme-region platform device.

From: Alan Tull <atull@kernel.org>
Date: 2018-03-01 15:59:59

On Wed, Feb 28, 2018 at 11:49 PM, Wu Hao [off-list ref] wrote:

On Wed, Feb 28, 2018 at 05:06:57PM -0600, Alan Tull wrote:

quoted

On Tue, Feb 13, 2018 at 3:24 AM, Wu Hao [off-list ref] wrote:

quoted

+
+       compat_id.id_l = readq(fme_pr + FME_PR_INTFC_ID_L);
+       compat_id.id_h = readq(fme_pr + FME_PR_INTFC_ID_H);

I tried building for 32 bit ARM and ran into readq and writeq not
being defined.  v2 had proper '#indef readq/writeq', so looks like we
need that after all.

Sorry, it's missing a header file for dfl-fme-pr.h, as some files don't include
dfl.h at this version. That header file has definitions for readq/writeq.

+#include <linux/io-64-nonatomic-lo-hi.h>

will fix it.

Hi Hao,

That sounds good.  I was able to build if I added it.  There are a
couple issues in the enumeration code:

In file included from
/home/atull/repos/linux-socfpga/include/linux/kernel.h:11:0,
                 from /home/atull/repos/linux-socfpga/include/linux/list.h:9,
                 from /home/atull/repos/linux-socfpga/include/linux/module.h:9,
                 from /home/atull/repos/linux-socfpga/drivers/fpga/dfl.c:13:
/home/atull/repos/linux-socfpga/drivers/fpga/dfl.c: In function
'parse_feature_list':
/home/atull/repos/linux-socfpga/include/linux/bitops.h:7:24: warning:
left shift count >= width of type [-Wshift-count-overflow]
 #define BIT(nr)   (1UL << (nr))
                        ^
/home/atull/repos/linux-socfpga/drivers/fpga/dfl.h:73:19: note: in
expansion of macro 'BIT'
 #define DFH_EOL   BIT(40)   /* End of list */
                   ^
/home/atull/repos/linux-socfpga/drivers/fpga/dfl.c:643:12: note: in
expansion of macro 'DFH_EOL'
   if ((v & DFH_EOL) || !ofst)
            ^

and

  CC      drivers/fpga/dfl-fme-mgr.o
In file included from
/home/atull/repos/linux-socfpga/include/linux/kernel.h:11:0,
                 from /home/atull/repos/linux-socfpga/include/linux/list.h:9,
                 from
/home/atull/repos/linux-socfpga/include/linux/kobject.h:19,
                 from /home/atull/repos/linux-socfpga/include/linux/device.h:16,
                 from
/home/atull/repos/linux-socfpga/drivers/fpga/dfl-fme-pr.c:20:
/home/atull/repos/linux-socfpga/drivers/fpga/dfl-fme-pr.c: In function
'pr_mgmt_init':
/home/atull/repos/linux-socfpga/include/linux/bitops.h:7:24: warning:
left shift count >= width of type [-Wshift-count-overflow]
 #define BIT(nr)   (1UL << (nr))
                        ^
/home/atull/repos/linux-socfpga/drivers/fpga/dfl.h:113:27: note: in
expansion of macro 'BIT'
 #define FME_PORT_OFST_IMP BIT(60)
                           ^
/home/atull/repos/linux-socfpga/drivers/fpga/dfl-fme-pr.c:449:23:
note: in expansion of macro 'FME_PORT_OFST_IMP'
   if (!(port_offset & FME_PORT_OFST_IMP))
                       ^
Alan

Thanks
Hao

quoted

Alan

Re: [PATCH v4 15/24] fpga: dfl-fme-pr: add compat_id support for dfl-fme-region platform device.

From: Wu Hao <hidden>
Date: 2018-03-01 16:04:52

On Thu, Mar 01, 2018 at 09:59:16AM -0600, Alan Tull wrote:

On Wed, Feb 28, 2018 at 11:49 PM, Wu Hao [off-list ref] wrote:

quoted

On Wed, Feb 28, 2018 at 05:06:57PM -0600, Alan Tull wrote:

quoted

On Tue, Feb 13, 2018 at 3:24 AM, Wu Hao [off-list ref] wrote:

quoted

+
+       compat_id.id_l = readq(fme_pr + FME_PR_INTFC_ID_L);
+       compat_id.id_h = readq(fme_pr + FME_PR_INTFC_ID_H);

I tried building for 32 bit ARM and ran into readq and writeq not
being defined.  v2 had proper '#indef readq/writeq', so looks like we
need that after all.

Sorry, it's missing a header file for dfl-fme-pr.h, as some files don't include
dfl.h at this version. That header file has definitions for readq/writeq.

+#include <linux/io-64-nonatomic-lo-hi.h>

will fix it.

Hi Hao,

That sounds good.  I was able to build if I added it.  There are a
couple issues in the enumeration code:

In file included from
/home/atull/repos/linux-socfpga/include/linux/kernel.h:11:0,
                 from /home/atull/repos/linux-socfpga/include/linux/list.h:9,
                 from /home/atull/repos/linux-socfpga/include/linux/module.h:9,
                 from /home/atull/repos/linux-socfpga/drivers/fpga/dfl.c:13:
/home/atull/repos/linux-socfpga/drivers/fpga/dfl.c: In function
'parse_feature_list':
/home/atull/repos/linux-socfpga/include/linux/bitops.h:7:24: warning:
left shift count >= width of type [-Wshift-count-overflow]
 #define BIT(nr)   (1UL << (nr))
                        ^
/home/atull/repos/linux-socfpga/drivers/fpga/dfl.h:73:19: note: in
expansion of macro 'BIT'
 #define DFH_EOL   BIT(40)   /* End of list */
                   ^
/home/atull/repos/linux-socfpga/drivers/fpga/dfl.c:643:12: note: in
expansion of macro 'DFH_EOL'
   if ((v & DFH_EOL) || !ofst)
            ^

and

  CC      drivers/fpga/dfl-fme-mgr.o
In file included from
/home/atull/repos/linux-socfpga/include/linux/kernel.h:11:0,
                 from /home/atull/repos/linux-socfpga/include/linux/list.h:9,
                 from
/home/atull/repos/linux-socfpga/include/linux/kobject.h:19,
                 from /home/atull/repos/linux-socfpga/include/linux/device.h:16,
                 from
/home/atull/repos/linux-socfpga/drivers/fpga/dfl-fme-pr.c:20:
/home/atull/repos/linux-socfpga/drivers/fpga/dfl-fme-pr.c: In function
'pr_mgmt_init':
/home/atull/repos/linux-socfpga/include/linux/bitops.h:7:24: warning:
left shift count >= width of type [-Wshift-count-overflow]
 #define BIT(nr)   (1UL << (nr))
                        ^
/home/atull/repos/linux-socfpga/drivers/fpga/dfl.h:113:27: note: in
expansion of macro 'BIT'
 #define FME_PORT_OFST_IMP BIT(60)
                           ^
/home/atull/repos/linux-socfpga/drivers/fpga/dfl-fme-pr.c:449:23:
note: in expansion of macro 'FME_PORT_OFST_IMP'
   if (!(port_offset & FME_PORT_OFST_IMP))

Hi Alan

I think it needs to use BIT_ULL instead of BIT.
will fix them in the next version.

Thanks
Hao

                       ^
Alan

quoted

Thanks
Hao

quoted

Alan

Re: [PATCH v4 13/24] fpga: region: add compat_id support

From: Alan Tull <atull@kernel.org>
Date: 2018-03-05 19:43:24

On Thu, Mar 1, 2018 at 12:17 AM, Wu Hao [off-list ref] wrote:

On Wed, Feb 28, 2018 at 04:55:15PM -0600, Alan Tull wrote:

quoted

On Tue, Feb 13, 2018 at 3:24 AM, Wu Hao [off-list ref] wrote:

Hi Hao,

Hi Alan,

Thanks for the review.

quoted

This patch introduces a compat_id member and sysfs interface for each
fpga-region, e.g userspace applications could read the compat_id
from the sysfs interface for compatibility checking before PR.

Signed-off-by: Wu Hao <redacted>
---
 Documentation/ABI/testing/sysfs-class-fpga-region |  5 +++++
 drivers/fpga/fpga-region.c                        | 19 +++++++++++++++++++
 include/linux/fpga/fpga-region.h                  | 13 +++++++++++++
 3 files changed, 37 insertions(+)
 create mode 100644 Documentation/ABI/testing/sysfs-class-fpga-region

diff --git a/Documentation/ABI/testing/sysfs-class-fpga-region b/Documentation/ABI/testing/sysfs-class-fpga-region
new file mode 100644
index 0000000..419d930
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-class-fpga-region

@@ -0,0 +1,5 @@
+What:          /sys/class/fpga_region/<region>/compat_id
+Date:          February 2018
+KernelVersion: 4.16
+Contact:       Wu Hao <hao.wu@intel.com>
+Description:   FPGA region id for compatibility check.

It would be helpful to add some explanation here that although the
intended function of compat_id is set, the way the actual value is
defined or calculated is set by the layer that is creating the FPGA
region.

quoted

diff --git a/drivers/fpga/fpga-region.c b/drivers/fpga/fpga-region.c
index 660a91b..babec96 100644
--- a/drivers/fpga/fpga-region.c
+++ b/drivers/fpga/fpga-region.c

@@ -162,6 +162,24 @@ int fpga_region_program_fpga(struct fpga_region *region)
 }
 EXPORT_SYMBOL_GPL(fpga_region_program_fpga);

+static ssize_t compat_id_show(struct device *dev,
+                             struct device_attribute *attr, char *buf)
+{
+       struct fpga_region *region = to_fpga_region(dev);

This looks good, but not all users of FPGA are going to use compat_id.
How would you feel about making it a pointer in struct fpga_region?
With compat_id as a pointer, could check for non-null compat_id
pointer and return an error if it wasn't initialized.

It sounds good to me.

if (!region->compat_id)
        return -ENOENT;

Yes, thanks!

Alan

quoted

+
+       return sprintf(buf, "%016llx%016llx\n",
+                      (unsigned long long)region->compat_id.id_h,
+                      (unsigned long long)region->compat_id.id_l);
+}
+
+static DEVICE_ATTR_RO(compat_id);
+
+static struct attribute *fpga_region_attrs[] = {
+       &dev_attr_compat_id.attr,
+       NULL,
+};
+ATTRIBUTE_GROUPS(fpga_region);
+
 int fpga_region_register(struct fpga_region *region)
 {
        struct device *dev = region->parent;

@@ -226,6 +244,7 @@ static int __init fpga_region_init(void)
        if (IS_ERR(fpga_region_class))
                return PTR_ERR(fpga_region_class);

+       fpga_region_class->dev_groups = fpga_region_groups;
        fpga_region_class->dev_release = fpga_region_dev_release;

        return 0;

diff --git a/include/linux/fpga/fpga-region.h b/include/linux/fpga/fpga-region.h
index 423c87e..bf97dcc 100644
--- a/include/linux/fpga/fpga-region.h
+++ b/include/linux/fpga/fpga-region.h

@@ -6,6 +6,17 @@
 #include <linux/fpga/fpga-bridge.h>

 /**
+ * struct fpga_region_compat_id - FPGA Region id for compatibility check
+ *
+ * @id_h: high 64bit of the compat_id
+ * @id_l: low 64bit of the compat_id
+ */
+struct fpga_region_compat_id {
+       u64 id_h;
+       u64 id_l;

I guess each user will choose how to define these bits.

Yes.

quoted

+};
+
+/**
  * struct fpga_region - FPGA Region structure
  * @dev: FPGA Region device
  * @parent: parent device

@@ -13,6 +24,7 @@
  * @bridge_list: list of FPGA bridges specified in region
  * @mgr: FPGA manager
  * @info: FPGA image info
+ * @compat_id: FPGA region id for compatibility check.
  * @priv: private data
  * @get_bridges: optional function to get bridges to a list
  * @groups: optional attribute groups.
@@ -24,6 +36,7 @@ struct fpga_region {
        struct list_head bridge_list;
        struct fpga_manager *mgr;
        struct fpga_image_info *info;
+       struct fpga_region_compat_id compat_id;

Here it would be a pointer instead.

Yes. Will update this patch.

Thanks
Hao

quoted

Alan

quoted

        void *priv;
        int (*get_bridges)(struct fpga_region *region);
        const struct attribute_group **groups;
--
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v4 14/24] fpga: dfl: fme: add partial reconfiguration sub feature support

From: Alan Tull <atull@kernel.org>
Date: 2018-03-05 22:46:46

On Tue, Feb 13, 2018 at 3:24 AM, Wu Hao [off-list ref] wrote:

Hi Hao,

We are going to want to be able use different FPGA managers with this
framework.  The different manager may be part of a different FME in
fabric or it may be a hardware FPGA manager.  Fortunately, at this
point now the changes, noted below, to get there are pretty small.

quoted hunk

From: Kang Luwei <redacted>

Partial Reconfiguration (PR) is the most important function for FME. It
allows reconfiguration for given Port/Accelerated Function Unit (AFU).

It creates platform devices for fpga-mgr, fpga-regions and fpga-bridges,
and invokes fpga-region's interface (fpga_region_program_fpga) for PR
operation once PR request received via ioctl. Below user space interface
is exposed by this sub feature.

Ioctl interface:
* FPGA_FME_PORT_PR
  Do partial reconfiguration per information from userspace, including
  target port(AFU), buffer size and address info. It returns error code
  to userspace if failed. For detailed PR error information, user needs
  to read fpga-mgr's status sysfs interface.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Kang Luwei <redacted>
Signed-off-by: Xiao Guangrong <redacted>
Signed-off-by: Wu Hao <redacted>
---
v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
    switched to GPLv2 license.
    removed status from FPGA_FME_PORT_PR ioctl data structure.
    added platform devices creation for fpga-mgr/fpga-region/fpga-bridge.
    switched to fpga-region interface fpga_region_program_fpga for PR.
    fixed comments from Alan Tull on FPGA_MGR_PARTIAL_RECONFIG flag usage.
    fixed kbuild warnings.
v3: rename driver files to dfl-fme-*.
    rebase due to fpga APIs change.
    replace bitfields.
    switch to fpga_cdev_find_port to find port device.
v4: rebase and correct comments for some function.
    fix SPDX license issue.
    remove unnecessary input parameter for destroy_bridge/region function.
    add dfl-fme-pr.h for PR sub feature data structure and registers.
---
 drivers/fpga/Makefile         |   2 +-
 drivers/fpga/dfl-fme-main.c   |  45 +++-
 drivers/fpga/dfl-fme-pr.c     | 497 ++++++++++++++++++++++++++++++++++++++++++
 drivers/fpga/dfl-fme-pr.h     | 113 ++++++++++
 drivers/fpga/dfl-fme.h        |  38 ++++
 include/uapi/linux/fpga-dfl.h |  27 +++
 6 files changed, 720 insertions(+), 2 deletions(-)
 create mode 100644 drivers/fpga/dfl-fme-pr.c
 create mode 100644 drivers/fpga/dfl-fme-pr.h
 create mode 100644 drivers/fpga/dfl-fme.h

diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index fbd1c85..3c44fc9 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile

@@ -32,7 +32,7 @@ obj-$(CONFIG_OF_FPGA_REGION)          += of-fpga-region.o
 obj-$(CONFIG_FPGA_DFL)                 += dfl.o
 obj-$(CONFIG_FPGA_DFL_FME)             += dfl-fme.o

-dfl-fme-objs := dfl-fme-main.o
+dfl-fme-objs := dfl-fme-main.o dfl-fme-pr.o

 # Drivers for FPGAs which implement DFL
 obj-$(CONFIG_FPGA_DFL_PCI)             += dfl-pci.o

diff --git a/drivers/fpga/dfl-fme-main.c b/drivers/fpga/dfl-fme-main.c
index 1a9929c..967a44c 100644
--- a/drivers/fpga/dfl-fme-main.c
+++ b/drivers/fpga/dfl-fme-main.c

@@ -19,6 +19,7 @@
 #include <linux/fpga-dfl.h>

 #include "dfl.h"
+#include "dfl-fme.h"

 static ssize_t ports_num_show(struct device *dev,
                              struct device_attribute *attr, char *buf)

@@ -111,6 +112,10 @@ static struct feature_driver fme_feature_drvs[] = {
                .ops = &fme_hdr_ops,
        },
        {
+               .id = FME_FEATURE_ID_PR_MGMT,
+               .ops = &pr_mgmt_ops,
+       },
+       {
                .ops = NULL,
        },
 };

@@ -194,14 +199,49 @@ static const struct file_operations fme_fops = {
        .unlocked_ioctl = fme_ioctl,
 };

+static int fme_dev_init(struct platform_device *pdev)
+{
+       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+       struct fpga_fme *fme;
+
+       fme = devm_kzalloc(&pdev->dev, sizeof(*fme), GFP_KERNEL);
+       if (!fme)
+               return -ENOMEM;
+
+       fme->pdata = pdata;
+
+       mutex_lock(&pdata->lock);
+       fpga_pdata_set_private(pdata, fme);
+       mutex_unlock(&pdata->lock);
+
+       return 0;
+}
+
+static void fme_dev_destroy(struct platform_device *pdev)
+{
+       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+       struct fpga_fme *fme;
+
+       mutex_lock(&pdata->lock);
+       fme = fpga_pdata_get_private(pdata);
+       fpga_pdata_set_private(pdata, NULL);
+       mutex_unlock(&pdata->lock);
+
+       devm_kfree(&pdev->dev, fme);

Is this needed?  This is called when the device is being destroyed anyway.

quoted hunk

+}
+
 static int fme_probe(struct platform_device *pdev)
 {
        int ret;

-       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
+       ret = fme_dev_init(pdev);
        if (ret)
                goto exit;

+       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
+       if (ret)
+               goto dev_destroy;
+
        ret = fpga_register_dev_ops(pdev, &fme_fops, THIS_MODULE);
        if (ret)
                goto feature_uinit;

@@ -210,6 +250,8 @@ static int fme_probe(struct platform_device *pdev)

 feature_uinit:
        fpga_dev_feature_uinit(pdev);
+dev_destroy:
+       fme_dev_destroy(pdev);
 exit:
        return ret;
 }

@@ -218,6 +260,7 @@ static int fme_remove(struct platform_device *pdev)
 {
        fpga_dev_feature_uinit(pdev);
        fpga_unregister_dev_ops(pdev);
+       fme_dev_destroy(pdev);

        return 0;
 }

diff --git a/drivers/fpga/dfl-fme-pr.c b/drivers/fpga/dfl-fme-pr.c
new file mode 100644
index 0000000..526e90b
--- /dev/null
+++ b/drivers/fpga/dfl-fme-pr.c

@@ -0,0 +1,497 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Driver for FPGA Management Engine (FME) Partial Reconfiguration
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Christopher Rauer <christopher.rauer@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ */
+
+#include <linux/types.h>
+#include <linux/device.h>
+#include <linux/vmalloc.h>
+#include <linux/uaccess.h>
+#include <linux/fpga/fpga-mgr.h>
+#include <linux/fpga/fpga-bridge.h>
+#include <linux/fpga/fpga-region.h>
+#include <linux/fpga-dfl.h>
+
+#include "dfl.h"
+#include "dfl-fme.h"
+#include "dfl-fme-pr.h"
+
+static struct fme_region *
+find_fme_region_by_port_id(struct fpga_fme *fme, int port_id)
+{
+       struct fme_region *fme_region;
+
+       list_for_each_entry(fme_region, &fme->region_list, node)
+               if (fme_region->port_id == port_id)
+                       return fme_region;
+
+       return NULL;
+}
+
+static int fpga_fme_region_match(struct device *dev, const void *data)
+{
+       return dev->parent == data;
+}
+
+static struct fpga_region *
+fpga_fme_region_find(struct fpga_fme *fme, int port_id)
+{
+       struct fme_region *fme_region;
+       struct fpga_region *region;
+
+       fme_region = find_fme_region_by_port_id(fme, port_id);
+       if (!fme_region)
+               return NULL;
+
+       region = fpga_region_class_find(NULL, &fme_region->region->dev,
+                                       fpga_fme_region_match);
+       if (!region)
+               return NULL;
+
+       return region;
+}
+
+static int fme_pr(struct platform_device *pdev, unsigned long arg)
+{
+       void __user *argp = (void __user *)arg;
+       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+       struct fpga_fme *fme;
+       struct fpga_image_info *info;
+       struct fpga_region *region;
+       struct fpga_fme_port_pr port_pr;
+       unsigned long minsz;
+       void __iomem *fme_hdr;
+       void *buf = NULL;
+       int ret = 0;
+       u64 v;
+
+       minsz = offsetofend(struct fpga_fme_port_pr, buffer_address);
+
+       if (copy_from_user(&port_pr, argp, minsz))
+               return -EFAULT;
+
+       if (port_pr.argsz < minsz || port_pr.flags)
+               return -EINVAL;
+
+       if (!IS_ALIGNED(port_pr.buffer_size, 4))
+               return -EINVAL;
+
+       /* get fme header region */
+       fme_hdr = get_feature_ioaddr_by_id(&pdev->dev,
+                                          FME_FEATURE_ID_HEADER);
+
+       /* check port id */
+       v = readq(fme_hdr + FME_HDR_CAP);
+       if (port_pr.port_id >= FIELD_GET(FME_CAP_NUM_PORTS, v)) {
+               dev_dbg(&pdev->dev, "port number more than maximum\n");
+               return -EINVAL;
+       }
+
+       if (!access_ok(VERIFY_READ,
+                      (void __user *)(unsigned long)port_pr.buffer_address,
+                      port_pr.buffer_size))
+               return -EFAULT;
+
+       buf = vmalloc(port_pr.buffer_size);
+       if (!buf)
+               return -ENOMEM;
+
+       if (copy_from_user(buf,
+                          (void __user *)(unsigned long)port_pr.buffer_address,
+                          port_pr.buffer_size)) {
+               ret = -EFAULT;
+               goto free_exit;
+       }
+
+       /* prepare fpga_image_info for PR */
+       info = fpga_image_info_alloc(&pdev->dev);
+       if (!info) {
+               ret = -ENOMEM;
+               goto free_exit;
+       }
+
+       info->flags |= FPGA_MGR_PARTIAL_RECONFIG;
+
+       mutex_lock(&pdata->lock);
+       fme = fpga_pdata_get_private(pdata);
+       /* fme device has been unregistered. */
+       if (!fme) {
+               ret = -EINVAL;
+               goto unlock_exit;
+       }
+
+       region = fpga_fme_region_find(fme, port_pr.port_id);
+       if (!region) {
+               ret = -EINVAL;
+               goto unlock_exit;
+       }
+
+       fpga_image_info_free(region->info);
+
+       info->buf = buf;
+       info->count = port_pr.buffer_size;
+       info->region_id = port_pr.port_id;
+       region->info = info;
+
+       ret = fpga_region_program_fpga(region);
+
+       if (region->get_bridges)
+               fpga_bridges_put(&region->bridge_list);
+
+       put_device(&region->dev);
+unlock_exit:
+       mutex_unlock(&pdata->lock);
+free_exit:
+       vfree(buf);
+       if (copy_to_user((void __user *)arg, &port_pr, minsz))
+               return -EFAULT;
+
+       return ret;
+}
+
+/**
+ * fpga_fme_create_mgr - create fpga mgr platform device as child device
+ *
+ * @pdata: fme platform_device's pdata
+ *
+ * Return: mgr platform device if successful, and error code otherwise.
+ */
+static struct platform_device *
+fpga_fme_create_mgr(struct feature_platform_data *pdata)
+{
+       struct platform_device *mgr, *fme = pdata->dev;
+       struct feature *feature;
+       struct resource res;
+       struct resource *pres;
+       int ret = -ENOMEM;
+
+       feature = get_feature_by_id(&pdata->dev->dev, FME_FEATURE_ID_PR_MGMT);
+       if (!feature)
+               return ERR_PTR(-ENODEV);
+
+       /*
+        * Each FME has only one fpga-mgr, so allocate platform device using
+        * the same FME platform device id.
+        */
+       mgr = platform_device_alloc(FPGA_DFL_FME_MGR, fme->id);

At this point, the framework is assuming all FME's include the same
FPGA manager device which would use the driver in dfl-fme-mgr.c.

I'm thinking of two cases where the manager isn't the same as a
dfl-fme-mgr.c manager are a bit different:

(1) a FME-based FPGA manager, but different implementation, different
registers.  The constraint is that the port implementation has to be
similar enough to use the rest of the base FME code.   I am wondering
if the FPGA manager can be added to the DFL.  At this point, the DFL
would drive which FPGA manager is alloc'd.   That way the user gets to
use all this code in dfl-fme-pr.c but with their FPGA manager.

(2) a FPGA manager that can be added by device tree in the case of a
platform that is using device tree.  I think this will be pretty
simple and can be done later when someone is actually bringing this
framework up on a FPGA running under device tree.  I'm thinking that
the base DFL device that reads the dfl data from hardware can have a
DT property that points to the FPGA manager.  That manager can be
saved somewhere handy like the pdata and passed down to this code,
which realizes it can use that existing device and doesn't need to
alloc a platform device.  But again, that's probably best done later.

quoted hunk

+       if (!mgr)
+               return ERR_PTR(ret);
+
+       mgr->dev.parent = &fme->dev;
+
+       pres = platform_get_resource(fme, IORESOURCE_MEM,
+                                    feature->resource_index);
+       if (!pres) {
+               ret = -ENODEV;
+               goto create_mgr_err;
+       }
+
+       memset(&res, 0, sizeof(struct resource));
+
+       res.start = pres->start;
+       res.end = pres->end;
+       res.name = pres->name;
+       res.flags = IORESOURCE_MEM;
+
+       ret = platform_device_add_resources(mgr, &res, 1);
+       if (ret)
+               goto create_mgr_err;
+
+       ret = platform_device_add(mgr);
+       if (ret)
+               goto create_mgr_err;
+
+       return mgr;
+
+create_mgr_err:
+       platform_device_put(mgr);
+       return ERR_PTR(ret);
+}
+
+/**
+ * fpga_fme_destroy_mgr - destroy fpga mgr platform device
+ * @pdata: fme platform device's pdata
+ */
+static void fpga_fme_destroy_mgr(struct feature_platform_data *pdata)
+{
+       struct fpga_fme *priv = fpga_pdata_get_private(pdata);
+
+       platform_device_unregister(priv->mgr);
+}
+
+/**
+ * fpga_fme_create_bridge - create fme fpga bridge platform device as child
+ *
+ * @pdata: fme platform device's pdata
+ * @port_id: port id for the bridge to be created.
+ *
+ * Return: bridge platform device if successful, and error code otherwise.
+ */
+static struct fme_bridge *
+fpga_fme_create_bridge(struct feature_platform_data *pdata, int port_id)
+{
+       struct device *dev = &pdata->dev->dev;
+       struct fme_br_pdata br_pdata;
+       struct fme_bridge *fme_br;
+       int ret = -ENOMEM;
+
+       fme_br = devm_kzalloc(dev, sizeof(*fme_br), GFP_KERNEL);
+       if (!fme_br)
+               return ERR_PTR(ret);
+
+       br_pdata.port = fpga_cdev_find_port(fpga_pdata_to_fpga_cdev(pdata),
+                                           &port_id, fpga_port_check_id);
+       if (!br_pdata.port)
+               return ERR_PTR(-ENODEV);
+
+       /*
+        * Each FPGA device may have more than one port, so allocate platform
+        * device using the same port platform device id.
+        */
+       fme_br->br = platform_device_alloc(FPGA_DFL_FME_BRIDGE,
+                                          br_pdata.port->id);
+       if (!fme_br->br) {
+               ret = -ENOMEM;
+               goto create_br_err;
+       }
+
+       fme_br->br->dev.parent = dev;
+
+       ret = platform_device_add_data(fme_br->br, &br_pdata, sizeof(br_pdata));
+       if (ret)
+               goto create_br_err;
+
+       ret = platform_device_add(fme_br->br);
+       if (ret)
+               goto create_br_err;
+
+       return fme_br;
+
+create_br_err:
+       platform_device_put(fme_br->br);
+       put_device(&br_pdata.port->dev);
+       return ERR_PTR(ret);
+}
+
+/**
+ * fpga_fme_destroy_bridge - destroy fpga bridge platform device
+ * @fme_br: fme bridge to destroy
+ */
+static void fpga_fme_destroy_bridge(struct fme_bridge *fme_br)
+{
+       struct fme_br_pdata *br_pdata = dev_get_platdata(&fme_br->br->dev);
+
+       put_device(&br_pdata->port->dev);
+       platform_device_unregister(fme_br->br);
+}
+
+/**
+ * fpga_fme_destroy_bridge - destroy all fpga bridge platform device
+ * @pdata: fme platform device's pdata
+ */
+static void fpga_fme_destroy_bridges(struct feature_platform_data *pdata)
+{
+       struct fpga_fme *priv = fpga_pdata_get_private(pdata);
+       struct fme_bridge *fbridge, *tmp;
+
+       list_for_each_entry_safe(fbridge, tmp, &priv->bridge_list, node) {
+               list_del(&fbridge->node);
+               fpga_fme_destroy_bridge(fbridge);
+       }
+}
+
+/**
+ * fpga_fme_create_region - create fpga region platform device as child
+ *
+ * @pdata: fme platform device's pdata
+ * @mgr: mgr platform device needed for region
+ * @br: br platform device needed for region
+ * @port_id: port id
+ *
+ * Return: fme region if successful, and error code otherwise.
+ */
+static struct fme_region *
+fpga_fme_create_region(struct feature_platform_data *pdata,
+                      struct platform_device *mgr,
+                      struct platform_device *br, int port_id)
+{
+       struct device *dev = &pdata->dev->dev;
+       struct fme_region_pdata region_pdata;
+       struct fme_region *fme_region;
+       int ret = -ENOMEM;
+
+       fme_region = devm_kzalloc(dev, sizeof(*fme_region), GFP_KERNEL);
+       if (!fme_region)
+               return ERR_PTR(ret);
+
+       region_pdata.mgr = mgr;
+       region_pdata.br = br;
+
+       /*
+        * Each FPGA device may have more than one port, so allocate platform
+        * device using the same port platform device id.
+        */
+       fme_region->region = platform_device_alloc(FPGA_DFL_FME_REGION, br->id);
+       if (!fme_region->region)
+               return ERR_PTR(ret);
+
+       fme_region->region->dev.parent = dev;
+
+       ret = platform_device_add_data(fme_region->region, &region_pdata,
+                                      sizeof(region_pdata));
+       if (ret)
+               goto create_region_err;
+
+       ret = platform_device_add(fme_region->region);
+       if (ret)
+               goto create_region_err;
+
+       fme_region->port_id = port_id;
+
+       return fme_region;
+
+create_region_err:
+       platform_device_put(fme_region->region);
+       return ERR_PTR(ret);
+}
+
+/**
+ * fpga_fme_destroy_region - destroy fme region
+ * @fme_region: fme region to destroy
+ */
+static void fpga_fme_destroy_region(struct fme_region *fme_region)
+{
+       platform_device_unregister(fme_region->region);
+}
+
+/**
+ * fpga_fme_destroy_regions - destroy all fme regions
+ * @pdata: fme platform device's pdata
+ */
+static void fpga_fme_destroy_regions(struct feature_platform_data *pdata)
+{
+       struct fpga_fme *priv = fpga_pdata_get_private(pdata);
+       struct fme_region *fme_region, *tmp;
+
+       list_for_each_entry_safe(fme_region, tmp, &priv->region_list, node) {
+               list_del(&fme_region->node);
+               fpga_fme_destroy_region(fme_region);
+       }
+}
+
+static int pr_mgmt_init(struct platform_device *pdev, struct feature *feature)
+{
+       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+       void __iomem *fme_hdr;
+       struct platform_device *mgr;
+       struct fme_region *fme_region;
+       struct fme_bridge *fme_br;
+       struct fpga_fme *priv;
+       int ret = -ENODEV, i = 0;
+       u64 fme_cap, port_offset;
+
+       fme_hdr = get_feature_ioaddr_by_id(&pdev->dev,
+                                          FME_FEATURE_ID_HEADER);
+
+       mutex_lock(&pdata->lock);
+       priv = fpga_pdata_get_private(pdata);
+
+       /* Initialize the region and bridge sub device list */
+       INIT_LIST_HEAD(&priv->region_list);
+       INIT_LIST_HEAD(&priv->bridge_list);
+
+       /* Create fpga mgr platform device */
+       mgr = fpga_fme_create_mgr(pdata);
+       if (IS_ERR(mgr)) {
+               dev_err(&pdev->dev, "fail to create fpga mgr pdev\n");
+               goto unlock;
+       }
+
+       priv->mgr = mgr;
+
+       /* Read capability register to check number of regions and bridges */
+       fme_cap = readq(fme_hdr + FME_HDR_CAP);
+       for (; i < FIELD_GET(FME_CAP_NUM_PORTS, fme_cap); i++) {
+               port_offset = readq(fme_hdr + FME_HDR_PORT_OFST(i));
+               if (!(port_offset & FME_PORT_OFST_IMP))
+                       continue;
+
+               /* Create bridge for each port */
+               fme_br = fpga_fme_create_bridge(pdata, i);
+               if (IS_ERR(fme_br)) {
+                       ret = PTR_ERR(fme_br);
+                       goto destroy_region;
+               }
+
+               list_add(&fme_br->node, &priv->bridge_list);
+
+               /* Create region for each port */
+               fme_region = fpga_fme_create_region(pdata, mgr, fme_br->br, i);
+               if (!fme_region) {
+                       ret = PTR_ERR(fme_region);
+                       goto destroy_region;
+               }
+
+               list_add(&fme_region->node, &priv->region_list);
+       }
+       mutex_unlock(&pdata->lock);
+
+       return 0;
+
+destroy_region:
+       fpga_fme_destroy_regions(pdata);
+       fpga_fme_destroy_bridges(pdata);
+       fpga_fme_destroy_mgr(pdata);
+unlock:
+       mutex_unlock(&pdata->lock);
+       return ret;
+}
+
+static void pr_mgmt_uinit(struct platform_device *pdev, struct feature *feature)
+{
+       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+       struct fpga_fme *priv;
+
+       mutex_lock(&pdata->lock);
+       priv = fpga_pdata_get_private(pdata);
+
+       fpga_fme_destroy_regions(pdata);
+       fpga_fme_destroy_bridges(pdata);
+       fpga_fme_destroy_mgr(pdata);
+       mutex_unlock(&pdata->lock);
+}
+
+static long fme_pr_ioctl(struct platform_device *pdev, struct feature *feature,
+                        unsigned int cmd, unsigned long arg)
+{
+       long ret;
+
+       switch (cmd) {
+       case FPGA_FME_PORT_PR:
+               ret = fme_pr(pdev, arg);
+               break;
+       default:
+               ret = -ENODEV;
+       }
+
+       return ret;
+}
+
+const struct feature_ops pr_mgmt_ops = {
+       .init = pr_mgmt_init,
+       .uinit = pr_mgmt_uinit,
+       .ioctl = fme_pr_ioctl,
+};

diff --git a/drivers/fpga/dfl-fme-pr.h b/drivers/fpga/dfl-fme-pr.h
new file mode 100644
index 0000000..11bd001
--- /dev/null
+++ b/drivers/fpga/dfl-fme-pr.h

@@ -0,0 +1,113 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Header file for FPGA Management Engine (FME) Partial Reconfiguration Driver
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ */
+
+#ifndef __DFL_FME_PR_H
+#define __DFL_FME_PR_H
+
+#include <linux/platform_device.h>
+
+/**
+ * struct fme_region - FME fpga region data structure
+ *
+ * @region: platform device of the FPGA region.
+ * @node: used to link fme_region to a list.
+ * @port_id: indicate which port this region connected to.
+ */
+struct fme_region {
+       struct platform_device *region;
+       struct list_head node;
+       int port_id;
+};
+
+/**
+ * struct fme_region_pdata - platform data for FME region platform device.
+ *
+ * @mgr: platform device of the FPGA manager.
+ * @br: platform device of the FPGA bridge.
+ * @region_id: region id (same as port_id).
+ */
+struct fme_region_pdata {
+       struct platform_device *mgr;
+       struct platform_device *br;
+       int region_id;
+};
+
+/**
+ * struct fme_bridge - FME fpga bridge data structure
+ *
+ * @br: platform device of the FPGA bridge.
+ * @node: used to link fme_bridge to a list.
+ */
+struct fme_bridge {
+       struct platform_device *br;
+       struct list_head node;
+};
+
+/**
+ * struct fme_bridge_pdata - platform data for FME bridge platform device.
+ *
+ * @port: platform device of the port feature dev.
+ */
+struct fme_br_pdata {
+       struct platform_device *port;
+};
+
+#define FPGA_DFL_FME_MGR       "dfl-fme-mgr"
+#define FPGA_DFL_FME_BRIDGE    "dfl-fme-bridge"
+#define FPGA_DFL_FME_REGION    "dfl-fme-region"
+

Everything in dfl-fme-pr.h up to this point is good and general and
should remain in dfl-fme-pr.h.  The register #defines for this FME's
FPGA manager device below should be associated with the FPGA manager
driver.  Sorry if the way I stated that in the v3 review wasn't clear.

+/* FME Partial Reconfiguration Sub Feature Register Set */
+#define FME_PR_DFH             0x0
+#define FME_PR_CTRL            0x8
+#define FME_PR_STS             0x10
+#define FME_PR_DATA            0x18
+#define FME_PR_ERR             0x20
+#define FME_PR_INTFC_ID_H      0xA8
+#define FME_PR_INTFC_ID_L      0xB0
+
+/* FME PR Control Register Bitfield */
+#define FME_PR_CTRL_PR_RST     BIT(0)  /* Reset PR engine */
+#define FME_PR_CTRL_PR_RSTACK  BIT(4)  /* Ack for PR engine reset */
+#define FME_PR_CTRL_PR_RGN_ID  GENMASK_ULL(9, 7)       /* PR Region ID */
+#define FME_PR_CTRL_PR_START   BIT(12) /* Start to request for PR service */
+#define FME_PR_CTRL_PR_COMPLETE        BIT(13) /* PR data push complete notification */
+
+/* FME PR Status Register Bitfield */
+/* Number of available entries in HW queue inside the PR engine. */
+#define FME_PR_STS_PR_CREDIT   GENMASK_ULL(8, 0)
+#define FME_PR_STS_PR_STS      BIT(16) /* PR operation status */
+#define FME_PR_STS_PR_STS_IDLE 0
+#define FME_PR_STS_PR_CTRLR_STS        GENMASK_ULL(22, 20)     /* Controller status */
+#define FME_PR_STS_PR_HOST_STS GENMASK_ULL(27, 24)     /* PR host status */
+
+/* FME PR Data Register Bitfield */
+/* PR data from the raw-binary file. */
+#define FME_PR_DATA_PR_DATA_RAW        GENMASK_ULL(32, 0)
+
+/* FME PR Error Register */
+/* PR Operation errors detected. */
+#define FME_PR_ERR_OPERATION_ERR       BIT(0)
+/* CRC error detected. */
+#define FME_PR_ERR_CRC_ERR             BIT(1)
+/* Incompatible PR bitstream detected. */
+#define FME_PR_ERR_INCOMPATIBLE_BS     BIT(2)
+/* PR data push protocol violated. */
+#define FME_PR_ERR_PROTOCOL_ERR                BIT(3)
+/* PR data fifo overflow error detected */
+#define FME_PR_ERR_FIFO_OVERFLOW       BIT(4)

This stuff is specific to this FPGA manager device, so it should
either be in the dfl-fme-mgr.c or in a dfl-fme-mgr.h

quoted hunk

+
+#endif /* __DFL_FME_PR_H */

diff --git a/drivers/fpga/dfl-fme.h b/drivers/fpga/dfl-fme.h
new file mode 100644
index 0000000..c8bd48a
--- /dev/null
+++ b/drivers/fpga/dfl-fme.h

@@ -0,0 +1,38 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Header file for FPGA Management Engine (FME) Driver
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ */
+
+#ifndef __DFL_FME_H
+#define __DFL_FME_H
+
+/**
+ * struct fpga_fme - fpga fme private data
+ *
+ * @mgr: FME's FPGA manager platform device.
+ * @region_list: link list of FME's FPGA regions.
+ * @bridge_list: link list of FME's FPGA bridges.
+ * @pdata: fme platform device's pdata.
+ */
+struct fpga_fme {
+       struct platform_device *mgr;
+       struct list_head region_list;
+       struct list_head bridge_list;
+       struct feature_platform_data *pdata;
+};
+
+extern const struct feature_ops pr_mgmt_ops;
+
+#endif /* __DFL_FME_H */

diff --git a/include/uapi/linux/fpga-dfl.h b/include/uapi/linux/fpga-dfl.h
index 9321ee9..50ee831 100644
--- a/include/uapi/linux/fpga-dfl.h
+++ b/include/uapi/linux/fpga-dfl.h

@@ -14,6 +14,8 @@
 #ifndef _UAPI_LINUX_FPGA_DFL_H
 #define _UAPI_LINUX_FPGA_DFL_H

+#include <linux/types.h>
+
 #define FPGA_API_VERSION 0

 /*

@@ -26,6 +28,7 @@
 #define FPGA_MAGIC 0xB6

 #define FPGA_BASE 0
+#define FME_BASE 0x80

 /**
  * FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)

@@ -45,4 +48,28 @@

 #define FPGA_CHECK_EXTENSION   _IO(FPGA_MAGIC, FPGA_BASE + 1)

+/* IOCTLs for FME file descriptor */
+
+/**
+ * FPGA_FME_PORT_PR - _IOW(FPGA_MAGIC, FME_BASE + 0, struct fpga_fme_port_pr)
+ *
+ * Driver does Partial Reconfiguration based on Port ID and Buffer (Image)
+ * provided by caller.
+ * Return: 0 on success, -errno on failure.
+ * If FPGA_FME_PORT_PR returns -EIO, that indicates the HW has detected
+ * some errors during PR, under this case, the user can fetch HW error info
+ * from the status of FME's fpga manager.
+ */
+
+struct fpga_fme_port_pr {
+       /* Input */
+       __u32 argsz;            /* Structure length */
+       __u32 flags;            /* Zero for now */
+       __u32 port_id;
+       __u32 buffer_size;
+       __u64 buffer_address;   /* Userspace address to the buffer for PR */
+};
+
+#define FPGA_FME_PORT_PR       _IO(FPGA_MAGIC, FME_BASE + 0)
+
 #endif /* _UAPI_LINUX_FPGA_DFL_H */
--

2.7.4

Thanks,
Alan

Re: [PATCH v4 13/24] fpga: region: add compat_id support

From: Wu Hao <hidden>
Date: 2018-03-06 01:06:46

On Mon, Mar 05, 2018 at 01:42:41PM -0600, Alan Tull wrote:

On Thu, Mar 1, 2018 at 12:17 AM, Wu Hao [off-list ref] wrote:

quoted

On Wed, Feb 28, 2018 at 04:55:15PM -0600, Alan Tull wrote:

quoted

On Tue, Feb 13, 2018 at 3:24 AM, Wu Hao [off-list ref] wrote:

Hi Hao,

Hi Alan,

Thanks for the review.

quoted

This patch introduces a compat_id member and sysfs interface for each
fpga-region, e.g userspace applications could read the compat_id
from the sysfs interface for compatibility checking before PR.

Signed-off-by: Wu Hao <redacted>
---
 Documentation/ABI/testing/sysfs-class-fpga-region |  5 +++++
 drivers/fpga/fpga-region.c                        | 19 +++++++++++++++++++
 include/linux/fpga/fpga-region.h                  | 13 +++++++++++++
 3 files changed, 37 insertions(+)
 create mode 100644 Documentation/ABI/testing/sysfs-class-fpga-region

diff --git a/Documentation/ABI/testing/sysfs-class-fpga-region b/Documentation/ABI/testing/sysfs-class-fpga-region
new file mode 100644
index 0000000..419d930
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-class-fpga-region

@@ -0,0 +1,5 @@
+What:          /sys/class/fpga_region/<region>/compat_id
+Date:          February 2018
+KernelVersion: 4.16
+Contact:       Wu Hao <hao.wu@intel.com>
+Description:   FPGA region id for compatibility check.

It would be helpful to add some explanation here that although the
intended function of compat_id is set, the way the actual value is
defined or calculated is set by the layer that is creating the FPGA
region.

Sure.

Description: FPGA region id for compatibility check, e.g compatibility
             of the FPGA reconfiguration hardware and image. This value
             is defined or calculated by the layer that is creating the
             FPGA region. This interface returns the compat_id value or
             just error code -ENOENT in case compat_id is not used.

quoted

diff --git a/drivers/fpga/fpga-region.c b/drivers/fpga/fpga-region.c
index 660a91b..babec96 100644
--- a/drivers/fpga/fpga-region.c
+++ b/drivers/fpga/fpga-region.c

@@ -162,6 +162,24 @@ int fpga_region_program_fpga(struct fpga_region *region)
 }
 EXPORT_SYMBOL_GPL(fpga_region_program_fpga);

+static ssize_t compat_id_show(struct device *dev,
+                             struct device_attribute *attr, char *buf)
+{
+       struct fpga_region *region = to_fpga_region(dev);

This looks good, but not all users of FPGA are going to use compat_id.
How would you feel about making it a pointer in struct fpga_region?
With compat_id as a pointer, could check for non-null compat_id
pointer and return an error if it wasn't initialized.

It sounds good to me.

if (!region->compat_id)
        return -ENOENT;

Yes, thanks!

Thanks for the review.

Hao

Alan

quoted

+
+       return sprintf(buf, "%016llx%016llx\n",
+                      (unsigned long long)region->compat_id.id_h,
+                      (unsigned long long)region->compat_id.id_l);
+}
+
+static DEVICE_ATTR_RO(compat_id);
+
+static struct attribute *fpga_region_attrs[] = {
+       &dev_attr_compat_id.attr,
+       NULL,
+};
+ATTRIBUTE_GROUPS(fpga_region);
+
 int fpga_region_register(struct fpga_region *region)
 {
        struct device *dev = region->parent;

@@ -226,6 +244,7 @@ static int __init fpga_region_init(void)
        if (IS_ERR(fpga_region_class))
                return PTR_ERR(fpga_region_class);

+       fpga_region_class->dev_groups = fpga_region_groups;
        fpga_region_class->dev_release = fpga_region_dev_release;

        return 0;

diff --git a/include/linux/fpga/fpga-region.h b/include/linux/fpga/fpga-region.h
index 423c87e..bf97dcc 100644
--- a/include/linux/fpga/fpga-region.h
+++ b/include/linux/fpga/fpga-region.h

@@ -6,6 +6,17 @@
 #include <linux/fpga/fpga-bridge.h>

 /**
+ * struct fpga_region_compat_id - FPGA Region id for compatibility check
+ *
+ * @id_h: high 64bit of the compat_id
+ * @id_l: low 64bit of the compat_id
+ */
+struct fpga_region_compat_id {
+       u64 id_h;
+       u64 id_l;

I guess each user will choose how to define these bits.

Yes.

quoted

+};
+
+/**
  * struct fpga_region - FPGA Region structure
  * @dev: FPGA Region device
  * @parent: parent device

@@ -13,6 +24,7 @@
  * @bridge_list: list of FPGA bridges specified in region
  * @mgr: FPGA manager
  * @info: FPGA image info
+ * @compat_id: FPGA region id for compatibility check.
  * @priv: private data
  * @get_bridges: optional function to get bridges to a list
  * @groups: optional attribute groups.
@@ -24,6 +36,7 @@ struct fpga_region {
        struct list_head bridge_list;
        struct fpga_manager *mgr;
        struct fpga_image_info *info;
+       struct fpga_region_compat_id compat_id;

Here it would be a pointer instead.

Yes. Will update this patch.

Thanks
Hao

quoted

Alan

quoted

        void *priv;
        int (*get_bridges)(struct fpga_region *region);
        const struct attribute_group **groups;
--
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v4 14/24] fpga: dfl: fme: add partial reconfiguration sub feature support

From: Wu Hao <hidden>
Date: 2018-03-06 02:18:27

On Mon, Mar 05, 2018 at 04:46:02PM -0600, Alan Tull wrote:

On Tue, Feb 13, 2018 at 3:24 AM, Wu Hao [off-list ref] wrote:

Hi Hao,

Hi Alan,

Thanks for the comments.

We are going to want to be able use different FPGA managers with this
framework.  The different manager may be part of a different FME in
fabric or it may be a hardware FPGA manager.  Fortunately, at this
point now the changes, noted below, to get there are pretty small.

Yes, understand these could be some cases that FME having different PR
hardware.

quoted

From: Kang Luwei <redacted>

Partial Reconfiguration (PR) is the most important function for FME. It
allows reconfiguration for given Port/Accelerated Function Unit (AFU).

It creates platform devices for fpga-mgr, fpga-regions and fpga-bridges,
and invokes fpga-region's interface (fpga_region_program_fpga) for PR
operation once PR request received via ioctl. Below user space interface
is exposed by this sub feature.

Ioctl interface:
* FPGA_FME_PORT_PR
  Do partial reconfiguration per information from userspace, including
  target port(AFU), buffer size and address info. It returns error code
  to userspace if failed. For detailed PR error information, user needs
  to read fpga-mgr's status sysfs interface.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Kang Luwei <redacted>
Signed-off-by: Xiao Guangrong <redacted>
Signed-off-by: Wu Hao <redacted>
---
v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
    switched to GPLv2 license.
    removed status from FPGA_FME_PORT_PR ioctl data structure.
    added platform devices creation for fpga-mgr/fpga-region/fpga-bridge.
    switched to fpga-region interface fpga_region_program_fpga for PR.
    fixed comments from Alan Tull on FPGA_MGR_PARTIAL_RECONFIG flag usage.
    fixed kbuild warnings.
v3: rename driver files to dfl-fme-*.
    rebase due to fpga APIs change.
    replace bitfields.
    switch to fpga_cdev_find_port to find port device.
v4: rebase and correct comments for some function.
    fix SPDX license issue.
    remove unnecessary input parameter for destroy_bridge/region function.
    add dfl-fme-pr.h for PR sub feature data structure and registers.
---
 drivers/fpga/Makefile         |   2 +-
 drivers/fpga/dfl-fme-main.c   |  45 +++-
 drivers/fpga/dfl-fme-pr.c     | 497 ++++++++++++++++++++++++++++++++++++++++++
 drivers/fpga/dfl-fme-pr.h     | 113 ++++++++++
 drivers/fpga/dfl-fme.h        |  38 ++++
 include/uapi/linux/fpga-dfl.h |  27 +++
 6 files changed, 720 insertions(+), 2 deletions(-)
 create mode 100644 drivers/fpga/dfl-fme-pr.c
 create mode 100644 drivers/fpga/dfl-fme-pr.h
 create mode 100644 drivers/fpga/dfl-fme.h

diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index fbd1c85..3c44fc9 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile

@@ -32,7 +32,7 @@ obj-$(CONFIG_OF_FPGA_REGION)          += of-fpga-region.o
 obj-$(CONFIG_FPGA_DFL)                 += dfl.o
 obj-$(CONFIG_FPGA_DFL_FME)             += dfl-fme.o

-dfl-fme-objs := dfl-fme-main.o
+dfl-fme-objs := dfl-fme-main.o dfl-fme-pr.o

 # Drivers for FPGAs which implement DFL
 obj-$(CONFIG_FPGA_DFL_PCI)             += dfl-pci.o

diff --git a/drivers/fpga/dfl-fme-main.c b/drivers/fpga/dfl-fme-main.c
index 1a9929c..967a44c 100644
--- a/drivers/fpga/dfl-fme-main.c
+++ b/drivers/fpga/dfl-fme-main.c

@@ -19,6 +19,7 @@
 #include <linux/fpga-dfl.h>

 #include "dfl.h"
+#include "dfl-fme.h"

 static ssize_t ports_num_show(struct device *dev,
                              struct device_attribute *attr, char *buf)

@@ -111,6 +112,10 @@ static struct feature_driver fme_feature_drvs[] = {
                .ops = &fme_hdr_ops,
        },
        {
+               .id = FME_FEATURE_ID_PR_MGMT,
+               .ops = &pr_mgmt_ops,
+       },
+       {
                .ops = NULL,
        },
 };

@@ -194,14 +199,49 @@ static const struct file_operations fme_fops = {
        .unlocked_ioctl = fme_ioctl,
 };

+static int fme_dev_init(struct platform_device *pdev)
+{
+       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+       struct fpga_fme *fme;
+
+       fme = devm_kzalloc(&pdev->dev, sizeof(*fme), GFP_KERNEL);
+       if (!fme)
+               return -ENOMEM;
+
+       fme->pdata = pdata;
+
+       mutex_lock(&pdata->lock);
+       fpga_pdata_set_private(pdata, fme);
+       mutex_unlock(&pdata->lock);
+
+       return 0;
+}
+
+static void fme_dev_destroy(struct platform_device *pdev)
+{
+       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+       struct fpga_fme *fme;
+
+       mutex_lock(&pdata->lock);
+       fme = fpga_pdata_get_private(pdata);
+       fpga_pdata_set_private(pdata, NULL);
+       mutex_unlock(&pdata->lock);
+
+       devm_kfree(&pdev->dev, fme);

Is this needed?  This is called when the device is being destroyed anyway.

I see, will remove this line.

quoted

+}
+
 static int fme_probe(struct platform_device *pdev)
 {
        int ret;

-       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
+       ret = fme_dev_init(pdev);
        if (ret)
                goto exit;

+       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
+       if (ret)
+               goto dev_destroy;
+
        ret = fpga_register_dev_ops(pdev, &fme_fops, THIS_MODULE);
        if (ret)
                goto feature_uinit;

@@ -210,6 +250,8 @@ static int fme_probe(struct platform_device *pdev)

 feature_uinit:
        fpga_dev_feature_uinit(pdev);
+dev_destroy:
+       fme_dev_destroy(pdev);
 exit:
        return ret;
 }

@@ -218,6 +260,7 @@ static int fme_remove(struct platform_device *pdev)
 {
        fpga_dev_feature_uinit(pdev);
        fpga_unregister_dev_ops(pdev);
+       fme_dev_destroy(pdev);

        return 0;
 }

diff --git a/drivers/fpga/dfl-fme-pr.c b/drivers/fpga/dfl-fme-pr.c
new file mode 100644
index 0000000..526e90b
--- /dev/null
+++ b/drivers/fpga/dfl-fme-pr.c

@@ -0,0 +1,497 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Driver for FPGA Management Engine (FME) Partial Reconfiguration
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Christopher Rauer <christopher.rauer@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ */
+
+#include <linux/types.h>
+#include <linux/device.h>
+#include <linux/vmalloc.h>
+#include <linux/uaccess.h>
+#include <linux/fpga/fpga-mgr.h>
+#include <linux/fpga/fpga-bridge.h>
+#include <linux/fpga/fpga-region.h>
+#include <linux/fpga-dfl.h>
+
+#include "dfl.h"
+#include "dfl-fme.h"
+#include "dfl-fme-pr.h"
+
+static struct fme_region *
+find_fme_region_by_port_id(struct fpga_fme *fme, int port_id)
+{
+       struct fme_region *fme_region;
+
+       list_for_each_entry(fme_region, &fme->region_list, node)
+               if (fme_region->port_id == port_id)
+                       return fme_region;
+
+       return NULL;
+}
+
+static int fpga_fme_region_match(struct device *dev, const void *data)
+{
+       return dev->parent == data;
+}
+
+static struct fpga_region *
+fpga_fme_region_find(struct fpga_fme *fme, int port_id)
+{
+       struct fme_region *fme_region;
+       struct fpga_region *region;
+
+       fme_region = find_fme_region_by_port_id(fme, port_id);
+       if (!fme_region)
+               return NULL;
+
+       region = fpga_region_class_find(NULL, &fme_region->region->dev,
+                                       fpga_fme_region_match);
+       if (!region)
+               return NULL;
+
+       return region;
+}
+
+static int fme_pr(struct platform_device *pdev, unsigned long arg)
+{
+       void __user *argp = (void __user *)arg;
+       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+       struct fpga_fme *fme;
+       struct fpga_image_info *info;
+       struct fpga_region *region;
+       struct fpga_fme_port_pr port_pr;
+       unsigned long minsz;
+       void __iomem *fme_hdr;
+       void *buf = NULL;
+       int ret = 0;
+       u64 v;
+
+       minsz = offsetofend(struct fpga_fme_port_pr, buffer_address);
+
+       if (copy_from_user(&port_pr, argp, minsz))
+               return -EFAULT;
+
+       if (port_pr.argsz < minsz || port_pr.flags)
+               return -EINVAL;
+
+       if (!IS_ALIGNED(port_pr.buffer_size, 4))
+               return -EINVAL;
+
+       /* get fme header region */
+       fme_hdr = get_feature_ioaddr_by_id(&pdev->dev,
+                                          FME_FEATURE_ID_HEADER);
+
+       /* check port id */
+       v = readq(fme_hdr + FME_HDR_CAP);
+       if (port_pr.port_id >= FIELD_GET(FME_CAP_NUM_PORTS, v)) {
+               dev_dbg(&pdev->dev, "port number more than maximum\n");
+               return -EINVAL;
+       }
+
+       if (!access_ok(VERIFY_READ,
+                      (void __user *)(unsigned long)port_pr.buffer_address,
+                      port_pr.buffer_size))
+               return -EFAULT;
+
+       buf = vmalloc(port_pr.buffer_size);
+       if (!buf)
+               return -ENOMEM;
+
+       if (copy_from_user(buf,
+                          (void __user *)(unsigned long)port_pr.buffer_address,
+                          port_pr.buffer_size)) {
+               ret = -EFAULT;
+               goto free_exit;
+       }
+
+       /* prepare fpga_image_info for PR */
+       info = fpga_image_info_alloc(&pdev->dev);
+       if (!info) {
+               ret = -ENOMEM;
+               goto free_exit;
+       }
+
+       info->flags |= FPGA_MGR_PARTIAL_RECONFIG;
+
+       mutex_lock(&pdata->lock);
+       fme = fpga_pdata_get_private(pdata);
+       /* fme device has been unregistered. */
+       if (!fme) {
+               ret = -EINVAL;
+               goto unlock_exit;
+       }
+
+       region = fpga_fme_region_find(fme, port_pr.port_id);
+       if (!region) {
+               ret = -EINVAL;
+               goto unlock_exit;
+       }
+
+       fpga_image_info_free(region->info);
+
+       info->buf = buf;
+       info->count = port_pr.buffer_size;
+       info->region_id = port_pr.port_id;
+       region->info = info;
+
+       ret = fpga_region_program_fpga(region);
+
+       if (region->get_bridges)
+               fpga_bridges_put(&region->bridge_list);
+
+       put_device(&region->dev);
+unlock_exit:
+       mutex_unlock(&pdata->lock);
+free_exit:
+       vfree(buf);
+       if (copy_to_user((void __user *)arg, &port_pr, minsz))
+               return -EFAULT;
+
+       return ret;
+}
+
+/**
+ * fpga_fme_create_mgr - create fpga mgr platform device as child device
+ *
+ * @pdata: fme platform_device's pdata
+ *
+ * Return: mgr platform device if successful, and error code otherwise.
+ */
+static struct platform_device *
+fpga_fme_create_mgr(struct feature_platform_data *pdata)
+{
+       struct platform_device *mgr, *fme = pdata->dev;
+       struct feature *feature;
+       struct resource res;
+       struct resource *pres;
+       int ret = -ENOMEM;
+
+       feature = get_feature_by_id(&pdata->dev->dev, FME_FEATURE_ID_PR_MGMT);
+       if (!feature)
+               return ERR_PTR(-ENODEV);
+
+       /*
+        * Each FME has only one fpga-mgr, so allocate platform device using
+        * the same FME platform device id.
+        */
+       mgr = platform_device_alloc(FPGA_DFL_FME_MGR, fme->id);

At this point, the framework is assuming all FME's include the same
FPGA manager device which would use the driver in dfl-fme-mgr.c.

I'm thinking of two cases where the manager isn't the same as a
dfl-fme-mgr.c manager are a bit different:

(1) a FME-based FPGA manager, but different implementation, different
registers.  The constraint is that the port implementation has to be
similar enough to use the rest of the base FME code.   I am wondering
if the FPGA manager can be added to the DFL.  At this point, the DFL
would drive which FPGA manager is alloc'd.   That way the user gets to
use all this code in dfl-fme-pr.c but with their FPGA manager.

Actually I'm not sure how this will be implemented in the hardware in the
future, but from my understanding, there may be several methods to add this
support (a different PR hardware) to FME.

1) introduce a new PR sub feature to FME.
   driver knows it by different feature id, and create different fpga
   manager platform device, but may not be able to reuse dfl-fme-pr.c.

2) add a different PR hardware support to current PR sub feature.
   It requires hardware to add registers to indicate this is a different
   PR hardware than dfl-fme-mgr.c, and its register region information
   (e.g location and size). Then this dfl-fme-pr.c driver could read
   these information from hardware and create a different fpga manager
   platform device with correct resources.

I think current driver framework could support both cases above for sure.

(2) a FPGA manager that can be added by device tree in the case of a
platform that is using device tree.  I think this will be pretty
simple and can be done later when someone is actually bringing this
framework up on a FPGA running under device tree.  I'm thinking that
the base DFL device that reads the dfl data from hardware can have a
DT property that points to the FPGA manager.  That manager can be
saved somewhere handy like the pdata and passed down to this code,
which realizes it can use that existing device and doesn't need to
alloc a platform device.  But again, that's probably best done later.

Sure, we can discuss further when we really need it. Actually per my
understanding, if hardware describes itself clearly, we may not have to
use DT for fpga manager, as driver could create it automatically based
on information read from hardware. :)

quoted

+       if (!mgr)
+               return ERR_PTR(ret);
+
+       mgr->dev.parent = &fme->dev;
+
+       pres = platform_get_resource(fme, IORESOURCE_MEM,
+                                    feature->resource_index);
+       if (!pres) {
+               ret = -ENODEV;
+               goto create_mgr_err;
+       }
+
+       memset(&res, 0, sizeof(struct resource));
+
+       res.start = pres->start;
+       res.end = pres->end;
+       res.name = pres->name;
+       res.flags = IORESOURCE_MEM;
+
+       ret = platform_device_add_resources(mgr, &res, 1);
+       if (ret)
+               goto create_mgr_err;
+
+       ret = platform_device_add(mgr);
+       if (ret)
+               goto create_mgr_err;
+
+       return mgr;
+
+create_mgr_err:
+       platform_device_put(mgr);
+       return ERR_PTR(ret);
+}
+
+/**
+ * fpga_fme_destroy_mgr - destroy fpga mgr platform device
+ * @pdata: fme platform device's pdata
+ */
+static void fpga_fme_destroy_mgr(struct feature_platform_data *pdata)
+{
+       struct fpga_fme *priv = fpga_pdata_get_private(pdata);
+
+       platform_device_unregister(priv->mgr);
+}
+
+/**
+ * fpga_fme_create_bridge - create fme fpga bridge platform device as child
+ *
+ * @pdata: fme platform device's pdata
+ * @port_id: port id for the bridge to be created.
+ *
+ * Return: bridge platform device if successful, and error code otherwise.
+ */
+static struct fme_bridge *
+fpga_fme_create_bridge(struct feature_platform_data *pdata, int port_id)
+{
+       struct device *dev = &pdata->dev->dev;
+       struct fme_br_pdata br_pdata;
+       struct fme_bridge *fme_br;
+       int ret = -ENOMEM;
+
+       fme_br = devm_kzalloc(dev, sizeof(*fme_br), GFP_KERNEL);
+       if (!fme_br)
+               return ERR_PTR(ret);
+
+       br_pdata.port = fpga_cdev_find_port(fpga_pdata_to_fpga_cdev(pdata),
+                                           &port_id, fpga_port_check_id);
+       if (!br_pdata.port)
+               return ERR_PTR(-ENODEV);
+
+       /*
+        * Each FPGA device may have more than one port, so allocate platform
+        * device using the same port platform device id.
+        */
+       fme_br->br = platform_device_alloc(FPGA_DFL_FME_BRIDGE,
+                                          br_pdata.port->id);
+       if (!fme_br->br) {
+               ret = -ENOMEM;
+               goto create_br_err;
+       }
+
+       fme_br->br->dev.parent = dev;
+
+       ret = platform_device_add_data(fme_br->br, &br_pdata, sizeof(br_pdata));
+       if (ret)
+               goto create_br_err;
+
+       ret = platform_device_add(fme_br->br);
+       if (ret)
+               goto create_br_err;
+
+       return fme_br;
+
+create_br_err:
+       platform_device_put(fme_br->br);
+       put_device(&br_pdata.port->dev);
+       return ERR_PTR(ret);
+}
+
+/**
+ * fpga_fme_destroy_bridge - destroy fpga bridge platform device
+ * @fme_br: fme bridge to destroy
+ */
+static void fpga_fme_destroy_bridge(struct fme_bridge *fme_br)
+{
+       struct fme_br_pdata *br_pdata = dev_get_platdata(&fme_br->br->dev);
+
+       put_device(&br_pdata->port->dev);
+       platform_device_unregister(fme_br->br);
+}
+
+/**
+ * fpga_fme_destroy_bridge - destroy all fpga bridge platform device
+ * @pdata: fme platform device's pdata
+ */
+static void fpga_fme_destroy_bridges(struct feature_platform_data *pdata)
+{
+       struct fpga_fme *priv = fpga_pdata_get_private(pdata);
+       struct fme_bridge *fbridge, *tmp;
+
+       list_for_each_entry_safe(fbridge, tmp, &priv->bridge_list, node) {
+               list_del(&fbridge->node);
+               fpga_fme_destroy_bridge(fbridge);
+       }
+}
+
+/**
+ * fpga_fme_create_region - create fpga region platform device as child
+ *
+ * @pdata: fme platform device's pdata
+ * @mgr: mgr platform device needed for region
+ * @br: br platform device needed for region
+ * @port_id: port id
+ *
+ * Return: fme region if successful, and error code otherwise.
+ */
+static struct fme_region *
+fpga_fme_create_region(struct feature_platform_data *pdata,
+                      struct platform_device *mgr,
+                      struct platform_device *br, int port_id)
+{
+       struct device *dev = &pdata->dev->dev;
+       struct fme_region_pdata region_pdata;
+       struct fme_region *fme_region;
+       int ret = -ENOMEM;
+
+       fme_region = devm_kzalloc(dev, sizeof(*fme_region), GFP_KERNEL);
+       if (!fme_region)
+               return ERR_PTR(ret);
+
+       region_pdata.mgr = mgr;
+       region_pdata.br = br;
+
+       /*
+        * Each FPGA device may have more than one port, so allocate platform
+        * device using the same port platform device id.
+        */
+       fme_region->region = platform_device_alloc(FPGA_DFL_FME_REGION, br->id);
+       if (!fme_region->region)
+               return ERR_PTR(ret);
+
+       fme_region->region->dev.parent = dev;
+
+       ret = platform_device_add_data(fme_region->region, &region_pdata,
+                                      sizeof(region_pdata));
+       if (ret)
+               goto create_region_err;
+
+       ret = platform_device_add(fme_region->region);
+       if (ret)
+               goto create_region_err;
+
+       fme_region->port_id = port_id;
+
+       return fme_region;
+
+create_region_err:
+       platform_device_put(fme_region->region);
+       return ERR_PTR(ret);
+}
+
+/**
+ * fpga_fme_destroy_region - destroy fme region
+ * @fme_region: fme region to destroy
+ */
+static void fpga_fme_destroy_region(struct fme_region *fme_region)
+{
+       platform_device_unregister(fme_region->region);
+}
+
+/**
+ * fpga_fme_destroy_regions - destroy all fme regions
+ * @pdata: fme platform device's pdata
+ */
+static void fpga_fme_destroy_regions(struct feature_platform_data *pdata)
+{
+       struct fpga_fme *priv = fpga_pdata_get_private(pdata);
+       struct fme_region *fme_region, *tmp;
+
+       list_for_each_entry_safe(fme_region, tmp, &priv->region_list, node) {
+               list_del(&fme_region->node);
+               fpga_fme_destroy_region(fme_region);
+       }
+}
+
+static int pr_mgmt_init(struct platform_device *pdev, struct feature *feature)
+{
+       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+       void __iomem *fme_hdr;
+       struct platform_device *mgr;
+       struct fme_region *fme_region;
+       struct fme_bridge *fme_br;
+       struct fpga_fme *priv;
+       int ret = -ENODEV, i = 0;
+       u64 fme_cap, port_offset;
+
+       fme_hdr = get_feature_ioaddr_by_id(&pdev->dev,
+                                          FME_FEATURE_ID_HEADER);
+
+       mutex_lock(&pdata->lock);
+       priv = fpga_pdata_get_private(pdata);
+
+       /* Initialize the region and bridge sub device list */
+       INIT_LIST_HEAD(&priv->region_list);
+       INIT_LIST_HEAD(&priv->bridge_list);
+
+       /* Create fpga mgr platform device */
+       mgr = fpga_fme_create_mgr(pdata);
+       if (IS_ERR(mgr)) {
+               dev_err(&pdev->dev, "fail to create fpga mgr pdev\n");
+               goto unlock;
+       }
+
+       priv->mgr = mgr;
+
+       /* Read capability register to check number of regions and bridges */
+       fme_cap = readq(fme_hdr + FME_HDR_CAP);
+       for (; i < FIELD_GET(FME_CAP_NUM_PORTS, fme_cap); i++) {
+               port_offset = readq(fme_hdr + FME_HDR_PORT_OFST(i));
+               if (!(port_offset & FME_PORT_OFST_IMP))
+                       continue;
+
+               /* Create bridge for each port */
+               fme_br = fpga_fme_create_bridge(pdata, i);
+               if (IS_ERR(fme_br)) {
+                       ret = PTR_ERR(fme_br);
+                       goto destroy_region;
+               }
+
+               list_add(&fme_br->node, &priv->bridge_list);
+
+               /* Create region for each port */
+               fme_region = fpga_fme_create_region(pdata, mgr, fme_br->br, i);
+               if (!fme_region) {
+                       ret = PTR_ERR(fme_region);
+                       goto destroy_region;
+               }
+
+               list_add(&fme_region->node, &priv->region_list);
+       }
+       mutex_unlock(&pdata->lock);
+
+       return 0;
+
+destroy_region:
+       fpga_fme_destroy_regions(pdata);
+       fpga_fme_destroy_bridges(pdata);
+       fpga_fme_destroy_mgr(pdata);
+unlock:
+       mutex_unlock(&pdata->lock);
+       return ret;
+}
+
+static void pr_mgmt_uinit(struct platform_device *pdev, struct feature *feature)
+{
+       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+       struct fpga_fme *priv;
+
+       mutex_lock(&pdata->lock);
+       priv = fpga_pdata_get_private(pdata);
+
+       fpga_fme_destroy_regions(pdata);
+       fpga_fme_destroy_bridges(pdata);
+       fpga_fme_destroy_mgr(pdata);
+       mutex_unlock(&pdata->lock);
+}
+
+static long fme_pr_ioctl(struct platform_device *pdev, struct feature *feature,
+                        unsigned int cmd, unsigned long arg)
+{
+       long ret;
+
+       switch (cmd) {
+       case FPGA_FME_PORT_PR:
+               ret = fme_pr(pdev, arg);
+               break;
+       default:
+               ret = -ENODEV;
+       }
+
+       return ret;
+}
+
+const struct feature_ops pr_mgmt_ops = {
+       .init = pr_mgmt_init,
+       .uinit = pr_mgmt_uinit,
+       .ioctl = fme_pr_ioctl,
+};

diff --git a/drivers/fpga/dfl-fme-pr.h b/drivers/fpga/dfl-fme-pr.h
new file mode 100644
index 0000000..11bd001
--- /dev/null
+++ b/drivers/fpga/dfl-fme-pr.h

@@ -0,0 +1,113 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Header file for FPGA Management Engine (FME) Partial Reconfiguration Driver
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ */
+
+#ifndef __DFL_FME_PR_H
+#define __DFL_FME_PR_H
+
+#include <linux/platform_device.h>
+
+/**
+ * struct fme_region - FME fpga region data structure
+ *
+ * @region: platform device of the FPGA region.
+ * @node: used to link fme_region to a list.
+ * @port_id: indicate which port this region connected to.
+ */
+struct fme_region {
+       struct platform_device *region;
+       struct list_head node;
+       int port_id;
+};
+
+/**
+ * struct fme_region_pdata - platform data for FME region platform device.
+ *
+ * @mgr: platform device of the FPGA manager.
+ * @br: platform device of the FPGA bridge.
+ * @region_id: region id (same as port_id).
+ */
+struct fme_region_pdata {
+       struct platform_device *mgr;
+       struct platform_device *br;
+       int region_id;
+};
+
+/**
+ * struct fme_bridge - FME fpga bridge data structure
+ *
+ * @br: platform device of the FPGA bridge.
+ * @node: used to link fme_bridge to a list.
+ */
+struct fme_bridge {
+       struct platform_device *br;
+       struct list_head node;
+};
+
+/**
+ * struct fme_bridge_pdata - platform data for FME bridge platform device.
+ *
+ * @port: platform device of the port feature dev.
+ */
+struct fme_br_pdata {
+       struct platform_device *port;
+};
+
+#define FPGA_DFL_FME_MGR       "dfl-fme-mgr"
+#define FPGA_DFL_FME_BRIDGE    "dfl-fme-bridge"
+#define FPGA_DFL_FME_REGION    "dfl-fme-region"
+

Everything in dfl-fme-pr.h up to this point is good and general and
should remain in dfl-fme-pr.h.  The register #defines for this FME's
FPGA manager device below should be associated with the FPGA manager
driver.  Sorry if the way I stated that in the v3 review wasn't clear.

Actually I put the PR sub feature register set definitions in this header
file (dfl-fme-pr.h), because it's possible the driver (dfl-fme-pr.c) of
this PR sub feature access some of the registers in the future. e.g read
some PR sub feature registers to create different fpga manager platform
devices as I mentioned above. 

I have to say this is only future consideration, and in this dfl-fme-pr.c
driver there is no code to access below registers currently. I can move all
of them to dfl-fme-mgr.h or dfl-fme-mgr.c in the next version if this is
preferred. : )

quoted

+/* FME Partial Reconfiguration Sub Feature Register Set */
+#define FME_PR_DFH             0x0
+#define FME_PR_CTRL            0x8
+#define FME_PR_STS             0x10
+#define FME_PR_DATA            0x18
+#define FME_PR_ERR             0x20
+#define FME_PR_INTFC_ID_H      0xA8
+#define FME_PR_INTFC_ID_L      0xB0
+
+/* FME PR Control Register Bitfield */
+#define FME_PR_CTRL_PR_RST     BIT(0)  /* Reset PR engine */
+#define FME_PR_CTRL_PR_RSTACK  BIT(4)  /* Ack for PR engine reset */
+#define FME_PR_CTRL_PR_RGN_ID  GENMASK_ULL(9, 7)       /* PR Region ID */
+#define FME_PR_CTRL_PR_START   BIT(12) /* Start to request for PR service */
+#define FME_PR_CTRL_PR_COMPLETE        BIT(13) /* PR data push complete notification */
+
+/* FME PR Status Register Bitfield */
+/* Number of available entries in HW queue inside the PR engine. */
+#define FME_PR_STS_PR_CREDIT   GENMASK_ULL(8, 0)
+#define FME_PR_STS_PR_STS      BIT(16) /* PR operation status */
+#define FME_PR_STS_PR_STS_IDLE 0
+#define FME_PR_STS_PR_CTRLR_STS        GENMASK_ULL(22, 20)     /* Controller status */
+#define FME_PR_STS_PR_HOST_STS GENMASK_ULL(27, 24)     /* PR host status */
+
+/* FME PR Data Register Bitfield */
+/* PR data from the raw-binary file. */
+#define FME_PR_DATA_PR_DATA_RAW        GENMASK_ULL(32, 0)
+
+/* FME PR Error Register */
+/* PR Operation errors detected. */
+#define FME_PR_ERR_OPERATION_ERR       BIT(0)
+/* CRC error detected. */
+#define FME_PR_ERR_CRC_ERR             BIT(1)
+/* Incompatible PR bitstream detected. */
+#define FME_PR_ERR_INCOMPATIBLE_BS     BIT(2)
+/* PR data push protocol violated. */
+#define FME_PR_ERR_PROTOCOL_ERR                BIT(3)
+/* PR data fifo overflow error detected */
+#define FME_PR_ERR_FIFO_OVERFLOW       BIT(4)

This stuff is specific to this FPGA manager device, so it should
either be in the dfl-fme-mgr.c or in a dfl-fme-mgr.h

same as above, I can fix this in the next version.

Thanks
Hao

quoted

+
+#endif /* __DFL_FME_PR_H */

diff --git a/drivers/fpga/dfl-fme.h b/drivers/fpga/dfl-fme.h
new file mode 100644
index 0000000..c8bd48a
--- /dev/null
+++ b/drivers/fpga/dfl-fme.h

@@ -0,0 +1,38 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Header file for FPGA Management Engine (FME) Driver
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ */
+
+#ifndef __DFL_FME_H
+#define __DFL_FME_H
+
+/**
+ * struct fpga_fme - fpga fme private data
+ *
+ * @mgr: FME's FPGA manager platform device.
+ * @region_list: link list of FME's FPGA regions.
+ * @bridge_list: link list of FME's FPGA bridges.
+ * @pdata: fme platform device's pdata.
+ */
+struct fpga_fme {
+       struct platform_device *mgr;
+       struct list_head region_list;
+       struct list_head bridge_list;
+       struct feature_platform_data *pdata;
+};
+
+extern const struct feature_ops pr_mgmt_ops;
+
+#endif /* __DFL_FME_H */

diff --git a/include/uapi/linux/fpga-dfl.h b/include/uapi/linux/fpga-dfl.h
index 9321ee9..50ee831 100644
--- a/include/uapi/linux/fpga-dfl.h
+++ b/include/uapi/linux/fpga-dfl.h

@@ -14,6 +14,8 @@
 #ifndef _UAPI_LINUX_FPGA_DFL_H
 #define _UAPI_LINUX_FPGA_DFL_H

+#include <linux/types.h>
+
 #define FPGA_API_VERSION 0

 /*

@@ -26,6 +28,7 @@
 #define FPGA_MAGIC 0xB6

 #define FPGA_BASE 0
+#define FME_BASE 0x80

 /**
  * FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)

@@ -45,4 +48,28 @@

 #define FPGA_CHECK_EXTENSION   _IO(FPGA_MAGIC, FPGA_BASE + 1)

+/* IOCTLs for FME file descriptor */
+
+/**
+ * FPGA_FME_PORT_PR - _IOW(FPGA_MAGIC, FME_BASE + 0, struct fpga_fme_port_pr)
+ *
+ * Driver does Partial Reconfiguration based on Port ID and Buffer (Image)
+ * provided by caller.
+ * Return: 0 on success, -errno on failure.
+ * If FPGA_FME_PORT_PR returns -EIO, that indicates the HW has detected
+ * some errors during PR, under this case, the user can fetch HW error info
+ * from the status of FME's fpga manager.
+ */
+
+struct fpga_fme_port_pr {
+       /* Input */
+       __u32 argsz;            /* Structure length */
+       __u32 flags;            /* Zero for now */
+       __u32 port_id;
+       __u32 buffer_size;
+       __u64 buffer_address;   /* Userspace address to the buffer for PR */
+};
+
+#define FPGA_FME_PORT_PR       _IO(FPGA_MAGIC, FME_BASE + 0)
+
 #endif /* _UAPI_LINUX_FPGA_DFL_H */
--

2.7.4

Thanks,
Alan

Re: [PATCH v4 14/24] fpga: dfl: fme: add partial reconfiguration sub feature support

From: Alan Tull <atull@kernel.org>
Date: 2018-03-06 18:30:19

  On Mon, Mar 5, 2018 at 8:08 PM, Wu Hao [off-list ref] wrote:

On Mon, Mar 05, 2018 at 04:46:02PM -0600, Alan Tull wrote:

quoted

On Tue, Feb 13, 2018 at 3:24 AM, Wu Hao [off-list ref] wrote:

Hi Hao,

Hi Alan,

Thanks for the comments.

quoted

We are going to want to be able use different FPGA managers with this
framework.  The different manager may be part of a different FME in
fabric or it may be a hardware FPGA manager.  Fortunately, at this
point now the changes, noted below, to get there are pretty small.

Yes, understand these could be some cases that FME having different PR
hardware.

Or supporting reduced FME plus hardware-based FPGA manager.

Just to re-emphasize, the basic intent of the FPGA manager subsystem
in the first place is to have FPGA managers separate from higher level
frameworks so that the higher level frameworks will be able to able to
use different FPGAs.

quoted

+/**
+ * fpga_fme_create_mgr - create fpga mgr platform device as child device
+ *
+ * @pdata: fme platform_device's pdata
+ *
+ * Return: mgr platform device if successful, and error code otherwise.
+ */
+static struct platform_device *
+fpga_fme_create_mgr(struct feature_platform_data *pdata)
+{
+       struct platform_device *mgr, *fme = pdata->dev;
+       struct feature *feature;
+       struct resource res;
+       struct resource *pres;
+       int ret = -ENOMEM;
+
+       feature = get_feature_by_id(&pdata->dev->dev, FME_FEATURE_ID_PR_MGMT);
+       if (!feature)
+               return ERR_PTR(-ENODEV);
+
+       /*
+        * Each FME has only one fpga-mgr, so allocate platform device using
+        * the same FME platform device id.
+        */
+       mgr = platform_device_alloc(FPGA_DFL_FME_MGR, fme->id);

At this point, the framework is assuming all FME's include the same
FPGA manager device which would use the driver in dfl-fme-mgr.c.

I'm thinking of two cases where the manager isn't the same as a
dfl-fme-mgr.c manager are a bit different:

(1) a FME-based FPGA manager, but different implementation, different
registers.  The constraint is that the port implementation has to be
similar enough to use the rest of the base FME code.   I am wondering
if the FPGA manager can be added to the DFL.  At this point, the DFL
would drive which FPGA manager is alloc'd.   That way the user gets to
use all this code in dfl-fme-pr.c but with their FPGA manager.

Actually I'm not sure how this will be implemented in the hardware in the
future, but from my understanding, there may be several methods to add this
support (a different PR hardware) to FME.

1) introduce a new PR sub feature to FME.
   driver knows it by different feature id, and create different fpga
   manager platform device, but may not be able to reuse dfl-fme-pr.c.

What would prevent reusing dfl-fme-pr.c?  It looks like this is 98% of
the way there and only needs a way of knowing which FPGA manager
driver to alloc here. I am hoping that a new PR sub feature could be
added and that dfl-fme-pr.c can be reused.

2) add a different PR hardware support to current PR sub feature.
   It requires hardware to add registers to indicate this is a different
   PR hardware than dfl-fme-mgr.c, and its register region information
   (e.g location and size). Then this dfl-fme-pr.c driver could read
   these information from hardware and create a different fpga manager
   platform device with correct resources.

So this dfl-fme-pr.c would have to know where some ID registers are
and the enumeration gets messier.  Some of the enumeration would be
DFL and some would be from information that is not in the DFL headers.
The DFL should contain the knowledge of which mgr to use.

I think current driver framework could support both cases above for sure.

quoted

(2) a FPGA manager that can be added by device tree in the case of a
platform that is using device tree.  I think this will be pretty
simple and can be done later when someone is actually bringing this
framework up on a FPGA running under device tree.  I'm thinking that
the base DFL device that reads the dfl data from hardware can have a
DT property that points to the FPGA manager.  That manager can be
saved somewhere handy like the pdata and passed down to this code,
which realizes it can use that existing device and doesn't need to
alloc a platform device.  But again, that's probably best done later.

Sure, we can discuss further when we really need it. Actually per my
understanding, if hardware describes itself clearly, we may not have to
use DT for fpga manager, as driver could create it automatically based
on information read from hardware. :)

DT exists for busses that don't have that kind of discovery.  For a
concrete example, consider how the Arria10 driver (socfpga-a10.c)
probe function is getting its two mmio spaces and clock.

quoted

+       if (!mgr)
+               return ERR_PTR(ret);
+
+       mgr->dev.parent = &fme->dev;
+
+       pres = platform_get_resource(fme, IORESOURCE_MEM,
+                                    feature->resource_index);
+       if (!pres) {
+               ret = -ENODEV;
+               goto create_mgr_err;
+       }
+
+       memset(&res, 0, sizeof(struct resource));
+
+       res.start = pres->start;
+       res.end = pres->end;
+       res.name = pres->name;
+       res.flags = IORESOURCE_MEM;
+
+       ret = platform_device_add_resources(mgr, &res, 1);
+       if (ret)
+               goto create_mgr_err;
+
+       ret = platform_device_add(mgr);
+       if (ret)
+               goto create_mgr_err;
+
+       return mgr;
+
+create_mgr_err:
+       platform_device_put(mgr);
+       return ERR_PTR(ret);
+}

quoted

diff --git a/drivers/fpga/dfl-fme-pr.h b/drivers/fpga/dfl-fme-pr.h
new file mode 100644
index 0000000..11bd001
--- /dev/null
+++ b/drivers/fpga/dfl-fme-pr.h

@@ -0,0 +1,113 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Header file for FPGA Management Engine (FME) Partial Reconfiguration Driver
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ */
+
+#ifndef __DFL_FME_PR_H
+#define __DFL_FME_PR_H
+
+#include <linux/platform_device.h>
+
+/**
+ * struct fme_region - FME fpga region data structure
+ *
+ * @region: platform device of the FPGA region.
+ * @node: used to link fme_region to a list.
+ * @port_id: indicate which port this region connected to.
+ */
+struct fme_region {
+       struct platform_device *region;
+       struct list_head node;
+       int port_id;
+};
+
+/**
+ * struct fme_region_pdata - platform data for FME region platform device.
+ *
+ * @mgr: platform device of the FPGA manager.
+ * @br: platform device of the FPGA bridge.
+ * @region_id: region id (same as port_id).
+ */
+struct fme_region_pdata {
+       struct platform_device *mgr;
+       struct platform_device *br;
+       int region_id;
+};
+
+/**
+ * struct fme_bridge - FME fpga bridge data structure
+ *
+ * @br: platform device of the FPGA bridge.
+ * @node: used to link fme_bridge to a list.
+ */
+struct fme_bridge {
+       struct platform_device *br;
+       struct list_head node;
+};
+
+/**
+ * struct fme_bridge_pdata - platform data for FME bridge platform device.
+ *
+ * @port: platform device of the port feature dev.
+ */
+struct fme_br_pdata {
+       struct platform_device *port;
+};
+
+#define FPGA_DFL_FME_MGR       "dfl-fme-mgr"
+#define FPGA_DFL_FME_BRIDGE    "dfl-fme-bridge"
+#define FPGA_DFL_FME_REGION    "dfl-fme-region"
+

Everything in dfl-fme-pr.h up to this point is good and general and
should remain in dfl-fme-pr.h.  The register #defines for this FME's
FPGA manager device below should be associated with the FPGA manager
driver.  Sorry if the way I stated that in the v3 review wasn't clear.

Actually I put the PR sub feature register set definitions in this header
file (dfl-fme-pr.h), because it's possible the driver (dfl-fme-pr.c) of
this PR sub feature access some of the registers in the future. e.g read
some PR sub feature registers to create different fpga manager platform
devices as I mentioned above.

That sounds like a workaround.  Since you're adding a new method of
enumeration, you should use that new method of enumeration to choose
which FPGA manager is being used.  Otherwise we are ending up with
multi-level enumeration, i.e. look at the DFL and then look at a
specific register location in the device.

I have to say this is only future consideration, and in this dfl-fme-pr.c
driver there is no code to access below registers currently. I can move all
of them to dfl-fme-mgr.h or dfl-fme-mgr.c in the next version if this is
preferred. : )

That sounds good.  That makes the mgr driver its own separate thing
which is what is supposed to happen in this framework.

quoted

+/* FME Partial Reconfiguration Sub Feature Register Set */
+#define FME_PR_DFH             0x0
+#define FME_PR_CTRL            0x8
+#define FME_PR_STS             0x10
+#define FME_PR_DATA            0x18
+#define FME_PR_ERR             0x20
+#define FME_PR_INTFC_ID_H      0xA8
+#define FME_PR_INTFC_ID_L      0xB0
+
+/* FME PR Control Register Bitfield */
+#define FME_PR_CTRL_PR_RST     BIT(0)  /* Reset PR engine */
+#define FME_PR_CTRL_PR_RSTACK  BIT(4)  /* Ack for PR engine reset */
+#define FME_PR_CTRL_PR_RGN_ID  GENMASK_ULL(9, 7)       /* PR Region ID */
+#define FME_PR_CTRL_PR_START   BIT(12) /* Start to request for PR service */
+#define FME_PR_CTRL_PR_COMPLETE        BIT(13) /* PR data push complete notification */
+
+/* FME PR Status Register Bitfield */
+/* Number of available entries in HW queue inside the PR engine. */
+#define FME_PR_STS_PR_CREDIT   GENMASK_ULL(8, 0)
+#define FME_PR_STS_PR_STS      BIT(16) /* PR operation status */
+#define FME_PR_STS_PR_STS_IDLE 0
+#define FME_PR_STS_PR_CTRLR_STS        GENMASK_ULL(22, 20)     /* Controller status */
+#define FME_PR_STS_PR_HOST_STS GENMASK_ULL(27, 24)     /* PR host status */
+
+/* FME PR Data Register Bitfield */
+/* PR data from the raw-binary file. */
+#define FME_PR_DATA_PR_DATA_RAW        GENMASK_ULL(32, 0)
+
+/* FME PR Error Register */
+/* PR Operation errors detected. */
+#define FME_PR_ERR_OPERATION_ERR       BIT(0)
+/* CRC error detected. */
+#define FME_PR_ERR_CRC_ERR             BIT(1)
+/* Incompatible PR bitstream detected. */
+#define FME_PR_ERR_INCOMPATIBLE_BS     BIT(2)
+/* PR data push protocol violated. */
+#define FME_PR_ERR_PROTOCOL_ERR                BIT(3)
+/* PR data fifo overflow error detected */
+#define FME_PR_ERR_FIFO_OVERFLOW       BIT(4)

This stuff is specific to this FPGA manager device, so it should
either be in the dfl-fme-mgr.c or in a dfl-fme-mgr.h

same as above, I can fix this in the next version.

Thanks
Hao

Thanks,
Alan

Re: [PATCH v4 14/24] fpga: dfl: fme: add partial reconfiguration sub feature support

From: Wu Hao <hidden>
Date: 2018-03-07 04:49:10

On Tue, Mar 06, 2018 at 12:29:35PM -0600, Alan Tull wrote:

  On Mon, Mar 5, 2018 at 8:08 PM, Wu Hao [off-list ref] wrote:

quoted

On Mon, Mar 05, 2018 at 04:46:02PM -0600, Alan Tull wrote:

quoted

On Tue, Feb 13, 2018 at 3:24 AM, Wu Hao [off-list ref] wrote:

Hi Hao,

Hi Alan,

Thanks for the comments.

quoted

We are going to want to be able use different FPGA managers with this
framework.  The different manager may be part of a different FME in
fabric or it may be a hardware FPGA manager.  Fortunately, at this
point now the changes, noted below, to get there are pretty small.

Yes, understand these could be some cases that FME having different PR
hardware.

Or supporting reduced FME plus hardware-based FPGA manager.

Just to re-emphasize, the basic intent of the FPGA manager subsystem
in the first place is to have FPGA managers separate from higher level
frameworks so that the higher level frameworks will be able to able to
use different FPGAs.

quoted

+/**
+ * fpga_fme_create_mgr - create fpga mgr platform device as child device
+ *
+ * @pdata: fme platform_device's pdata
+ *
+ * Return: mgr platform device if successful, and error code otherwise.
+ */
+static struct platform_device *
+fpga_fme_create_mgr(struct feature_platform_data *pdata)
+{
+       struct platform_device *mgr, *fme = pdata->dev;
+       struct feature *feature;
+       struct resource res;
+       struct resource *pres;
+       int ret = -ENOMEM;
+
+       feature = get_feature_by_id(&pdata->dev->dev, FME_FEATURE_ID_PR_MGMT);
+       if (!feature)
+               return ERR_PTR(-ENODEV);
+
+       /*
+        * Each FME has only one fpga-mgr, so allocate platform device using
+        * the same FME platform device id.
+        */
+       mgr = platform_device_alloc(FPGA_DFL_FME_MGR, fme->id);

At this point, the framework is assuming all FME's include the same
FPGA manager device which would use the driver in dfl-fme-mgr.c.

I'm thinking of two cases where the manager isn't the same as a
dfl-fme-mgr.c manager are a bit different:

(1) a FME-based FPGA manager, but different implementation, different
registers.  The constraint is that the port implementation has to be
similar enough to use the rest of the base FME code.   I am wondering
if the FPGA manager can be added to the DFL.  At this point, the DFL
would drive which FPGA manager is alloc'd.   That way the user gets to
use all this code in dfl-fme-pr.c but with their FPGA manager.

Actually I'm not sure how this will be implemented in the hardware in the
future, but from my understanding, there may be several methods to add this
support (a different PR hardware) to FME.

1) introduce a new PR sub feature to FME.
   driver knows it by different feature id, and create different fpga
   manager platform device, but may not be able to reuse dfl-fme-pr.c.

What would prevent reusing dfl-fme-pr.c?  It looks like this is 98% of
the way there and only needs a way of knowing which FPGA manager
driver to alloc here. I am hoping that a new PR sub feature could be
added and that dfl-fme-pr.c can be reused.

It depeneds on how hardware implements it. : )
I agree that if it follows the same way of current PR sub feature, then we
could reuse the dfl-fme-pr.c for sure.

quoted

2) add a different PR hardware support to current PR sub feature.
   It requires hardware to add registers to indicate this is a different
   PR hardware than dfl-fme-mgr.c, and its register region information
   (e.g location and size). Then this dfl-fme-pr.c driver could read
   these information from hardware and create a different fpga manager
   platform device with correct resources.

So this dfl-fme-pr.c would have to know where some ID registers are
and the enumeration gets messier.  Some of the enumeration would be
DFL and some would be from information that is not in the DFL headers.
The DFL should contain the knowledge of which mgr to use.

Actually I don't know how hardware will implement this in the future, but I
just listed my ideas here. Per my understanding, driver (reuse dfl-fme-pr.c)
needs some more information to decide which platform device to create (for
fpga manager).

1) introduce a new PR sub feature. Then it has a different private feature
id in DFH (Device Feature Header). driver could use this id to create a
different platform device.

2) introduce some registers inside the current PR sub feature. Then driver
could read these registers to know which platform device to create for fpga
manager.

I think either 1) or 2) will only require small changes to current driver
code, I don't have any concern on supporting different PR hardware. :)

I understand your concern on case 2), everybody who wants to reuse dfl-fme-pr.c
needs to implement ID register which is not defined by DFL, so I guess we
should suggest 1).

quoted

I think current driver framework could support both cases above for sure.

quoted

(2) a FPGA manager that can be added by device tree in the case of a
platform that is using device tree.  I think this will be pretty
simple and can be done later when someone is actually bringing this
framework up on a FPGA running under device tree.  I'm thinking that
the base DFL device that reads the dfl data from hardware can have a
DT property that points to the FPGA manager.  That manager can be
saved somewhere handy like the pdata and passed down to this code,
which realizes it can use that existing device and doesn't need to
alloc a platform device.  But again, that's probably best done later.

Sure, we can discuss further when we really need it. Actually per my
understanding, if hardware describes itself clearly, we may not have to
use DT for fpga manager, as driver could create it automatically based
on information read from hardware. :)

DT exists for busses that don't have that kind of discovery.  For a
concrete example, consider how the Arria10 driver (socfpga-a10.c)
probe function is getting its two mmio spaces and clock.

I see. If we have to create fpga mgr using DT some cases, it makes sense
that we just link the existing one instead.

quoted

+       if (!mgr)
+               return ERR_PTR(ret);
+
+       mgr->dev.parent = &fme->dev;
+
+       pres = platform_get_resource(fme, IORESOURCE_MEM,
+                                    feature->resource_index);
+       if (!pres) {
+               ret = -ENODEV;
+               goto create_mgr_err;
+       }
+
+       memset(&res, 0, sizeof(struct resource));
+
+       res.start = pres->start;
+       res.end = pres->end;
+       res.name = pres->name;
+       res.flags = IORESOURCE_MEM;
+
+       ret = platform_device_add_resources(mgr, &res, 1);
+       if (ret)
+               goto create_mgr_err;
+
+       ret = platform_device_add(mgr);
+       if (ret)
+               goto create_mgr_err;
+
+       return mgr;
+
+create_mgr_err:
+       platform_device_put(mgr);
+       return ERR_PTR(ret);
+}

quoted

diff --git a/drivers/fpga/dfl-fme-pr.h b/drivers/fpga/dfl-fme-pr.h
new file mode 100644
index 0000000..11bd001
--- /dev/null
+++ b/drivers/fpga/dfl-fme-pr.h

@@ -0,0 +1,113 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Header file for FPGA Management Engine (FME) Partial Reconfiguration Driver
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ */
+
+#ifndef __DFL_FME_PR_H
+#define __DFL_FME_PR_H
+
+#include <linux/platform_device.h>
+
+/**
+ * struct fme_region - FME fpga region data structure
+ *
+ * @region: platform device of the FPGA region.
+ * @node: used to link fme_region to a list.
+ * @port_id: indicate which port this region connected to.
+ */
+struct fme_region {
+       struct platform_device *region;
+       struct list_head node;
+       int port_id;
+};
+
+/**
+ * struct fme_region_pdata - platform data for FME region platform device.
+ *
+ * @mgr: platform device of the FPGA manager.
+ * @br: platform device of the FPGA bridge.
+ * @region_id: region id (same as port_id).
+ */
+struct fme_region_pdata {
+       struct platform_device *mgr;
+       struct platform_device *br;
+       int region_id;
+};
+
+/**
+ * struct fme_bridge - FME fpga bridge data structure
+ *
+ * @br: platform device of the FPGA bridge.
+ * @node: used to link fme_bridge to a list.
+ */
+struct fme_bridge {
+       struct platform_device *br;
+       struct list_head node;
+};
+
+/**
+ * struct fme_bridge_pdata - platform data for FME bridge platform device.
+ *
+ * @port: platform device of the port feature dev.
+ */
+struct fme_br_pdata {
+       struct platform_device *port;
+};
+
+#define FPGA_DFL_FME_MGR       "dfl-fme-mgr"
+#define FPGA_DFL_FME_BRIDGE    "dfl-fme-bridge"
+#define FPGA_DFL_FME_REGION    "dfl-fme-region"
+

Everything in dfl-fme-pr.h up to this point is good and general and
should remain in dfl-fme-pr.h.  The register #defines for this FME's
FPGA manager device below should be associated with the FPGA manager
driver.  Sorry if the way I stated that in the v3 review wasn't clear.

Actually I put the PR sub feature register set definitions in this header
file (dfl-fme-pr.h), because it's possible the driver (dfl-fme-pr.c) of
this PR sub feature access some of the registers in the future. e.g read
some PR sub feature registers to create different fpga manager platform
devices as I mentioned above.

That sounds like a workaround.  Since you're adding a new method of
enumeration, you should use that new method of enumeration to choose
which FPGA manager is being used.  Otherwise we are ending up with
multi-level enumeration, i.e. look at the DFL and then look at a
specific register location in the device.

quoted

I have to say this is only future consideration, and in this dfl-fme-pr.c
driver there is no code to access below registers currently. I can move all
of them to dfl-fme-mgr.h or dfl-fme-mgr.c in the next version if this is
preferred. : )

That sounds good.  That makes the mgr driver its own separate thing
which is what is supposed to happen in this framework.

Sure, will fix this.

Thanks
Hao

quoted

+/* FME Partial Reconfiguration Sub Feature Register Set */
+#define FME_PR_DFH             0x0
+#define FME_PR_CTRL            0x8
+#define FME_PR_STS             0x10
+#define FME_PR_DATA            0x18
+#define FME_PR_ERR             0x20
+#define FME_PR_INTFC_ID_H      0xA8
+#define FME_PR_INTFC_ID_L      0xB0
+
+/* FME PR Control Register Bitfield */
+#define FME_PR_CTRL_PR_RST     BIT(0)  /* Reset PR engine */
+#define FME_PR_CTRL_PR_RSTACK  BIT(4)  /* Ack for PR engine reset */
+#define FME_PR_CTRL_PR_RGN_ID  GENMASK_ULL(9, 7)       /* PR Region ID */
+#define FME_PR_CTRL_PR_START   BIT(12) /* Start to request for PR service */
+#define FME_PR_CTRL_PR_COMPLETE        BIT(13) /* PR data push complete notification */
+
+/* FME PR Status Register Bitfield */
+/* Number of available entries in HW queue inside the PR engine. */
+#define FME_PR_STS_PR_CREDIT   GENMASK_ULL(8, 0)
+#define FME_PR_STS_PR_STS      BIT(16) /* PR operation status */
+#define FME_PR_STS_PR_STS_IDLE 0
+#define FME_PR_STS_PR_CTRLR_STS        GENMASK_ULL(22, 20)     /* Controller status */
+#define FME_PR_STS_PR_HOST_STS GENMASK_ULL(27, 24)     /* PR host status */
+
+/* FME PR Data Register Bitfield */
+/* PR data from the raw-binary file. */
+#define FME_PR_DATA_PR_DATA_RAW        GENMASK_ULL(32, 0)
+
+/* FME PR Error Register */
+/* PR Operation errors detected. */
+#define FME_PR_ERR_OPERATION_ERR       BIT(0)
+/* CRC error detected. */
+#define FME_PR_ERR_CRC_ERR             BIT(1)
+/* Incompatible PR bitstream detected. */
+#define FME_PR_ERR_INCOMPATIBLE_BS     BIT(2)
+/* PR data push protocol violated. */
+#define FME_PR_ERR_PROTOCOL_ERR                BIT(3)
+/* PR data fifo overflow error detected */
+#define FME_PR_ERR_FIFO_OVERFLOW       BIT(4)

This stuff is specific to this FPGA manager device, so it should
either be in the dfl-fme-mgr.c or in a dfl-fme-mgr.h

same as above, I can fix this in the next version.

Thanks
Hao

Thanks,
Alan

Re: [PATCH v4 14/24] fpga: dfl: fme: add partial reconfiguration sub feature support

From: <hidden>
Date: 2018-03-11 20:09:35


On Mon, 5 Mar 2018, Alan Tull wrote:


Hi Hao,

I do think we should consider different hw implementations with this code
because it does look like most of it is generic.  Specifically, I think
we should consider DFH based fpga images that have been shipped already,
and I think we need to consider new hardware implementations as well.
Full disclosure, I am particularly interested in porting to a new hw
implementation for partial reconfiguration.

Please see some comments below.

Matthew Gerlach

On Tue, Feb 13, 2018 at 3:24 AM, Wu Hao [off-list ref] wrote:

Hi Hao,

We are going to want to be able use different FPGA managers with this
framework.  The different manager may be part of a different FME in
fabric or it may be a hardware FPGA manager.  Fortunately, at this
point now the changes, noted below, to get there are pretty small.

quoted

From: Kang Luwei <redacted>

Partial Reconfiguration (PR) is the most important function for FME. It
allows reconfiguration for given Port/Accelerated Function Unit (AFU).

It creates platform devices for fpga-mgr, fpga-regions and fpga-bridges,
and invokes fpga-region's interface (fpga_region_program_fpga) for PR
operation once PR request received via ioctl. Below user space interface
is exposed by this sub feature.

Ioctl interface:
* FPGA_FME_PORT_PR
  Do partial reconfiguration per information from userspace, including
  target port(AFU), buffer size and address info. It returns error code
  to userspace if failed. For detailed PR error information, user needs
  to read fpga-mgr's status sysfs interface.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Kang Luwei <redacted>
Signed-off-by: Xiao Guangrong <redacted>
Signed-off-by: Wu Hao <redacted>
---
v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
    switched to GPLv2 license.
    removed status from FPGA_FME_PORT_PR ioctl data structure.
    added platform devices creation for fpga-mgr/fpga-region/fpga-bridge.
    switched to fpga-region interface fpga_region_program_fpga for PR.
    fixed comments from Alan Tull on FPGA_MGR_PARTIAL_RECONFIG flag usage.
    fixed kbuild warnings.
v3: rename driver files to dfl-fme-*.
    rebase due to fpga APIs change.
    replace bitfields.
    switch to fpga_cdev_find_port to find port device.
v4: rebase and correct comments for some function.
    fix SPDX license issue.
    remove unnecessary input parameter for destroy_bridge/region function.
    add dfl-fme-pr.h for PR sub feature data structure and registers.
---
 drivers/fpga/Makefile         |   2 +-
 drivers/fpga/dfl-fme-main.c   |  45 +++-
 drivers/fpga/dfl-fme-pr.c     | 497 ++++++++++++++++++++++++++++++++++++++++++
 drivers/fpga/dfl-fme-pr.h     | 113 ++++++++++
 drivers/fpga/dfl-fme.h        |  38 ++++
 include/uapi/linux/fpga-dfl.h |  27 +++
 6 files changed, 720 insertions(+), 2 deletions(-)
 create mode 100644 drivers/fpga/dfl-fme-pr.c
 create mode 100644 drivers/fpga/dfl-fme-pr.h
 create mode 100644 drivers/fpga/dfl-fme.h

diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index fbd1c85..3c44fc9 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile

@@ -32,7 +32,7 @@ obj-$(CONFIG_OF_FPGA_REGION)          += of-fpga-region.o
 obj-$(CONFIG_FPGA_DFL)                 += dfl.o
 obj-$(CONFIG_FPGA_DFL_FME)             += dfl-fme.o

-dfl-fme-objs := dfl-fme-main.o
+dfl-fme-objs := dfl-fme-main.o dfl-fme-pr.o

 # Drivers for FPGAs which implement DFL
 obj-$(CONFIG_FPGA_DFL_PCI)             += dfl-pci.o

diff --git a/drivers/fpga/dfl-fme-main.c b/drivers/fpga/dfl-fme-main.c
index 1a9929c..967a44c 100644
--- a/drivers/fpga/dfl-fme-main.c
+++ b/drivers/fpga/dfl-fme-main.c

@@ -19,6 +19,7 @@
 #include <linux/fpga-dfl.h>

 #include "dfl.h"
+#include "dfl-fme.h"

 static ssize_t ports_num_show(struct device *dev,
                              struct device_attribute *attr, char *buf)

@@ -111,6 +112,10 @@ static struct feature_driver fme_feature_drvs[] = {
                .ops = &fme_hdr_ops,
        },
        {
+               .id = FME_FEATURE_ID_PR_MGMT,
+               .ops = &pr_mgmt_ops,
+       },
+       {
                .ops = NULL,
        },
 };

@@ -194,14 +199,49 @@ static const struct file_operations fme_fops = {
        .unlocked_ioctl = fme_ioctl,
 };

+static int fme_dev_init(struct platform_device *pdev)
+{
+       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+       struct fpga_fme *fme;
+
+       fme = devm_kzalloc(&pdev->dev, sizeof(*fme), GFP_KERNEL);
+       if (!fme)
+               return -ENOMEM;
+
+       fme->pdata = pdata;
+
+       mutex_lock(&pdata->lock);
+       fpga_pdata_set_private(pdata, fme);
+       mutex_unlock(&pdata->lock);
+
+       return 0;
+}
+
+static void fme_dev_destroy(struct platform_device *pdev)
+{
+       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+       struct fpga_fme *fme;
+
+       mutex_lock(&pdata->lock);
+       fme = fpga_pdata_get_private(pdata);
+       fpga_pdata_set_private(pdata, NULL);
+       mutex_unlock(&pdata->lock);
+
+       devm_kfree(&pdev->dev, fme);

Is this needed?  This is called when the device is being destroyed anyway.

quoted

+}
+
 static int fme_probe(struct platform_device *pdev)
 {
        int ret;

-       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
+       ret = fme_dev_init(pdev);
        if (ret)
                goto exit;

+       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
+       if (ret)
+               goto dev_destroy;
+
        ret = fpga_register_dev_ops(pdev, &fme_fops, THIS_MODULE);
        if (ret)
                goto feature_uinit;

@@ -210,6 +250,8 @@ static int fme_probe(struct platform_device *pdev)

 feature_uinit:
        fpga_dev_feature_uinit(pdev);
+dev_destroy:
+       fme_dev_destroy(pdev);
 exit:
        return ret;
 }

@@ -218,6 +260,7 @@ static int fme_remove(struct platform_device *pdev)
 {
        fpga_dev_feature_uinit(pdev);
        fpga_unregister_dev_ops(pdev);
+       fme_dev_destroy(pdev);

        return 0;
 }

diff --git a/drivers/fpga/dfl-fme-pr.c b/drivers/fpga/dfl-fme-pr.c
new file mode 100644
index 0000000..526e90b
--- /dev/null
+++ b/drivers/fpga/dfl-fme-pr.c

@@ -0,0 +1,497 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Driver for FPGA Management Engine (FME) Partial Reconfiguration
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Christopher Rauer <christopher.rauer@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ */
+
+#include <linux/types.h>
+#include <linux/device.h>
+#include <linux/vmalloc.h>
+#include <linux/uaccess.h>
+#include <linux/fpga/fpga-mgr.h>
+#include <linux/fpga/fpga-bridge.h>
+#include <linux/fpga/fpga-region.h>
+#include <linux/fpga-dfl.h>
+
+#include "dfl.h"
+#include "dfl-fme.h"
+#include "dfl-fme-pr.h"
+
+static struct fme_region *
+find_fme_region_by_port_id(struct fpga_fme *fme, int port_id)
+{
+       struct fme_region *fme_region;
+
+       list_for_each_entry(fme_region, &fme->region_list, node)
+               if (fme_region->port_id == port_id)
+                       return fme_region;
+
+       return NULL;
+}
+
+static int fpga_fme_region_match(struct device *dev, const void *data)
+{
+       return dev->parent == data;
+}
+
+static struct fpga_region *
+fpga_fme_region_find(struct fpga_fme *fme, int port_id)
+{
+       struct fme_region *fme_region;
+       struct fpga_region *region;
+
+       fme_region = find_fme_region_by_port_id(fme, port_id);
+       if (!fme_region)
+               return NULL;
+
+       region = fpga_region_class_find(NULL, &fme_region->region->dev,
+                                       fpga_fme_region_match);
+       if (!region)
+               return NULL;
+
+       return region;
+}
+
+static int fme_pr(struct platform_device *pdev, unsigned long arg)
+{
+       void __user *argp = (void __user *)arg;
+       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+       struct fpga_fme *fme;
+       struct fpga_image_info *info;
+       struct fpga_region *region;
+       struct fpga_fme_port_pr port_pr;
+       unsigned long minsz;
+       void __iomem *fme_hdr;
+       void *buf = NULL;
+       int ret = 0;
+       u64 v;
+
+       minsz = offsetofend(struct fpga_fme_port_pr, buffer_address);
+
+       if (copy_from_user(&port_pr, argp, minsz))
+               return -EFAULT;
+
+       if (port_pr.argsz < minsz || port_pr.flags)
+               return -EINVAL;
+
+       if (!IS_ALIGNED(port_pr.buffer_size, 4))
+               return -EINVAL;
+
+       /* get fme header region */
+       fme_hdr = get_feature_ioaddr_by_id(&pdev->dev,
+                                          FME_FEATURE_ID_HEADER);
+
+       /* check port id */
+       v = readq(fme_hdr + FME_HDR_CAP);
+       if (port_pr.port_id >= FIELD_GET(FME_CAP_NUM_PORTS, v)) {
+               dev_dbg(&pdev->dev, "port number more than maximum\n");
+               return -EINVAL;
+       }
+
+       if (!access_ok(VERIFY_READ,
+                      (void __user *)(unsigned long)port_pr.buffer_address,
+                      port_pr.buffer_size))
+               return -EFAULT;
+
+       buf = vmalloc(port_pr.buffer_size);
+       if (!buf)
+               return -ENOMEM;
+
+       if (copy_from_user(buf,
+                          (void __user *)(unsigned long)port_pr.buffer_address,
+                          port_pr.buffer_size)) {
+               ret = -EFAULT;
+               goto free_exit;
+       }
+
+       /* prepare fpga_image_info for PR */
+       info = fpga_image_info_alloc(&pdev->dev);
+       if (!info) {
+               ret = -ENOMEM;
+               goto free_exit;
+       }
+
+       info->flags |= FPGA_MGR_PARTIAL_RECONFIG;
+
+       mutex_lock(&pdata->lock);
+       fme = fpga_pdata_get_private(pdata);
+       /* fme device has been unregistered. */
+       if (!fme) {
+               ret = -EINVAL;
+               goto unlock_exit;
+       }
+
+       region = fpga_fme_region_find(fme, port_pr.port_id);
+       if (!region) {
+               ret = -EINVAL;
+               goto unlock_exit;
+       }
+
+       fpga_image_info_free(region->info);
+
+       info->buf = buf;
+       info->count = port_pr.buffer_size;
+       info->region_id = port_pr.port_id;
+       region->info = info;
+
+       ret = fpga_region_program_fpga(region);
+
+       if (region->get_bridges)
+               fpga_bridges_put(&region->bridge_list);
+
+       put_device(&region->dev);
+unlock_exit:
+       mutex_unlock(&pdata->lock);
+free_exit:
+       vfree(buf);
+       if (copy_to_user((void __user *)arg, &port_pr, minsz))
+               return -EFAULT;
+
+       return ret;
+}
+
+/**
+ * fpga_fme_create_mgr - create fpga mgr platform device as child device
+ *
+ * @pdata: fme platform_device's pdata
+ *
+ * Return: mgr platform device if successful, and error code otherwise.
+ */
+static struct platform_device *
+fpga_fme_create_mgr(struct feature_platform_data *pdata)
+{
+       struct platform_device *mgr, *fme = pdata->dev;
+       struct feature *feature;
+       struct resource res;
+       struct resource *pres;
+       int ret = -ENOMEM;
+
+       feature = get_feature_by_id(&pdata->dev->dev, FME_FEATURE_ID_PR_MGMT);
+       if (!feature)
+               return ERR_PTR(-ENODEV);
+
+       /*
+        * Each FME has only one fpga-mgr, so allocate platform device using
+        * the same FME platform device id.
+        */
+       mgr = platform_device_alloc(FPGA_DFL_FME_MGR, fme->id);

At this point, the framework is assuming all FME's include the same
FPGA manager device which would use the driver in dfl-fme-mgr.c.

I'm thinking of two cases where the manager isn't the same as a
dfl-fme-mgr.c manager are a bit different:

(1) a FME-based FPGA manager, but different implementation, different
registers.  The constraint is that the port implementation has to be
similar enough to use the rest of the base FME code.   I am wondering
if the FPGA manager can be added to the DFL.  At this point, the DFL
would drive which FPGA manager is alloc'd.   That way the user gets to
use all this code in dfl-fme-pr.c but with their FPGA manager.

I am thinking of the case of porting to a new hardware implementation
for Partial Reconfiguration.  Since the new hardware is completely 
different at the register level, we would need new a new feature id.  The
fme init code for this new feature should be able to call the generic
code here and pass in the fgpa_mgr_ops that are hardware specific.

(2) a FPGA manager that can be added by device tree in the case of a
platform that is using device tree.  I think this will be pretty
simple and can be done later when someone is actually bringing this
framework up on a FPGA running under device tree.  I'm thinking that
the base DFL device that reads the dfl data from hardware can have a
DT property that points to the FPGA manager.  That manager can be
saved somewhere handy like the pdata and passed down to this code,
which realizes it can use that existing device and doesn't need to
alloc a platform device.  But again, that's probably best done later.

quoted

+       if (!mgr)
+               return ERR_PTR(ret);
+
+       mgr->dev.parent = &fme->dev;
+
+       pres = platform_get_resource(fme, IORESOURCE_MEM,
+                                    feature->resource_index);
+       if (!pres) {
+               ret = -ENODEV;
+               goto create_mgr_err;
+       }
+
+       memset(&res, 0, sizeof(struct resource));
+
+       res.start = pres->start;
+       res.end = pres->end;
+       res.name = pres->name;
+       res.flags = IORESOURCE_MEM;
+
+       ret = platform_device_add_resources(mgr, &res, 1);
+       if (ret)
+               goto create_mgr_err;
+
+       ret = platform_device_add(mgr);
+       if (ret)
+               goto create_mgr_err;
+
+       return mgr;
+
+create_mgr_err:
+       platform_device_put(mgr);
+       return ERR_PTR(ret);
+}
+
+/**
+ * fpga_fme_destroy_mgr - destroy fpga mgr platform device
+ * @pdata: fme platform device's pdata
+ */
+static void fpga_fme_destroy_mgr(struct feature_platform_data *pdata)
+{
+       struct fpga_fme *priv = fpga_pdata_get_private(pdata);
+
+       platform_device_unregister(priv->mgr);
+}
+
+/**
+ * fpga_fme_create_bridge - create fme fpga bridge platform device as child
+ *
+ * @pdata: fme platform device's pdata
+ * @port_id: port id for the bridge to be created.
+ *
+ * Return: bridge platform device if successful, and error code otherwise.
+ */
+static struct fme_bridge *
+fpga_fme_create_bridge(struct feature_platform_data *pdata, int port_id)
+{
+       struct device *dev = &pdata->dev->dev;
+       struct fme_br_pdata br_pdata;
+       struct fme_bridge *fme_br;
+       int ret = -ENOMEM;
+
+       fme_br = devm_kzalloc(dev, sizeof(*fme_br), GFP_KERNEL);
+       if (!fme_br)
+               return ERR_PTR(ret);
+
+       br_pdata.port = fpga_cdev_find_port(fpga_pdata_to_fpga_cdev(pdata),
+                                           &port_id, fpga_port_check_id);
+       if (!br_pdata.port)
+               return ERR_PTR(-ENODEV);
+
+       /*
+        * Each FPGA device may have more than one port, so allocate platform
+        * device using the same port platform device id.
+        */
+       fme_br->br = platform_device_alloc(FPGA_DFL_FME_BRIDGE,
+                                          br_pdata.port->id);
+       if (!fme_br->br) {
+               ret = -ENOMEM;
+               goto create_br_err;
+       }
+
+       fme_br->br->dev.parent = dev;
+
+       ret = platform_device_add_data(fme_br->br, &br_pdata, sizeof(br_pdata));
+       if (ret)
+               goto create_br_err;
+
+       ret = platform_device_add(fme_br->br);
+       if (ret)
+               goto create_br_err;
+
+       return fme_br;
+
+create_br_err:
+       platform_device_put(fme_br->br);
+       put_device(&br_pdata.port->dev);
+       return ERR_PTR(ret);
+}
+
+/**
+ * fpga_fme_destroy_bridge - destroy fpga bridge platform device
+ * @fme_br: fme bridge to destroy
+ */
+static void fpga_fme_destroy_bridge(struct fme_bridge *fme_br)
+{
+       struct fme_br_pdata *br_pdata = dev_get_platdata(&fme_br->br->dev);
+
+       put_device(&br_pdata->port->dev);
+       platform_device_unregister(fme_br->br);
+}
+
+/**
+ * fpga_fme_destroy_bridge - destroy all fpga bridge platform device
+ * @pdata: fme platform device's pdata
+ */
+static void fpga_fme_destroy_bridges(struct feature_platform_data *pdata)
+{
+       struct fpga_fme *priv = fpga_pdata_get_private(pdata);
+       struct fme_bridge *fbridge, *tmp;
+
+       list_for_each_entry_safe(fbridge, tmp, &priv->bridge_list, node) {
+               list_del(&fbridge->node);
+               fpga_fme_destroy_bridge(fbridge);
+       }
+}
+
+/**
+ * fpga_fme_create_region - create fpga region platform device as child
+ *
+ * @pdata: fme platform device's pdata
+ * @mgr: mgr platform device needed for region
+ * @br: br platform device needed for region
+ * @port_id: port id
+ *
+ * Return: fme region if successful, and error code otherwise.
+ */
+static struct fme_region *
+fpga_fme_create_region(struct feature_platform_data *pdata,
+                      struct platform_device *mgr,
+                      struct platform_device *br, int port_id)
+{
+       struct device *dev = &pdata->dev->dev;
+       struct fme_region_pdata region_pdata;
+       struct fme_region *fme_region;
+       int ret = -ENOMEM;
+
+       fme_region = devm_kzalloc(dev, sizeof(*fme_region), GFP_KERNEL);
+       if (!fme_region)
+               return ERR_PTR(ret);
+
+       region_pdata.mgr = mgr;
+       region_pdata.br = br;
+
+       /*
+        * Each FPGA device may have more than one port, so allocate platform
+        * device using the same port platform device id.
+        */
+       fme_region->region = platform_device_alloc(FPGA_DFL_FME_REGION, br->id);
+       if (!fme_region->region)
+               return ERR_PTR(ret);
+
+       fme_region->region->dev.parent = dev;
+
+       ret = platform_device_add_data(fme_region->region, &region_pdata,
+                                      sizeof(region_pdata));
+       if (ret)
+               goto create_region_err;
+
+       ret = platform_device_add(fme_region->region);
+       if (ret)
+               goto create_region_err;
+
+       fme_region->port_id = port_id;
+
+       return fme_region;
+
+create_region_err:
+       platform_device_put(fme_region->region);
+       return ERR_PTR(ret);
+}
+
+/**
+ * fpga_fme_destroy_region - destroy fme region
+ * @fme_region: fme region to destroy
+ */
+static void fpga_fme_destroy_region(struct fme_region *fme_region)
+{
+       platform_device_unregister(fme_region->region);
+}
+
+/**
+ * fpga_fme_destroy_regions - destroy all fme regions
+ * @pdata: fme platform device's pdata
+ */
+static void fpga_fme_destroy_regions(struct feature_platform_data *pdata)
+{
+       struct fpga_fme *priv = fpga_pdata_get_private(pdata);
+       struct fme_region *fme_region, *tmp;
+
+       list_for_each_entry_safe(fme_region, tmp, &priv->region_list, node) {
+               list_del(&fme_region->node);
+               fpga_fme_destroy_region(fme_region);
+       }
+}
+
+static int pr_mgmt_init(struct platform_device *pdev, struct feature *feature)
+{
+       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+       void __iomem *fme_hdr;
+       struct platform_device *mgr;
+       struct fme_region *fme_region;
+       struct fme_bridge *fme_br;
+       struct fpga_fme *priv;
+       int ret = -ENODEV, i = 0;
+       u64 fme_cap, port_offset;
+
+       fme_hdr = get_feature_ioaddr_by_id(&pdev->dev,
+                                          FME_FEATURE_ID_HEADER);
+
+       mutex_lock(&pdata->lock);
+       priv = fpga_pdata_get_private(pdata);
+
+       /* Initialize the region and bridge sub device list */
+       INIT_LIST_HEAD(&priv->region_list);
+       INIT_LIST_HEAD(&priv->bridge_list);
+
+       /* Create fpga mgr platform device */
+       mgr = fpga_fme_create_mgr(pdata);
+       if (IS_ERR(mgr)) {
+               dev_err(&pdev->dev, "fail to create fpga mgr pdev\n");
+               goto unlock;
+       }
+
+       priv->mgr = mgr;
+
+       /* Read capability register to check number of regions and bridges */
+       fme_cap = readq(fme_hdr + FME_HDR_CAP);

I don't think this capability field exists in currently deployed FPGA 
images using DFH.  I believe this difference requires a new feature id
to differentiate between deployed FPGA images and images with this new 
"feature".

quoted

+       for (; i < FIELD_GET(FME_CAP_NUM_PORTS, fme_cap); i++) {
+               port_offset = readq(fme_hdr + FME_HDR_PORT_OFST(i));
+               if (!(port_offset & FME_PORT_OFST_IMP))
+                       continue;
+
+               /* Create bridge for each port */
+               fme_br = fpga_fme_create_bridge(pdata, i);
+               if (IS_ERR(fme_br)) {
+                       ret = PTR_ERR(fme_br);
+                       goto destroy_region;
+               }
+
+               list_add(&fme_br->node, &priv->bridge_list);
+
+               /* Create region for each port */
+               fme_region = fpga_fme_create_region(pdata, mgr, fme_br->br, i);
+               if (!fme_region) {
+                       ret = PTR_ERR(fme_region);
+                       goto destroy_region;
+               }
+
+               list_add(&fme_region->node, &priv->region_list);
+       }
+       mutex_unlock(&pdata->lock);
+
+       return 0;
+
+destroy_region:
+       fpga_fme_destroy_regions(pdata);
+       fpga_fme_destroy_bridges(pdata);
+       fpga_fme_destroy_mgr(pdata);
+unlock:
+       mutex_unlock(&pdata->lock);
+       return ret;
+}
+
+static void pr_mgmt_uinit(struct platform_device *pdev, struct feature *feature)
+{
+       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+       struct fpga_fme *priv;
+
+       mutex_lock(&pdata->lock);
+       priv = fpga_pdata_get_private(pdata);
+
+       fpga_fme_destroy_regions(pdata);
+       fpga_fme_destroy_bridges(pdata);
+       fpga_fme_destroy_mgr(pdata);
+       mutex_unlock(&pdata->lock);
+}
+
+static long fme_pr_ioctl(struct platform_device *pdev, struct feature *feature,
+                        unsigned int cmd, unsigned long arg)
+{
+       long ret;
+
+       switch (cmd) {
+       case FPGA_FME_PORT_PR:
+               ret = fme_pr(pdev, arg);
+               break;
+       default:
+               ret = -ENODEV;
+       }
+
+       return ret;
+}
+
+const struct feature_ops pr_mgmt_ops = {
+       .init = pr_mgmt_init,
+       .uinit = pr_mgmt_uinit,
+       .ioctl = fme_pr_ioctl,
+};

diff --git a/drivers/fpga/dfl-fme-pr.h b/drivers/fpga/dfl-fme-pr.h
new file mode 100644
index 0000000..11bd001
--- /dev/null
+++ b/drivers/fpga/dfl-fme-pr.h

@@ -0,0 +1,113 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Header file for FPGA Management Engine (FME) Partial Reconfiguration Driver
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ */
+
+#ifndef __DFL_FME_PR_H
+#define __DFL_FME_PR_H
+
+#include <linux/platform_device.h>
+
+/**
+ * struct fme_region - FME fpga region data structure
+ *
+ * @region: platform device of the FPGA region.
+ * @node: used to link fme_region to a list.
+ * @port_id: indicate which port this region connected to.
+ */
+struct fme_region {
+       struct platform_device *region;
+       struct list_head node;
+       int port_id;
+};
+
+/**
+ * struct fme_region_pdata - platform data for FME region platform device.
+ *
+ * @mgr: platform device of the FPGA manager.
+ * @br: platform device of the FPGA bridge.
+ * @region_id: region id (same as port_id).
+ */
+struct fme_region_pdata {
+       struct platform_device *mgr;
+       struct platform_device *br;
+       int region_id;
+};
+
+/**
+ * struct fme_bridge - FME fpga bridge data structure
+ *
+ * @br: platform device of the FPGA bridge.
+ * @node: used to link fme_bridge to a list.
+ */
+struct fme_bridge {
+       struct platform_device *br;
+       struct list_head node;
+};
+
+/**
+ * struct fme_bridge_pdata - platform data for FME bridge platform device.
+ *
+ * @port: platform device of the port feature dev.
+ */
+struct fme_br_pdata {
+       struct platform_device *port;
+};
+
+#define FPGA_DFL_FME_MGR       "dfl-fme-mgr"
+#define FPGA_DFL_FME_BRIDGE    "dfl-fme-bridge"
+#define FPGA_DFL_FME_REGION    "dfl-fme-region"
+

Everything in dfl-fme-pr.h up to this point is good and general and
should remain in dfl-fme-pr.h.  The register #defines for this FME's
FPGA manager device below should be associated with the FPGA manager
driver.  Sorry if the way I stated that in the v3 review wasn't clear.

quoted

+/* FME Partial Reconfiguration Sub Feature Register Set */
+#define FME_PR_DFH             0x0
+#define FME_PR_CTRL            0x8
+#define FME_PR_STS             0x10
+#define FME_PR_DATA            0x18
+#define FME_PR_ERR             0x20
+#define FME_PR_INTFC_ID_H      0xA8
+#define FME_PR_INTFC_ID_L      0xB0
+
+/* FME PR Control Register Bitfield */
+#define FME_PR_CTRL_PR_RST     BIT(0)  /* Reset PR engine */
+#define FME_PR_CTRL_PR_RSTACK  BIT(4)  /* Ack for PR engine reset */
+#define FME_PR_CTRL_PR_RGN_ID  GENMASK_ULL(9, 7)       /* PR Region ID */
+#define FME_PR_CTRL_PR_START   BIT(12) /* Start to request for PR service */
+#define FME_PR_CTRL_PR_COMPLETE        BIT(13) /* PR data push complete notification */
+
+/* FME PR Status Register Bitfield */
+/* Number of available entries in HW queue inside the PR engine. */
+#define FME_PR_STS_PR_CREDIT   GENMASK_ULL(8, 0)
+#define FME_PR_STS_PR_STS      BIT(16) /* PR operation status */
+#define FME_PR_STS_PR_STS_IDLE 0
+#define FME_PR_STS_PR_CTRLR_STS        GENMASK_ULL(22, 20)     /* Controller status */
+#define FME_PR_STS_PR_HOST_STS GENMASK_ULL(27, 24)     /* PR host status */
+
+/* FME PR Data Register Bitfield */
+/* PR data from the raw-binary file. */
+#define FME_PR_DATA_PR_DATA_RAW        GENMASK_ULL(32, 0)
+
+/* FME PR Error Register */
+/* PR Operation errors detected. */
+#define FME_PR_ERR_OPERATION_ERR       BIT(0)
+/* CRC error detected. */
+#define FME_PR_ERR_CRC_ERR             BIT(1)
+/* Incompatible PR bitstream detected. */
+#define FME_PR_ERR_INCOMPATIBLE_BS     BIT(2)
+/* PR data push protocol violated. */
+#define FME_PR_ERR_PROTOCOL_ERR                BIT(3)
+/* PR data fifo overflow error detected */
+#define FME_PR_ERR_FIFO_OVERFLOW       BIT(4)

This stuff is specific to this FPGA manager device, so it should
either be in the dfl-fme-mgr.c or in a dfl-fme-mgr.h

quoted

+
+#endif /* __DFL_FME_PR_H */

diff --git a/drivers/fpga/dfl-fme.h b/drivers/fpga/dfl-fme.h
new file mode 100644
index 0000000..c8bd48a
--- /dev/null
+++ b/drivers/fpga/dfl-fme.h

@@ -0,0 +1,38 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Header file for FPGA Management Engine (FME) Driver
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ */
+
+#ifndef __DFL_FME_H
+#define __DFL_FME_H
+
+/**
+ * struct fpga_fme - fpga fme private data
+ *
+ * @mgr: FME's FPGA manager platform device.
+ * @region_list: link list of FME's FPGA regions.
+ * @bridge_list: link list of FME's FPGA bridges.
+ * @pdata: fme platform device's pdata.
+ */
+struct fpga_fme {
+       struct platform_device *mgr;
+       struct list_head region_list;
+       struct list_head bridge_list;
+       struct feature_platform_data *pdata;
+};
+
+extern const struct feature_ops pr_mgmt_ops;
+
+#endif /* __DFL_FME_H */

diff --git a/include/uapi/linux/fpga-dfl.h b/include/uapi/linux/fpga-dfl.h
index 9321ee9..50ee831 100644
--- a/include/uapi/linux/fpga-dfl.h
+++ b/include/uapi/linux/fpga-dfl.h

@@ -14,6 +14,8 @@
 #ifndef _UAPI_LINUX_FPGA_DFL_H
 #define _UAPI_LINUX_FPGA_DFL_H

+#include <linux/types.h>
+
 #define FPGA_API_VERSION 0

 /*

@@ -26,6 +28,7 @@
 #define FPGA_MAGIC 0xB6

 #define FPGA_BASE 0
+#define FME_BASE 0x80

 /**
  * FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)

@@ -45,4 +48,28 @@

 #define FPGA_CHECK_EXTENSION   _IO(FPGA_MAGIC, FPGA_BASE + 1)

+/* IOCTLs for FME file descriptor */
+
+/**
+ * FPGA_FME_PORT_PR - _IOW(FPGA_MAGIC, FME_BASE + 0, struct fpga_fme_port_pr)
+ *
+ * Driver does Partial Reconfiguration based on Port ID and Buffer (Image)
+ * provided by caller.
+ * Return: 0 on success, -errno on failure.
+ * If FPGA_FME_PORT_PR returns -EIO, that indicates the HW has detected
+ * some errors during PR, under this case, the user can fetch HW error info
+ * from the status of FME's fpga manager.
+ */
+
+struct fpga_fme_port_pr {
+       /* Input */
+       __u32 argsz;            /* Structure length */
+       __u32 flags;            /* Zero for now */
+       __u32 port_id;
+       __u32 buffer_size;
+       __u64 buffer_address;   /* Userspace address to the buffer for PR */
+};
+
+#define FPGA_FME_PORT_PR       _IO(FPGA_MAGIC, FME_BASE + 0)
+
 #endif /* _UAPI_LINUX_FPGA_DFL_H */
--

2.7.4

Thanks,
Alan
--
To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v4 14/24] fpga: dfl: fme: add partial reconfiguration sub feature support

From: Wu Hao <hidden>
Date: 2018-03-12 04:39:33

On Sun, Mar 11, 2018 at 01:09:31PM -0700, matthew.gerlach@linux.intel.com wrote:


On Mon, 5 Mar 2018, Alan Tull wrote:


Hi Hao,

I do think we should consider different hw implementations with this code
because it does look like most of it is generic.  Specifically, I think
we should consider DFH based fpga images that have been shipped already,
and I think we need to consider new hardware implementations as well.
Full disclosure, I am particularly interested in porting to a new hw
implementation for partial reconfiguration.

Hi Matthew,

This dfl-fme-pr.c driver is developed for the PR sub feature (feature id
= 0x5), but we can reuse it for any cases if possible.

Please see some comments below.

Thanks for the comments, please see my comments inline.

Matthew Gerlach

quoted

On Tue, Feb 13, 2018 at 3:24 AM, Wu Hao [off-list ref] wrote:

Hi Hao,

We are going to want to be able use different FPGA managers with this
framework.  The different manager may be part of a different FME in
fabric or it may be a hardware FPGA manager.  Fortunately, at this
point now the changes, noted below, to get there are pretty small.

quoted

From: Kang Luwei <redacted>

Partial Reconfiguration (PR) is the most important function for FME. It
allows reconfiguration for given Port/Accelerated Function Unit (AFU).

It creates platform devices for fpga-mgr, fpga-regions and fpga-bridges,
and invokes fpga-region's interface (fpga_region_program_fpga) for PR
operation once PR request received via ioctl. Below user space interface
is exposed by this sub feature.

Ioctl interface:
* FPGA_FME_PORT_PR
 Do partial reconfiguration per information from userspace, including
 target port(AFU), buffer size and address info. It returns error code
 to userspace if failed. For detailed PR error information, user needs
 to read fpga-mgr's status sysfs interface.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Kang Luwei <redacted>
Signed-off-by: Xiao Guangrong <redacted>
Signed-off-by: Wu Hao <redacted>
---
v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
   switched to GPLv2 license.
   removed status from FPGA_FME_PORT_PR ioctl data structure.
   added platform devices creation for fpga-mgr/fpga-region/fpga-bridge.
   switched to fpga-region interface fpga_region_program_fpga for PR.
   fixed comments from Alan Tull on FPGA_MGR_PARTIAL_RECONFIG flag usage.
   fixed kbuild warnings.
v3: rename driver files to dfl-fme-*.
   rebase due to fpga APIs change.
   replace bitfields.
   switch to fpga_cdev_find_port to find port device.
v4: rebase and correct comments for some function.
   fix SPDX license issue.
   remove unnecessary input parameter for destroy_bridge/region function.
   add dfl-fme-pr.h for PR sub feature data structure and registers.
---
drivers/fpga/Makefile         |   2 +-
drivers/fpga/dfl-fme-main.c   |  45 +++-
drivers/fpga/dfl-fme-pr.c     | 497 ++++++++++++++++++++++++++++++++++++++++++
drivers/fpga/dfl-fme-pr.h     | 113 ++++++++++
drivers/fpga/dfl-fme.h        |  38 ++++
include/uapi/linux/fpga-dfl.h |  27 +++
6 files changed, 720 insertions(+), 2 deletions(-)
create mode 100644 drivers/fpga/dfl-fme-pr.c
create mode 100644 drivers/fpga/dfl-fme-pr.h
create mode 100644 drivers/fpga/dfl-fme.h

diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index fbd1c85..3c44fc9 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile

@@ -32,7 +32,7 @@ obj-$(CONFIG_OF_FPGA_REGION)          += of-fpga-region.o

obj-$(CONFIG_FPGA_DFL)                 += dfl.o
obj-$(CONFIG_FPGA_DFL_FME)             += dfl-fme.o

-dfl-fme-objs := dfl-fme-main.o
+dfl-fme-objs := dfl-fme-main.o dfl-fme-pr.o

# Drivers for FPGAs which implement DFL
obj-$(CONFIG_FPGA_DFL_PCI)             += dfl-pci.o

diff --git a/drivers/fpga/dfl-fme-main.c b/drivers/fpga/dfl-fme-main.c
index 1a9929c..967a44c 100644
--- a/drivers/fpga/dfl-fme-main.c
+++ b/drivers/fpga/dfl-fme-main.c

@@ -19,6 +19,7 @@

#include <linux/fpga-dfl.h>

#include "dfl.h"
+#include "dfl-fme.h"

static ssize_t ports_num_show(struct device *dev,
                             struct device_attribute *attr, char *buf)

@@ -111,6 +112,10 @@ static struct feature_driver fme_feature_drvs[] = {
               .ops = &fme_hdr_ops,
       },
       {
+               .id = FME_FEATURE_ID_PR_MGMT,
+               .ops = &pr_mgmt_ops,
+       },
+       {
               .ops = NULL,
       },

};

@@ -194,14 +199,49 @@ static const struct file_operations fme_fops = {
       .unlocked_ioctl = fme_ioctl,

};

+static int fme_dev_init(struct platform_device *pdev)
+{
+       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+       struct fpga_fme *fme;
+
+       fme = devm_kzalloc(&pdev->dev, sizeof(*fme), GFP_KERNEL);
+       if (!fme)
+               return -ENOMEM;
+
+       fme->pdata = pdata;
+
+       mutex_lock(&pdata->lock);
+       fpga_pdata_set_private(pdata, fme);
+       mutex_unlock(&pdata->lock);
+
+       return 0;
+}
+
+static void fme_dev_destroy(struct platform_device *pdev)
+{
+       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+       struct fpga_fme *fme;
+
+       mutex_lock(&pdata->lock);
+       fme = fpga_pdata_get_private(pdata);
+       fpga_pdata_set_private(pdata, NULL);
+       mutex_unlock(&pdata->lock);
+
+       devm_kfree(&pdev->dev, fme);

Is this needed?  This is called when the device is being destroyed anyway.

quoted

+}
+
static int fme_probe(struct platform_device *pdev)
{
       int ret;

-       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
+       ret = fme_dev_init(pdev);
       if (ret)
               goto exit;

+       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
+       if (ret)
+               goto dev_destroy;
+
       ret = fpga_register_dev_ops(pdev, &fme_fops, THIS_MODULE);
       if (ret)
               goto feature_uinit;

@@ -210,6 +250,8 @@ static int fme_probe(struct platform_device *pdev)

feature_uinit:
       fpga_dev_feature_uinit(pdev);
+dev_destroy:
+       fme_dev_destroy(pdev);
exit:
       return ret;
}

@@ -218,6 +260,7 @@ static int fme_remove(struct platform_device *pdev)

{
       fpga_dev_feature_uinit(pdev);
       fpga_unregister_dev_ops(pdev);
+       fme_dev_destroy(pdev);

       return 0;
}

diff --git a/drivers/fpga/dfl-fme-pr.c b/drivers/fpga/dfl-fme-pr.c
new file mode 100644
index 0000000..526e90b
--- /dev/null
+++ b/drivers/fpga/dfl-fme-pr.c

@@ -0,0 +1,497 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Driver for FPGA Management Engine (FME) Partial Reconfiguration
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Christopher Rauer <christopher.rauer@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ */
+
+#include <linux/types.h>
+#include <linux/device.h>
+#include <linux/vmalloc.h>
+#include <linux/uaccess.h>
+#include <linux/fpga/fpga-mgr.h>
+#include <linux/fpga/fpga-bridge.h>
+#include <linux/fpga/fpga-region.h>
+#include <linux/fpga-dfl.h>
+
+#include "dfl.h"
+#include "dfl-fme.h"
+#include "dfl-fme-pr.h"
+
+static struct fme_region *
+find_fme_region_by_port_id(struct fpga_fme *fme, int port_id)
+{
+       struct fme_region *fme_region;
+
+       list_for_each_entry(fme_region, &fme->region_list, node)
+               if (fme_region->port_id == port_id)
+                       return fme_region;
+
+       return NULL;
+}
+
+static int fpga_fme_region_match(struct device *dev, const void *data)
+{
+       return dev->parent == data;
+}
+
+static struct fpga_region *
+fpga_fme_region_find(struct fpga_fme *fme, int port_id)
+{
+       struct fme_region *fme_region;
+       struct fpga_region *region;
+
+       fme_region = find_fme_region_by_port_id(fme, port_id);
+       if (!fme_region)
+               return NULL;
+
+       region = fpga_region_class_find(NULL, &fme_region->region->dev,
+                                       fpga_fme_region_match);
+       if (!region)
+               return NULL;
+
+       return region;
+}
+
+static int fme_pr(struct platform_device *pdev, unsigned long arg)
+{
+       void __user *argp = (void __user *)arg;
+       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+       struct fpga_fme *fme;
+       struct fpga_image_info *info;
+       struct fpga_region *region;
+       struct fpga_fme_port_pr port_pr;
+       unsigned long minsz;
+       void __iomem *fme_hdr;
+       void *buf = NULL;
+       int ret = 0;
+       u64 v;
+
+       minsz = offsetofend(struct fpga_fme_port_pr, buffer_address);
+
+       if (copy_from_user(&port_pr, argp, minsz))
+               return -EFAULT;
+
+       if (port_pr.argsz < minsz || port_pr.flags)
+               return -EINVAL;
+
+       if (!IS_ALIGNED(port_pr.buffer_size, 4))
+               return -EINVAL;
+
+       /* get fme header region */
+       fme_hdr = get_feature_ioaddr_by_id(&pdev->dev,
+                                          FME_FEATURE_ID_HEADER);
+
+       /* check port id */
+       v = readq(fme_hdr + FME_HDR_CAP);
+       if (port_pr.port_id >= FIELD_GET(FME_CAP_NUM_PORTS, v)) {
+               dev_dbg(&pdev->dev, "port number more than maximum\n");
+               return -EINVAL;
+       }
+
+       if (!access_ok(VERIFY_READ,
+                      (void __user *)(unsigned long)port_pr.buffer_address,
+                      port_pr.buffer_size))
+               return -EFAULT;
+
+       buf = vmalloc(port_pr.buffer_size);
+       if (!buf)
+               return -ENOMEM;
+
+       if (copy_from_user(buf,
+                          (void __user *)(unsigned long)port_pr.buffer_address,
+                          port_pr.buffer_size)) {
+               ret = -EFAULT;
+               goto free_exit;
+       }
+
+       /* prepare fpga_image_info for PR */
+       info = fpga_image_info_alloc(&pdev->dev);
+       if (!info) {
+               ret = -ENOMEM;
+               goto free_exit;
+       }
+
+       info->flags |= FPGA_MGR_PARTIAL_RECONFIG;
+
+       mutex_lock(&pdata->lock);
+       fme = fpga_pdata_get_private(pdata);
+       /* fme device has been unregistered. */
+       if (!fme) {
+               ret = -EINVAL;
+               goto unlock_exit;
+       }
+
+       region = fpga_fme_region_find(fme, port_pr.port_id);
+       if (!region) {
+               ret = -EINVAL;
+               goto unlock_exit;
+       }
+
+       fpga_image_info_free(region->info);
+
+       info->buf = buf;
+       info->count = port_pr.buffer_size;
+       info->region_id = port_pr.port_id;
+       region->info = info;
+
+       ret = fpga_region_program_fpga(region);
+
+       if (region->get_bridges)
+               fpga_bridges_put(&region->bridge_list);
+
+       put_device(&region->dev);
+unlock_exit:
+       mutex_unlock(&pdata->lock);
+free_exit:
+       vfree(buf);
+       if (copy_to_user((void __user *)arg, &port_pr, minsz))
+               return -EFAULT;
+
+       return ret;
+}
+
+/**
+ * fpga_fme_create_mgr - create fpga mgr platform device as child device
+ *
+ * @pdata: fme platform_device's pdata
+ *
+ * Return: mgr platform device if successful, and error code otherwise.
+ */
+static struct platform_device *
+fpga_fme_create_mgr(struct feature_platform_data *pdata)
+{
+       struct platform_device *mgr, *fme = pdata->dev;
+       struct feature *feature;
+       struct resource res;
+       struct resource *pres;
+       int ret = -ENOMEM;
+
+       feature = get_feature_by_id(&pdata->dev->dev, FME_FEATURE_ID_PR_MGMT);
+       if (!feature)
+               return ERR_PTR(-ENODEV);
+
+       /*
+        * Each FME has only one fpga-mgr, so allocate platform device using
+        * the same FME platform device id.
+        */
+       mgr = platform_device_alloc(FPGA_DFL_FME_MGR, fme->id);

At this point, the framework is assuming all FME's include the same
FPGA manager device which would use the driver in dfl-fme-mgr.c.

I'm thinking of two cases where the manager isn't the same as a
dfl-fme-mgr.c manager are a bit different:

(1) a FME-based FPGA manager, but different implementation, different
registers.  The constraint is that the port implementation has to be
similar enough to use the rest of the base FME code.   I am wondering
if the FPGA manager can be added to the DFL.  At this point, the DFL
would drive which FPGA manager is alloc'd.   That way the user gets to
use all this code in dfl-fme-pr.c but with their FPGA manager.

I am thinking of the case of porting to a new hardware implementation
for Partial Reconfiguration.  Since the new hardware is completely different
at the register level, we would need new a new feature id.  The
fme init code for this new feature should be able to call the generic
code here and pass in the fgpa_mgr_ops that are hardware specific.

Per my understanding, if we introduced one new private feature (with a
different feature id), we could consider to reuse dfl-fme-pr.c firstly,
e.g in the init function, it could check the feature id and take different
code path for handling (create different fpga mgr). And for sure, we could
introduce another private feature driver if existing code can't be reused
at all. It depends the actual hardware implementation I think. : )

quoted

(2) a FPGA manager that can be added by device tree in the case of a
platform that is using device tree.  I think this will be pretty
simple and can be done later when someone is actually bringing this
framework up on a FPGA running under device tree.  I'm thinking that
the base DFL device that reads the dfl data from hardware can have a
DT property that points to the FPGA manager.  That manager can be
saved somewhere handy like the pdata and passed down to this code,
which realizes it can use that existing device and doesn't need to
alloc a platform device.  But again, that's probably best done later.

quoted

+       if (!mgr)
+               return ERR_PTR(ret);
+
+       mgr->dev.parent = &fme->dev;
+
+       pres = platform_get_resource(fme, IORESOURCE_MEM,
+                                    feature->resource_index);
+       if (!pres) {
+               ret = -ENODEV;
+               goto create_mgr_err;
+       }
+
+       memset(&res, 0, sizeof(struct resource));
+
+       res.start = pres->start;
+       res.end = pres->end;
+       res.name = pres->name;
+       res.flags = IORESOURCE_MEM;
+
+       ret = platform_device_add_resources(mgr, &res, 1);
+       if (ret)
+               goto create_mgr_err;
+
+       ret = platform_device_add(mgr);
+       if (ret)
+               goto create_mgr_err;
+
+       return mgr;
+
+create_mgr_err:
+       platform_device_put(mgr);
+       return ERR_PTR(ret);
+}
+
+/**
+ * fpga_fme_destroy_mgr - destroy fpga mgr platform device
+ * @pdata: fme platform device's pdata
+ */
+static void fpga_fme_destroy_mgr(struct feature_platform_data *pdata)
+{
+       struct fpga_fme *priv = fpga_pdata_get_private(pdata);
+
+       platform_device_unregister(priv->mgr);
+}
+
+/**
+ * fpga_fme_create_bridge - create fme fpga bridge platform device as child
+ *
+ * @pdata: fme platform device's pdata
+ * @port_id: port id for the bridge to be created.
+ *
+ * Return: bridge platform device if successful, and error code otherwise.
+ */
+static struct fme_bridge *
+fpga_fme_create_bridge(struct feature_platform_data *pdata, int port_id)
+{
+       struct device *dev = &pdata->dev->dev;
+       struct fme_br_pdata br_pdata;
+       struct fme_bridge *fme_br;
+       int ret = -ENOMEM;
+
+       fme_br = devm_kzalloc(dev, sizeof(*fme_br), GFP_KERNEL);
+       if (!fme_br)
+               return ERR_PTR(ret);
+
+       br_pdata.port = fpga_cdev_find_port(fpga_pdata_to_fpga_cdev(pdata),
+                                           &port_id, fpga_port_check_id);
+       if (!br_pdata.port)
+               return ERR_PTR(-ENODEV);
+
+       /*
+        * Each FPGA device may have more than one port, so allocate platform
+        * device using the same port platform device id.
+        */
+       fme_br->br = platform_device_alloc(FPGA_DFL_FME_BRIDGE,
+                                          br_pdata.port->id);
+       if (!fme_br->br) {
+               ret = -ENOMEM;
+               goto create_br_err;
+       }
+
+       fme_br->br->dev.parent = dev;
+
+       ret = platform_device_add_data(fme_br->br, &br_pdata, sizeof(br_pdata));
+       if (ret)
+               goto create_br_err;
+
+       ret = platform_device_add(fme_br->br);
+       if (ret)
+               goto create_br_err;
+
+       return fme_br;
+
+create_br_err:
+       platform_device_put(fme_br->br);
+       put_device(&br_pdata.port->dev);
+       return ERR_PTR(ret);
+}
+
+/**
+ * fpga_fme_destroy_bridge - destroy fpga bridge platform device
+ * @fme_br: fme bridge to destroy
+ */
+static void fpga_fme_destroy_bridge(struct fme_bridge *fme_br)
+{
+       struct fme_br_pdata *br_pdata = dev_get_platdata(&fme_br->br->dev);
+
+       put_device(&br_pdata->port->dev);
+       platform_device_unregister(fme_br->br);
+}
+
+/**
+ * fpga_fme_destroy_bridge - destroy all fpga bridge platform device
+ * @pdata: fme platform device's pdata
+ */
+static void fpga_fme_destroy_bridges(struct feature_platform_data *pdata)
+{
+       struct fpga_fme *priv = fpga_pdata_get_private(pdata);
+       struct fme_bridge *fbridge, *tmp;
+
+       list_for_each_entry_safe(fbridge, tmp, &priv->bridge_list, node) {
+               list_del(&fbridge->node);
+               fpga_fme_destroy_bridge(fbridge);
+       }
+}
+
+/**
+ * fpga_fme_create_region - create fpga region platform device as child
+ *
+ * @pdata: fme platform device's pdata
+ * @mgr: mgr platform device needed for region
+ * @br: br platform device needed for region
+ * @port_id: port id
+ *
+ * Return: fme region if successful, and error code otherwise.
+ */
+static struct fme_region *
+fpga_fme_create_region(struct feature_platform_data *pdata,
+                      struct platform_device *mgr,
+                      struct platform_device *br, int port_id)
+{
+       struct device *dev = &pdata->dev->dev;
+       struct fme_region_pdata region_pdata;
+       struct fme_region *fme_region;
+       int ret = -ENOMEM;
+
+       fme_region = devm_kzalloc(dev, sizeof(*fme_region), GFP_KERNEL);
+       if (!fme_region)
+               return ERR_PTR(ret);
+
+       region_pdata.mgr = mgr;
+       region_pdata.br = br;
+
+       /*
+        * Each FPGA device may have more than one port, so allocate platform
+        * device using the same port platform device id.
+        */
+       fme_region->region = platform_device_alloc(FPGA_DFL_FME_REGION, br->id);
+       if (!fme_region->region)
+               return ERR_PTR(ret);
+
+       fme_region->region->dev.parent = dev;
+
+       ret = platform_device_add_data(fme_region->region, &region_pdata,
+                                      sizeof(region_pdata));
+       if (ret)
+               goto create_region_err;
+
+       ret = platform_device_add(fme_region->region);
+       if (ret)
+               goto create_region_err;
+
+       fme_region->port_id = port_id;
+
+       return fme_region;
+
+create_region_err:
+       platform_device_put(fme_region->region);
+       return ERR_PTR(ret);
+}
+
+/**
+ * fpga_fme_destroy_region - destroy fme region
+ * @fme_region: fme region to destroy
+ */
+static void fpga_fme_destroy_region(struct fme_region *fme_region)
+{
+       platform_device_unregister(fme_region->region);
+}
+
+/**
+ * fpga_fme_destroy_regions - destroy all fme regions
+ * @pdata: fme platform device's pdata
+ */
+static void fpga_fme_destroy_regions(struct feature_platform_data *pdata)
+{
+       struct fpga_fme *priv = fpga_pdata_get_private(pdata);
+       struct fme_region *fme_region, *tmp;
+
+       list_for_each_entry_safe(fme_region, tmp, &priv->region_list, node) {
+               list_del(&fme_region->node);
+               fpga_fme_destroy_region(fme_region);
+       }
+}
+
+static int pr_mgmt_init(struct platform_device *pdev, struct feature *feature)
+{
+       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+       void __iomem *fme_hdr;
+       struct platform_device *mgr;
+       struct fme_region *fme_region;
+       struct fme_bridge *fme_br;
+       struct fpga_fme *priv;
+       int ret = -ENODEV, i = 0;
+       u64 fme_cap, port_offset;
+
+       fme_hdr = get_feature_ioaddr_by_id(&pdev->dev,
+                                          FME_FEATURE_ID_HEADER);
+
+       mutex_lock(&pdata->lock);
+       priv = fpga_pdata_get_private(pdata);
+
+       /* Initialize the region and bridge sub device list */
+       INIT_LIST_HEAD(&priv->region_list);
+       INIT_LIST_HEAD(&priv->bridge_list);
+
+       /* Create fpga mgr platform device */
+       mgr = fpga_fme_create_mgr(pdata);
+       if (IS_ERR(mgr)) {
+               dev_err(&pdev->dev, "fail to create fpga mgr pdev\n");
+               goto unlock;
+       }
+
+       priv->mgr = mgr;
+
+       /* Read capability register to check number of regions and bridges */
+       fme_cap = readq(fme_hdr + FME_HDR_CAP);

I don't think this capability field exists in currently deployed FPGA images
using DFH.  I believe this difference requires a new feature id
to differentiate between deployed FPGA images and images with this new
"feature".

I'm not sure about the "currently deployed FPGA images using DFH", this feature
driver is only for FME PR private feature (Yes, FME's private feature with
id = 0x5). But if they use DFH, so they must have their own feature ids as that
field is in the DFH, isn't it? Then we should be able to distinguish them based
on that.

Thanks
Hao

quoted

+       for (; i < FIELD_GET(FME_CAP_NUM_PORTS, fme_cap); i++) {
+               port_offset = readq(fme_hdr + FME_HDR_PORT_OFST(i));
+               if (!(port_offset & FME_PORT_OFST_IMP))
+                       continue;
+
+               /* Create bridge for each port */
+               fme_br = fpga_fme_create_bridge(pdata, i);
+               if (IS_ERR(fme_br)) {
+                       ret = PTR_ERR(fme_br);
+                       goto destroy_region;
+               }
+
+               list_add(&fme_br->node, &priv->bridge_list);
+
+               /* Create region for each port */
+               fme_region = fpga_fme_create_region(pdata, mgr, fme_br->br, i);
+               if (!fme_region) {
+                       ret = PTR_ERR(fme_region);
+                       goto destroy_region;
+               }
+
+               list_add(&fme_region->node, &priv->region_list);
+       }
+       mutex_unlock(&pdata->lock);
+
+       return 0;
+
+destroy_region:
+       fpga_fme_destroy_regions(pdata);
+       fpga_fme_destroy_bridges(pdata);
+       fpga_fme_destroy_mgr(pdata);
+unlock:
+       mutex_unlock(&pdata->lock);
+       return ret;
+}
+
+static void pr_mgmt_uinit(struct platform_device *pdev, struct feature *feature)
+{
+       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+       struct fpga_fme *priv;
+
+       mutex_lock(&pdata->lock);
+       priv = fpga_pdata_get_private(pdata);
+
+       fpga_fme_destroy_regions(pdata);
+       fpga_fme_destroy_bridges(pdata);
+       fpga_fme_destroy_mgr(pdata);
+       mutex_unlock(&pdata->lock);
+}
+
+static long fme_pr_ioctl(struct platform_device *pdev, struct feature *feature,
+                        unsigned int cmd, unsigned long arg)
+{
+       long ret;
+
+       switch (cmd) {
+       case FPGA_FME_PORT_PR:
+               ret = fme_pr(pdev, arg);
+               break;
+       default:
+               ret = -ENODEV;
+       }
+
+       return ret;
+}
+
+const struct feature_ops pr_mgmt_ops = {
+       .init = pr_mgmt_init,
+       .uinit = pr_mgmt_uinit,
+       .ioctl = fme_pr_ioctl,
+};

diff --git a/drivers/fpga/dfl-fme-pr.h b/drivers/fpga/dfl-fme-pr.h
new file mode 100644
index 0000000..11bd001
--- /dev/null
+++ b/drivers/fpga/dfl-fme-pr.h

@@ -0,0 +1,113 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Header file for FPGA Management Engine (FME) Partial Reconfiguration Driver
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ */
+
+#ifndef __DFL_FME_PR_H
+#define __DFL_FME_PR_H
+
+#include <linux/platform_device.h>
+
+/**
+ * struct fme_region - FME fpga region data structure
+ *
+ * @region: platform device of the FPGA region.
+ * @node: used to link fme_region to a list.
+ * @port_id: indicate which port this region connected to.
+ */
+struct fme_region {
+       struct platform_device *region;
+       struct list_head node;
+       int port_id;
+};
+
+/**
+ * struct fme_region_pdata - platform data for FME region platform device.
+ *
+ * @mgr: platform device of the FPGA manager.
+ * @br: platform device of the FPGA bridge.
+ * @region_id: region id (same as port_id).
+ */
+struct fme_region_pdata {
+       struct platform_device *mgr;
+       struct platform_device *br;
+       int region_id;
+};
+
+/**
+ * struct fme_bridge - FME fpga bridge data structure
+ *
+ * @br: platform device of the FPGA bridge.
+ * @node: used to link fme_bridge to a list.
+ */
+struct fme_bridge {
+       struct platform_device *br;
+       struct list_head node;
+};
+
+/**
+ * struct fme_bridge_pdata - platform data for FME bridge platform device.
+ *
+ * @port: platform device of the port feature dev.
+ */
+struct fme_br_pdata {
+       struct platform_device *port;
+};
+
+#define FPGA_DFL_FME_MGR       "dfl-fme-mgr"
+#define FPGA_DFL_FME_BRIDGE    "dfl-fme-bridge"
+#define FPGA_DFL_FME_REGION    "dfl-fme-region"
+

Everything in dfl-fme-pr.h up to this point is good and general and
should remain in dfl-fme-pr.h.  The register #defines for this FME's
FPGA manager device below should be associated with the FPGA manager
driver.  Sorry if the way I stated that in the v3 review wasn't clear.

quoted

+/* FME Partial Reconfiguration Sub Feature Register Set */
+#define FME_PR_DFH             0x0
+#define FME_PR_CTRL            0x8
+#define FME_PR_STS             0x10
+#define FME_PR_DATA            0x18
+#define FME_PR_ERR             0x20
+#define FME_PR_INTFC_ID_H      0xA8
+#define FME_PR_INTFC_ID_L      0xB0
+
+/* FME PR Control Register Bitfield */
+#define FME_PR_CTRL_PR_RST     BIT(0)  /* Reset PR engine */
+#define FME_PR_CTRL_PR_RSTACK  BIT(4)  /* Ack for PR engine reset */
+#define FME_PR_CTRL_PR_RGN_ID  GENMASK_ULL(9, 7)       /* PR Region ID */
+#define FME_PR_CTRL_PR_START   BIT(12) /* Start to request for PR service */
+#define FME_PR_CTRL_PR_COMPLETE        BIT(13) /* PR data push complete notification */
+
+/* FME PR Status Register Bitfield */
+/* Number of available entries in HW queue inside the PR engine. */
+#define FME_PR_STS_PR_CREDIT   GENMASK_ULL(8, 0)
+#define FME_PR_STS_PR_STS      BIT(16) /* PR operation status */
+#define FME_PR_STS_PR_STS_IDLE 0
+#define FME_PR_STS_PR_CTRLR_STS        GENMASK_ULL(22, 20)     /* Controller status */
+#define FME_PR_STS_PR_HOST_STS GENMASK_ULL(27, 24)     /* PR host status */
+
+/* FME PR Data Register Bitfield */
+/* PR data from the raw-binary file. */
+#define FME_PR_DATA_PR_DATA_RAW        GENMASK_ULL(32, 0)
+
+/* FME PR Error Register */
+/* PR Operation errors detected. */
+#define FME_PR_ERR_OPERATION_ERR       BIT(0)
+/* CRC error detected. */
+#define FME_PR_ERR_CRC_ERR             BIT(1)
+/* Incompatible PR bitstream detected. */
+#define FME_PR_ERR_INCOMPATIBLE_BS     BIT(2)
+/* PR data push protocol violated. */
+#define FME_PR_ERR_PROTOCOL_ERR                BIT(3)
+/* PR data fifo overflow error detected */
+#define FME_PR_ERR_FIFO_OVERFLOW       BIT(4)

This stuff is specific to this FPGA manager device, so it should
either be in the dfl-fme-mgr.c or in a dfl-fme-mgr.h

quoted

+
+#endif /* __DFL_FME_PR_H */

diff --git a/drivers/fpga/dfl-fme.h b/drivers/fpga/dfl-fme.h
new file mode 100644
index 0000000..c8bd48a
--- /dev/null
+++ b/drivers/fpga/dfl-fme.h

@@ -0,0 +1,38 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Header file for FPGA Management Engine (FME) Driver
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ */
+
+#ifndef __DFL_FME_H
+#define __DFL_FME_H
+
+/**
+ * struct fpga_fme - fpga fme private data
+ *
+ * @mgr: FME's FPGA manager platform device.
+ * @region_list: link list of FME's FPGA regions.
+ * @bridge_list: link list of FME's FPGA bridges.
+ * @pdata: fme platform device's pdata.
+ */
+struct fpga_fme {
+       struct platform_device *mgr;
+       struct list_head region_list;
+       struct list_head bridge_list;
+       struct feature_platform_data *pdata;
+};
+
+extern const struct feature_ops pr_mgmt_ops;
+
+#endif /* __DFL_FME_H */

diff --git a/include/uapi/linux/fpga-dfl.h b/include/uapi/linux/fpga-dfl.h
index 9321ee9..50ee831 100644
--- a/include/uapi/linux/fpga-dfl.h
+++ b/include/uapi/linux/fpga-dfl.h

@@ -14,6 +14,8 @@

#ifndef _UAPI_LINUX_FPGA_DFL_H
#define _UAPI_LINUX_FPGA_DFL_H

+#include <linux/types.h>
+
#define FPGA_API_VERSION 0

/*

@@ -26,6 +28,7 @@

#define FPGA_MAGIC 0xB6

#define FPGA_BASE 0
+#define FME_BASE 0x80

/**
 * FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)

@@ -45,4 +48,28 @@

#define FPGA_CHECK_EXTENSION   _IO(FPGA_MAGIC, FPGA_BASE + 1)

+/* IOCTLs for FME file descriptor */
+
+/**
+ * FPGA_FME_PORT_PR - _IOW(FPGA_MAGIC, FME_BASE + 0, struct fpga_fme_port_pr)
+ *
+ * Driver does Partial Reconfiguration based on Port ID and Buffer (Image)
+ * provided by caller.
+ * Return: 0 on success, -errno on failure.
+ * If FPGA_FME_PORT_PR returns -EIO, that indicates the HW has detected
+ * some errors during PR, under this case, the user can fetch HW error info
+ * from the status of FME's fpga manager.
+ */
+
+struct fpga_fme_port_pr {
+       /* Input */
+       __u32 argsz;            /* Structure length */
+       __u32 flags;            /* Zero for now */
+       __u32 port_id;
+       __u32 buffer_size;
+       __u64 buffer_address;   /* Userspace address to the buffer for PR */
+};
+
+#define FPGA_FME_PORT_PR       _IO(FPGA_MAGIC, FME_BASE + 0)
+
#endif /* _UAPI_LINUX_FPGA_DFL_H */
--
2.7.4

Thanks,
Alan
--
To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v4 14/24] fpga: dfl: fme: add partial reconfiguration sub feature support

From: Alan Tull <atull@kernel.org>
Date: 2018-03-12 18:54:40

On Sun, Mar 11, 2018 at 11:29 PM, Wu Hao [off-list ref] wrote:

On Sun, Mar 11, 2018 at 01:09:31PM -0700, matthew.gerlach@linux.intel.com wrote:

quoted

Hi Hao,

I do think we should consider different hw implementations with this code
because it does look like most of it is generic.  Specifically, I think
we should consider DFH based fpga images that have been shipped already,
and I think we need to consider new hardware implementations as well.
Full disclosure, I am particularly interested in porting to a new hw
implementation for partial reconfiguration.

Hi Matthew,

The manager may not be the only thing that has to change for a new
implementation, i.e. will your 'port' be able to work with this
patchset?  In the current implementation, the port is part of the dfl
enumeration code (dfl.c and dfl.h) rather than being part of the
bridge for some reason.  We discussed the possibility of putting the
port enable/disable code into the bridge driver [1], but that didn't
seem feasible at least last December.  I still would feel more
confident if port were part of the bridge instead of part of dfl.

Alan

[1] https://lkml.org/lkml/2017/12/21/62

Re: [PATCH v4 14/24] fpga: dfl: fme: add partial reconfiguration sub feature support

From: <hidden>
Date: 2018-03-12 21:36:50


On Mon, 12 Mar 2018, Wu Hao wrote:

Hi Hao,

Please see my two comments inline.

Thanks,
Matthew Gerlach

On Sun, Mar 11, 2018 at 01:09:31PM -0700, matthew.gerlach@linux.intel.com wrote:

quoted


On Mon, 5 Mar 2018, Alan Tull wrote:


Hi Hao,

I do think we should consider different hw implementations with this code
because it does look like most of it is generic.  Specifically, I think
we should consider DFH based fpga images that have been shipped already,
and I think we need to consider new hardware implementations as well.
Full disclosure, I am particularly interested in porting to a new hw
implementation for partial reconfiguration.

Hi Matthew,

This dfl-fme-pr.c driver is developed for the PR sub feature (feature id
= 0x5), but we can reuse it for any cases if possible.

quoted

Please see some comments below.

Thanks for the comments, please see my comments inline.

quoted

Matthew Gerlach

quoted

On Tue, Feb 13, 2018 at 3:24 AM, Wu Hao [off-list ref] wrote:

Hi Hao,

We are going to want to be able use different FPGA managers with this
framework.  The different manager may be part of a different FME in
fabric or it may be a hardware FPGA manager.  Fortunately, at this
point now the changes, noted below, to get there are pretty small.

quoted

From: Kang Luwei <redacted>

Partial Reconfiguration (PR) is the most important function for FME. It
allows reconfiguration for given Port/Accelerated Function Unit (AFU).

It creates platform devices for fpga-mgr, fpga-regions and fpga-bridges,
and invokes fpga-region's interface (fpga_region_program_fpga) for PR
operation once PR request received via ioctl. Below user space interface
is exposed by this sub feature.

Ioctl interface:
* FPGA_FME_PORT_PR
 Do partial reconfiguration per information from userspace, including
 target port(AFU), buffer size and address info. It returns error code
 to userspace if failed. For detailed PR error information, user needs
 to read fpga-mgr's status sysfs interface.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Kang Luwei <redacted>
Signed-off-by: Xiao Guangrong <redacted>
Signed-off-by: Wu Hao <redacted>
---
v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
   switched to GPLv2 license.
   removed status from FPGA_FME_PORT_PR ioctl data structure.
   added platform devices creation for fpga-mgr/fpga-region/fpga-bridge.
   switched to fpga-region interface fpga_region_program_fpga for PR.
   fixed comments from Alan Tull on FPGA_MGR_PARTIAL_RECONFIG flag usage.
   fixed kbuild warnings.
v3: rename driver files to dfl-fme-*.
   rebase due to fpga APIs change.
   replace bitfields.
   switch to fpga_cdev_find_port to find port device.
v4: rebase and correct comments for some function.
   fix SPDX license issue.
   remove unnecessary input parameter for destroy_bridge/region function.
   add dfl-fme-pr.h for PR sub feature data structure and registers.
---
drivers/fpga/Makefile         |   2 +-
drivers/fpga/dfl-fme-main.c   |  45 +++-
drivers/fpga/dfl-fme-pr.c     | 497 ++++++++++++++++++++++++++++++++++++++++++
drivers/fpga/dfl-fme-pr.h     | 113 ++++++++++
drivers/fpga/dfl-fme.h        |  38 ++++
include/uapi/linux/fpga-dfl.h |  27 +++
6 files changed, 720 insertions(+), 2 deletions(-)
create mode 100644 drivers/fpga/dfl-fme-pr.c
create mode 100644 drivers/fpga/dfl-fme-pr.h
create mode 100644 drivers/fpga/dfl-fme.h

diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index fbd1c85..3c44fc9 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile

@@ -32,7 +32,7 @@ obj-$(CONFIG_OF_FPGA_REGION)          += of-fpga-region.o

obj-$(CONFIG_FPGA_DFL)                 += dfl.o
obj-$(CONFIG_FPGA_DFL_FME)             += dfl-fme.o

-dfl-fme-objs := dfl-fme-main.o
+dfl-fme-objs := dfl-fme-main.o dfl-fme-pr.o

# Drivers for FPGAs which implement DFL
obj-$(CONFIG_FPGA_DFL_PCI)             += dfl-pci.o

diff --git a/drivers/fpga/dfl-fme-main.c b/drivers/fpga/dfl-fme-main.c
index 1a9929c..967a44c 100644
--- a/drivers/fpga/dfl-fme-main.c
+++ b/drivers/fpga/dfl-fme-main.c

@@ -19,6 +19,7 @@

#include <linux/fpga-dfl.h>

#include "dfl.h"
+#include "dfl-fme.h"

static ssize_t ports_num_show(struct device *dev,
                             struct device_attribute *attr, char *buf)

@@ -111,6 +112,10 @@ static struct feature_driver fme_feature_drvs[] = {
               .ops = &fme_hdr_ops,
       },
       {
+               .id = FME_FEATURE_ID_PR_MGMT,
+               .ops = &pr_mgmt_ops,
+       },
+       {
               .ops = NULL,
       },

};

@@ -194,14 +199,49 @@ static const struct file_operations fme_fops = {
       .unlocked_ioctl = fme_ioctl,

};

+static int fme_dev_init(struct platform_device *pdev)
+{
+       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+       struct fpga_fme *fme;
+
+       fme = devm_kzalloc(&pdev->dev, sizeof(*fme), GFP_KERNEL);
+       if (!fme)
+               return -ENOMEM;
+
+       fme->pdata = pdata;
+
+       mutex_lock(&pdata->lock);
+       fpga_pdata_set_private(pdata, fme);
+       mutex_unlock(&pdata->lock);
+
+       return 0;
+}
+
+static void fme_dev_destroy(struct platform_device *pdev)
+{
+       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+       struct fpga_fme *fme;
+
+       mutex_lock(&pdata->lock);
+       fme = fpga_pdata_get_private(pdata);
+       fpga_pdata_set_private(pdata, NULL);
+       mutex_unlock(&pdata->lock);
+
+       devm_kfree(&pdev->dev, fme);

Is this needed?  This is called when the device is being destroyed anyway.

quoted

+}
+
static int fme_probe(struct platform_device *pdev)
{
       int ret;

-       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
+       ret = fme_dev_init(pdev);
       if (ret)
               goto exit;

+       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
+       if (ret)
+               goto dev_destroy;
+
       ret = fpga_register_dev_ops(pdev, &fme_fops, THIS_MODULE);
       if (ret)
               goto feature_uinit;

@@ -210,6 +250,8 @@ static int fme_probe(struct platform_device *pdev)

feature_uinit:
       fpga_dev_feature_uinit(pdev);
+dev_destroy:
+       fme_dev_destroy(pdev);
exit:
       return ret;
}

@@ -218,6 +260,7 @@ static int fme_remove(struct platform_device *pdev)

{
       fpga_dev_feature_uinit(pdev);
       fpga_unregister_dev_ops(pdev);
+       fme_dev_destroy(pdev);

       return 0;
}

diff --git a/drivers/fpga/dfl-fme-pr.c b/drivers/fpga/dfl-fme-pr.c
new file mode 100644
index 0000000..526e90b
--- /dev/null
+++ b/drivers/fpga/dfl-fme-pr.c

@@ -0,0 +1,497 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Driver for FPGA Management Engine (FME) Partial Reconfiguration
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Christopher Rauer <christopher.rauer@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ */
+
+#include <linux/types.h>
+#include <linux/device.h>
+#include <linux/vmalloc.h>
+#include <linux/uaccess.h>
+#include <linux/fpga/fpga-mgr.h>
+#include <linux/fpga/fpga-bridge.h>
+#include <linux/fpga/fpga-region.h>
+#include <linux/fpga-dfl.h>
+
+#include "dfl.h"
+#include "dfl-fme.h"
+#include "dfl-fme-pr.h"
+
+static struct fme_region *
+find_fme_region_by_port_id(struct fpga_fme *fme, int port_id)
+{
+       struct fme_region *fme_region;
+
+       list_for_each_entry(fme_region, &fme->region_list, node)
+               if (fme_region->port_id == port_id)
+                       return fme_region;
+
+       return NULL;
+}
+
+static int fpga_fme_region_match(struct device *dev, const void *data)
+{
+       return dev->parent == data;
+}
+
+static struct fpga_region *
+fpga_fme_region_find(struct fpga_fme *fme, int port_id)
+{
+       struct fme_region *fme_region;
+       struct fpga_region *region;
+
+       fme_region = find_fme_region_by_port_id(fme, port_id);
+       if (!fme_region)
+               return NULL;
+
+       region = fpga_region_class_find(NULL, &fme_region->region->dev,
+                                       fpga_fme_region_match);
+       if (!region)
+               return NULL;
+
+       return region;
+}
+
+static int fme_pr(struct platform_device *pdev, unsigned long arg)
+{
+       void __user *argp = (void __user *)arg;
+       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+       struct fpga_fme *fme;
+       struct fpga_image_info *info;
+       struct fpga_region *region;
+       struct fpga_fme_port_pr port_pr;
+       unsigned long minsz;
+       void __iomem *fme_hdr;
+       void *buf = NULL;
+       int ret = 0;
+       u64 v;
+
+       minsz = offsetofend(struct fpga_fme_port_pr, buffer_address);
+
+       if (copy_from_user(&port_pr, argp, minsz))
+               return -EFAULT;
+
+       if (port_pr.argsz < minsz || port_pr.flags)
+               return -EINVAL;
+
+       if (!IS_ALIGNED(port_pr.buffer_size, 4))
+               return -EINVAL;
+
+       /* get fme header region */
+       fme_hdr = get_feature_ioaddr_by_id(&pdev->dev,
+                                          FME_FEATURE_ID_HEADER);
+
+       /* check port id */
+       v = readq(fme_hdr + FME_HDR_CAP);
+       if (port_pr.port_id >= FIELD_GET(FME_CAP_NUM_PORTS, v)) {
+               dev_dbg(&pdev->dev, "port number more than maximum\n");
+               return -EINVAL;
+       }
+
+       if (!access_ok(VERIFY_READ,
+                      (void __user *)(unsigned long)port_pr.buffer_address,
+                      port_pr.buffer_size))
+               return -EFAULT;
+
+       buf = vmalloc(port_pr.buffer_size);
+       if (!buf)
+               return -ENOMEM;
+
+       if (copy_from_user(buf,
+                          (void __user *)(unsigned long)port_pr.buffer_address,
+                          port_pr.buffer_size)) {
+               ret = -EFAULT;
+               goto free_exit;
+       }
+
+       /* prepare fpga_image_info for PR */
+       info = fpga_image_info_alloc(&pdev->dev);
+       if (!info) {
+               ret = -ENOMEM;
+               goto free_exit;
+       }
+
+       info->flags |= FPGA_MGR_PARTIAL_RECONFIG;
+
+       mutex_lock(&pdata->lock);
+       fme = fpga_pdata_get_private(pdata);
+       /* fme device has been unregistered. */
+       if (!fme) {
+               ret = -EINVAL;
+               goto unlock_exit;
+       }
+
+       region = fpga_fme_region_find(fme, port_pr.port_id);
+       if (!region) {
+               ret = -EINVAL;
+               goto unlock_exit;
+       }
+
+       fpga_image_info_free(region->info);
+
+       info->buf = buf;
+       info->count = port_pr.buffer_size;
+       info->region_id = port_pr.port_id;
+       region->info = info;
+
+       ret = fpga_region_program_fpga(region);
+
+       if (region->get_bridges)
+               fpga_bridges_put(&region->bridge_list);
+
+       put_device(&region->dev);
+unlock_exit:
+       mutex_unlock(&pdata->lock);
+free_exit:
+       vfree(buf);
+       if (copy_to_user((void __user *)arg, &port_pr, minsz))
+               return -EFAULT;
+
+       return ret;
+}
+
+/**
+ * fpga_fme_create_mgr - create fpga mgr platform device as child device
+ *
+ * @pdata: fme platform_device's pdata
+ *
+ * Return: mgr platform device if successful, and error code otherwise.
+ */
+static struct platform_device *
+fpga_fme_create_mgr(struct feature_platform_data *pdata)
+{
+       struct platform_device *mgr, *fme = pdata->dev;
+       struct feature *feature;
+       struct resource res;
+       struct resource *pres;
+       int ret = -ENOMEM;
+
+       feature = get_feature_by_id(&pdata->dev->dev, FME_FEATURE_ID_PR_MGMT);
+       if (!feature)
+               return ERR_PTR(-ENODEV);
+
+       /*
+        * Each FME has only one fpga-mgr, so allocate platform device using
+        * the same FME platform device id.
+        */
+       mgr = platform_device_alloc(FPGA_DFL_FME_MGR, fme->id);

At this point, the framework is assuming all FME's include the same
FPGA manager device which would use the driver in dfl-fme-mgr.c.

I'm thinking of two cases where the manager isn't the same as a
dfl-fme-mgr.c manager are a bit different:

(1) a FME-based FPGA manager, but different implementation, different
registers.  The constraint is that the port implementation has to be
similar enough to use the rest of the base FME code.   I am wondering
if the FPGA manager can be added to the DFL.  At this point, the DFL
would drive which FPGA manager is alloc'd.   That way the user gets to
use all this code in dfl-fme-pr.c but with their FPGA manager.

I am thinking of the case of porting to a new hardware implementation
for Partial Reconfiguration.  Since the new hardware is completely different
at the register level, we would need new a new feature id.  The
fme init code for this new feature should be able to call the generic
code here and pass in the fgpa_mgr_ops that are hardware specific.

Per my understanding, if we introduced one new private feature (with a
different feature id), we could consider to reuse dfl-fme-pr.c firstly,
e.g in the init function, it could check the feature id and take different
code path for handling (create different fpga mgr). And for sure, we could
introduce another private feature driver if existing code can't be reused
at all. It depends the actual hardware implementation I think. : )

It sounds to me like we are in aggreement that a lot of dfl-fme-pr.c could 
be reused by another hardware implementation.  I am currently scoping
Partial Reconfiguration for a FPGA that is not the Arria10, and I think 
this much of dfl-fme-pr.c could be reused for that different hw.

quoted

(2) a FPGA manager that can be added by device tree in the case of a
platform that is using device tree.  I think this will be pretty
simple and can be done later when someone is actually bringing this
framework up on a FPGA running under device tree.  I'm thinking that
the base DFL device that reads the dfl data from hardware can have a
DT property that points to the FPGA manager.  That manager can be
saved somewhere handy like the pdata and passed down to this code,
which realizes it can use that existing device and doesn't need to
alloc a platform device.  But again, that's probably best done later.

quoted

+       if (!mgr)
+               return ERR_PTR(ret);
+
+       mgr->dev.parent = &fme->dev;
+
+       pres = platform_get_resource(fme, IORESOURCE_MEM,
+                                    feature->resource_index);
+       if (!pres) {
+               ret = -ENODEV;
+               goto create_mgr_err;
+       }
+
+       memset(&res, 0, sizeof(struct resource));
+
+       res.start = pres->start;
+       res.end = pres->end;
+       res.name = pres->name;
+       res.flags = IORESOURCE_MEM;
+
+       ret = platform_device_add_resources(mgr, &res, 1);
+       if (ret)
+               goto create_mgr_err;
+
+       ret = platform_device_add(mgr);
+       if (ret)
+               goto create_mgr_err;
+
+       return mgr;
+
+create_mgr_err:
+       platform_device_put(mgr);
+       return ERR_PTR(ret);
+}
+
+/**
+ * fpga_fme_destroy_mgr - destroy fpga mgr platform device
+ * @pdata: fme platform device's pdata
+ */
+static void fpga_fme_destroy_mgr(struct feature_platform_data *pdata)
+{
+       struct fpga_fme *priv = fpga_pdata_get_private(pdata);
+
+       platform_device_unregister(priv->mgr);
+}
+
+/**
+ * fpga_fme_create_bridge - create fme fpga bridge platform device as child
+ *
+ * @pdata: fme platform device's pdata
+ * @port_id: port id for the bridge to be created.
+ *
+ * Return: bridge platform device if successful, and error code otherwise.
+ */
+static struct fme_bridge *
+fpga_fme_create_bridge(struct feature_platform_data *pdata, int port_id)
+{
+       struct device *dev = &pdata->dev->dev;
+       struct fme_br_pdata br_pdata;
+       struct fme_bridge *fme_br;
+       int ret = -ENOMEM;
+
+       fme_br = devm_kzalloc(dev, sizeof(*fme_br), GFP_KERNEL);
+       if (!fme_br)
+               return ERR_PTR(ret);
+
+       br_pdata.port = fpga_cdev_find_port(fpga_pdata_to_fpga_cdev(pdata),
+                                           &port_id, fpga_port_check_id);
+       if (!br_pdata.port)
+               return ERR_PTR(-ENODEV);
+
+       /*
+        * Each FPGA device may have more than one port, so allocate platform
+        * device using the same port platform device id.
+        */
+       fme_br->br = platform_device_alloc(FPGA_DFL_FME_BRIDGE,
+                                          br_pdata.port->id);
+       if (!fme_br->br) {
+               ret = -ENOMEM;
+               goto create_br_err;
+       }
+
+       fme_br->br->dev.parent = dev;
+
+       ret = platform_device_add_data(fme_br->br, &br_pdata, sizeof(br_pdata));
+       if (ret)
+               goto create_br_err;
+
+       ret = platform_device_add(fme_br->br);
+       if (ret)
+               goto create_br_err;
+
+       return fme_br;
+
+create_br_err:
+       platform_device_put(fme_br->br);
+       put_device(&br_pdata.port->dev);
+       return ERR_PTR(ret);
+}
+
+/**
+ * fpga_fme_destroy_bridge - destroy fpga bridge platform device
+ * @fme_br: fme bridge to destroy
+ */
+static void fpga_fme_destroy_bridge(struct fme_bridge *fme_br)
+{
+       struct fme_br_pdata *br_pdata = dev_get_platdata(&fme_br->br->dev);
+
+       put_device(&br_pdata->port->dev);
+       platform_device_unregister(fme_br->br);
+}
+
+/**
+ * fpga_fme_destroy_bridge - destroy all fpga bridge platform device
+ * @pdata: fme platform device's pdata
+ */
+static void fpga_fme_destroy_bridges(struct feature_platform_data *pdata)
+{
+       struct fpga_fme *priv = fpga_pdata_get_private(pdata);
+       struct fme_bridge *fbridge, *tmp;
+
+       list_for_each_entry_safe(fbridge, tmp, &priv->bridge_list, node) {
+               list_del(&fbridge->node);
+               fpga_fme_destroy_bridge(fbridge);
+       }
+}
+
+/**
+ * fpga_fme_create_region - create fpga region platform device as child
+ *
+ * @pdata: fme platform device's pdata
+ * @mgr: mgr platform device needed for region
+ * @br: br platform device needed for region
+ * @port_id: port id
+ *
+ * Return: fme region if successful, and error code otherwise.
+ */
+static struct fme_region *
+fpga_fme_create_region(struct feature_platform_data *pdata,
+                      struct platform_device *mgr,
+                      struct platform_device *br, int port_id)
+{
+       struct device *dev = &pdata->dev->dev;
+       struct fme_region_pdata region_pdata;
+       struct fme_region *fme_region;
+       int ret = -ENOMEM;
+
+       fme_region = devm_kzalloc(dev, sizeof(*fme_region), GFP_KERNEL);
+       if (!fme_region)
+               return ERR_PTR(ret);
+
+       region_pdata.mgr = mgr;
+       region_pdata.br = br;
+
+       /*
+        * Each FPGA device may have more than one port, so allocate platform
+        * device using the same port platform device id.
+        */
+       fme_region->region = platform_device_alloc(FPGA_DFL_FME_REGION, br->id);
+       if (!fme_region->region)
+               return ERR_PTR(ret);
+
+       fme_region->region->dev.parent = dev;
+
+       ret = platform_device_add_data(fme_region->region, &region_pdata,
+                                      sizeof(region_pdata));
+       if (ret)
+               goto create_region_err;
+
+       ret = platform_device_add(fme_region->region);
+       if (ret)
+               goto create_region_err;
+
+       fme_region->port_id = port_id;
+
+       return fme_region;
+
+create_region_err:
+       platform_device_put(fme_region->region);
+       return ERR_PTR(ret);
+}
+
+/**
+ * fpga_fme_destroy_region - destroy fme region
+ * @fme_region: fme region to destroy
+ */
+static void fpga_fme_destroy_region(struct fme_region *fme_region)
+{
+       platform_device_unregister(fme_region->region);
+}
+
+/**
+ * fpga_fme_destroy_regions - destroy all fme regions
+ * @pdata: fme platform device's pdata
+ */
+static void fpga_fme_destroy_regions(struct feature_platform_data *pdata)
+{
+       struct fpga_fme *priv = fpga_pdata_get_private(pdata);
+       struct fme_region *fme_region, *tmp;
+
+       list_for_each_entry_safe(fme_region, tmp, &priv->region_list, node) {
+               list_del(&fme_region->node);
+               fpga_fme_destroy_region(fme_region);
+       }
+}
+
+static int pr_mgmt_init(struct platform_device *pdev, struct feature *feature)
+{
+       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+       void __iomem *fme_hdr;
+       struct platform_device *mgr;
+       struct fme_region *fme_region;
+       struct fme_bridge *fme_br;
+       struct fpga_fme *priv;
+       int ret = -ENODEV, i = 0;
+       u64 fme_cap, port_offset;
+
+       fme_hdr = get_feature_ioaddr_by_id(&pdev->dev,
+                                          FME_FEATURE_ID_HEADER);
+
+       mutex_lock(&pdata->lock);
+       priv = fpga_pdata_get_private(pdata);
+
+       /* Initialize the region and bridge sub device list */
+       INIT_LIST_HEAD(&priv->region_list);
+       INIT_LIST_HEAD(&priv->bridge_list);
+
+       /* Create fpga mgr platform device */
+       mgr = fpga_fme_create_mgr(pdata);
+       if (IS_ERR(mgr)) {
+               dev_err(&pdev->dev, "fail to create fpga mgr pdev\n");
+               goto unlock;
+       }
+
+       priv->mgr = mgr;
+
+       /* Read capability register to check number of regions and bridges */
+       fme_cap = readq(fme_hdr + FME_HDR_CAP);

I don't think this capability field exists in currently deployed FPGA images
using DFH.  I believe this difference requires a new feature id
to differentiate between deployed FPGA images and images with this new
"feature".

I'm not sure about the "currently deployed FPGA images using DFH", this feature
driver is only for FME PR private feature (Yes, FME's private feature with
id = 0x5). But if they use DFH, so they must have their own feature ids as that
field is in the DFH, isn't it? Then we should be able to distinguish them based
on that.

Thanks
Hao

I am specifically thinking of the FPGA images associated with DCP 1.0. 
Should this driver work with those images?  It looks to me that this 
capability register is not in the DCP 1.0.

quoted

+       for (; i < FIELD_GET(FME_CAP_NUM_PORTS, fme_cap); i++) {
+               port_offset = readq(fme_hdr + FME_HDR_PORT_OFST(i));
+               if (!(port_offset & FME_PORT_OFST_IMP))
+                       continue;
+
+               /* Create bridge for each port */
+               fme_br = fpga_fme_create_bridge(pdata, i);
+               if (IS_ERR(fme_br)) {
+                       ret = PTR_ERR(fme_br);
+                       goto destroy_region;
+               }
+
+               list_add(&fme_br->node, &priv->bridge_list);
+
+               /* Create region for each port */
+               fme_region = fpga_fme_create_region(pdata, mgr, fme_br->br, i);
+               if (!fme_region) {
+                       ret = PTR_ERR(fme_region);
+                       goto destroy_region;
+               }
+
+               list_add(&fme_region->node, &priv->region_list);
+       }
+       mutex_unlock(&pdata->lock);
+
+       return 0;
+
+destroy_region:
+       fpga_fme_destroy_regions(pdata);
+       fpga_fme_destroy_bridges(pdata);
+       fpga_fme_destroy_mgr(pdata);
+unlock:
+       mutex_unlock(&pdata->lock);
+       return ret;
+}
+
+static void pr_mgmt_uinit(struct platform_device *pdev, struct feature *feature)
+{
+       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+       struct fpga_fme *priv;
+
+       mutex_lock(&pdata->lock);
+       priv = fpga_pdata_get_private(pdata);
+
+       fpga_fme_destroy_regions(pdata);
+       fpga_fme_destroy_bridges(pdata);
+       fpga_fme_destroy_mgr(pdata);
+       mutex_unlock(&pdata->lock);
+}
+
+static long fme_pr_ioctl(struct platform_device *pdev, struct feature *feature,
+                        unsigned int cmd, unsigned long arg)
+{
+       long ret;
+
+       switch (cmd) {
+       case FPGA_FME_PORT_PR:
+               ret = fme_pr(pdev, arg);
+               break;
+       default:
+               ret = -ENODEV;
+       }
+
+       return ret;
+}
+
+const struct feature_ops pr_mgmt_ops = {
+       .init = pr_mgmt_init,
+       .uinit = pr_mgmt_uinit,
+       .ioctl = fme_pr_ioctl,
+};

diff --git a/drivers/fpga/dfl-fme-pr.h b/drivers/fpga/dfl-fme-pr.h
new file mode 100644
index 0000000..11bd001
--- /dev/null
+++ b/drivers/fpga/dfl-fme-pr.h

@@ -0,0 +1,113 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Header file for FPGA Management Engine (FME) Partial Reconfiguration Driver
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ */
+
+#ifndef __DFL_FME_PR_H
+#define __DFL_FME_PR_H
+
+#include <linux/platform_device.h>
+
+/**
+ * struct fme_region - FME fpga region data structure
+ *
+ * @region: platform device of the FPGA region.
+ * @node: used to link fme_region to a list.
+ * @port_id: indicate which port this region connected to.
+ */
+struct fme_region {
+       struct platform_device *region;
+       struct list_head node;
+       int port_id;
+};
+
+/**
+ * struct fme_region_pdata - platform data for FME region platform device.
+ *
+ * @mgr: platform device of the FPGA manager.
+ * @br: platform device of the FPGA bridge.
+ * @region_id: region id (same as port_id).
+ */
+struct fme_region_pdata {
+       struct platform_device *mgr;
+       struct platform_device *br;
+       int region_id;
+};
+
+/**
+ * struct fme_bridge - FME fpga bridge data structure
+ *
+ * @br: platform device of the FPGA bridge.
+ * @node: used to link fme_bridge to a list.
+ */
+struct fme_bridge {
+       struct platform_device *br;
+       struct list_head node;
+};
+
+/**
+ * struct fme_bridge_pdata - platform data for FME bridge platform device.
+ *
+ * @port: platform device of the port feature dev.
+ */
+struct fme_br_pdata {
+       struct platform_device *port;
+};
+
+#define FPGA_DFL_FME_MGR       "dfl-fme-mgr"
+#define FPGA_DFL_FME_BRIDGE    "dfl-fme-bridge"
+#define FPGA_DFL_FME_REGION    "dfl-fme-region"
+

Everything in dfl-fme-pr.h up to this point is good and general and
should remain in dfl-fme-pr.h.  The register #defines for this FME's
FPGA manager device below should be associated with the FPGA manager
driver.  Sorry if the way I stated that in the v3 review wasn't clear.

quoted

+/* FME Partial Reconfiguration Sub Feature Register Set */
+#define FME_PR_DFH             0x0
+#define FME_PR_CTRL            0x8
+#define FME_PR_STS             0x10
+#define FME_PR_DATA            0x18
+#define FME_PR_ERR             0x20
+#define FME_PR_INTFC_ID_H      0xA8
+#define FME_PR_INTFC_ID_L      0xB0
+
+/* FME PR Control Register Bitfield */
+#define FME_PR_CTRL_PR_RST     BIT(0)  /* Reset PR engine */
+#define FME_PR_CTRL_PR_RSTACK  BIT(4)  /* Ack for PR engine reset */
+#define FME_PR_CTRL_PR_RGN_ID  GENMASK_ULL(9, 7)       /* PR Region ID */
+#define FME_PR_CTRL_PR_START   BIT(12) /* Start to request for PR service */
+#define FME_PR_CTRL_PR_COMPLETE        BIT(13) /* PR data push complete notification */
+
+/* FME PR Status Register Bitfield */
+/* Number of available entries in HW queue inside the PR engine. */
+#define FME_PR_STS_PR_CREDIT   GENMASK_ULL(8, 0)
+#define FME_PR_STS_PR_STS      BIT(16) /* PR operation status */
+#define FME_PR_STS_PR_STS_IDLE 0
+#define FME_PR_STS_PR_CTRLR_STS        GENMASK_ULL(22, 20)     /* Controller status */
+#define FME_PR_STS_PR_HOST_STS GENMASK_ULL(27, 24)     /* PR host status */
+
+/* FME PR Data Register Bitfield */
+/* PR data from the raw-binary file. */
+#define FME_PR_DATA_PR_DATA_RAW        GENMASK_ULL(32, 0)
+
+/* FME PR Error Register */
+/* PR Operation errors detected. */
+#define FME_PR_ERR_OPERATION_ERR       BIT(0)
+/* CRC error detected. */
+#define FME_PR_ERR_CRC_ERR             BIT(1)
+/* Incompatible PR bitstream detected. */
+#define FME_PR_ERR_INCOMPATIBLE_BS     BIT(2)
+/* PR data push protocol violated. */
+#define FME_PR_ERR_PROTOCOL_ERR                BIT(3)
+/* PR data fifo overflow error detected */
+#define FME_PR_ERR_FIFO_OVERFLOW       BIT(4)

This stuff is specific to this FPGA manager device, so it should
either be in the dfl-fme-mgr.c or in a dfl-fme-mgr.h

quoted

+
+#endif /* __DFL_FME_PR_H */

diff --git a/drivers/fpga/dfl-fme.h b/drivers/fpga/dfl-fme.h
new file mode 100644
index 0000000..c8bd48a
--- /dev/null
+++ b/drivers/fpga/dfl-fme.h

@@ -0,0 +1,38 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Header file for FPGA Management Engine (FME) Driver
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ */
+
+#ifndef __DFL_FME_H
+#define __DFL_FME_H
+
+/**
+ * struct fpga_fme - fpga fme private data
+ *
+ * @mgr: FME's FPGA manager platform device.
+ * @region_list: link list of FME's FPGA regions.
+ * @bridge_list: link list of FME's FPGA bridges.
+ * @pdata: fme platform device's pdata.
+ */
+struct fpga_fme {
+       struct platform_device *mgr;
+       struct list_head region_list;
+       struct list_head bridge_list;
+       struct feature_platform_data *pdata;
+};
+
+extern const struct feature_ops pr_mgmt_ops;
+
+#endif /* __DFL_FME_H */

diff --git a/include/uapi/linux/fpga-dfl.h b/include/uapi/linux/fpga-dfl.h
index 9321ee9..50ee831 100644
--- a/include/uapi/linux/fpga-dfl.h
+++ b/include/uapi/linux/fpga-dfl.h

@@ -14,6 +14,8 @@

#ifndef _UAPI_LINUX_FPGA_DFL_H
#define _UAPI_LINUX_FPGA_DFL_H

+#include <linux/types.h>
+
#define FPGA_API_VERSION 0

/*

@@ -26,6 +28,7 @@

#define FPGA_MAGIC 0xB6

#define FPGA_BASE 0
+#define FME_BASE 0x80

/**
 * FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)

@@ -45,4 +48,28 @@

#define FPGA_CHECK_EXTENSION   _IO(FPGA_MAGIC, FPGA_BASE + 1)

+/* IOCTLs for FME file descriptor */
+
+/**
+ * FPGA_FME_PORT_PR - _IOW(FPGA_MAGIC, FME_BASE + 0, struct fpga_fme_port_pr)
+ *
+ * Driver does Partial Reconfiguration based on Port ID and Buffer (Image)
+ * provided by caller.
+ * Return: 0 on success, -errno on failure.
+ * If FPGA_FME_PORT_PR returns -EIO, that indicates the HW has detected
+ * some errors during PR, under this case, the user can fetch HW error info
+ * from the status of FME's fpga manager.
+ */
+
+struct fpga_fme_port_pr {
+       /* Input */
+       __u32 argsz;            /* Structure length */
+       __u32 flags;            /* Zero for now */
+       __u32 port_id;
+       __u32 buffer_size;
+       __u64 buffer_address;   /* Userspace address to the buffer for PR */
+};
+
+#define FPGA_FME_PORT_PR       _IO(FPGA_MAGIC, FME_BASE + 0)
+
#endif /* _UAPI_LINUX_FPGA_DFL_H */
--
2.7.4

Thanks,
Alan
--
To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v4 14/24] fpga: dfl: fme: add partial reconfiguration sub feature support

From: Wu Hao <hidden>
Date: 2018-03-13 01:17:41

On Mon, Mar 12, 2018 at 02:36:48PM -0700, matthew.gerlach@linux.intel.com wrote:


On Mon, 12 Mar 2018, Wu Hao wrote:

Hi Hao,

Please see my two comments inline.

Thanks,
Matthew Gerlach

quoted

On Sun, Mar 11, 2018 at 01:09:31PM -0700, matthew.gerlach@linux.intel.com wrote:

quoted


On Mon, 5 Mar 2018, Alan Tull wrote:


Hi Hao,

I do think we should consider different hw implementations with this code
because it does look like most of it is generic.  Specifically, I think
we should consider DFH based fpga images that have been shipped already,
and I think we need to consider new hardware implementations as well.
Full disclosure, I am particularly interested in porting to a new hw
implementation for partial reconfiguration.

Hi Matthew,

This dfl-fme-pr.c driver is developed for the PR sub feature (feature id
= 0x5), but we can reuse it for any cases if possible.

quoted

Please see some comments below.

Thanks for the comments, please see my comments inline.

quoted

Matthew Gerlach

quoted

On Tue, Feb 13, 2018 at 3:24 AM, Wu Hao [off-list ref] wrote:

Hi Hao,

We are going to want to be able use different FPGA managers with this
framework.  The different manager may be part of a different FME in
fabric or it may be a hardware FPGA manager.  Fortunately, at this
point now the changes, noted below, to get there are pretty small.

quoted

From: Kang Luwei <redacted>

Partial Reconfiguration (PR) is the most important function for FME. It
allows reconfiguration for given Port/Accelerated Function Unit (AFU).

It creates platform devices for fpga-mgr, fpga-regions and fpga-bridges,
and invokes fpga-region's interface (fpga_region_program_fpga) for PR
operation once PR request received via ioctl. Below user space interface
is exposed by this sub feature.

Ioctl interface:
* FPGA_FME_PORT_PR
Do partial reconfiguration per information from userspace, including
target port(AFU), buffer size and address info. It returns error code
to userspace if failed. For detailed PR error information, user needs
to read fpga-mgr's status sysfs interface.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Kang Luwei <redacted>
Signed-off-by: Xiao Guangrong <redacted>
Signed-off-by: Wu Hao <redacted>
---
v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
  switched to GPLv2 license.
  removed status from FPGA_FME_PORT_PR ioctl data structure.
  added platform devices creation for fpga-mgr/fpga-region/fpga-bridge.
  switched to fpga-region interface fpga_region_program_fpga for PR.
  fixed comments from Alan Tull on FPGA_MGR_PARTIAL_RECONFIG flag usage.
  fixed kbuild warnings.
v3: rename driver files to dfl-fme-*.
  rebase due to fpga APIs change.
  replace bitfields.
  switch to fpga_cdev_find_port to find port device.
v4: rebase and correct comments for some function.
  fix SPDX license issue.
  remove unnecessary input parameter for destroy_bridge/region function.
  add dfl-fme-pr.h for PR sub feature data structure and registers.
---
drivers/fpga/Makefile         |   2 +-
drivers/fpga/dfl-fme-main.c   |  45 +++-
drivers/fpga/dfl-fme-pr.c     | 497 ++++++++++++++++++++++++++++++++++++++++++
drivers/fpga/dfl-fme-pr.h     | 113 ++++++++++
drivers/fpga/dfl-fme.h        |  38 ++++
include/uapi/linux/fpga-dfl.h |  27 +++
6 files changed, 720 insertions(+), 2 deletions(-)
create mode 100644 drivers/fpga/dfl-fme-pr.c
create mode 100644 drivers/fpga/dfl-fme-pr.h
create mode 100644 drivers/fpga/dfl-fme.h

diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index fbd1c85..3c44fc9 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile

@@ -32,7 +32,7 @@ obj-$(CONFIG_OF_FPGA_REGION)          += of-fpga-region.o

obj-$(CONFIG_FPGA_DFL)                 += dfl.o
obj-$(CONFIG_FPGA_DFL_FME)             += dfl-fme.o

-dfl-fme-objs := dfl-fme-main.o
+dfl-fme-objs := dfl-fme-main.o dfl-fme-pr.o

# Drivers for FPGAs which implement DFL
obj-$(CONFIG_FPGA_DFL_PCI)             += dfl-pci.o

diff --git a/drivers/fpga/dfl-fme-main.c b/drivers/fpga/dfl-fme-main.c
index 1a9929c..967a44c 100644
--- a/drivers/fpga/dfl-fme-main.c
+++ b/drivers/fpga/dfl-fme-main.c

@@ -19,6 +19,7 @@

#include <linux/fpga-dfl.h>

#include "dfl.h"
+#include "dfl-fme.h"

static ssize_t ports_num_show(struct device *dev,
                            struct device_attribute *attr, char *buf)

@@ -111,6 +112,10 @@ static struct feature_driver fme_feature_drvs[] = {
              .ops = &fme_hdr_ops,
      },
      {
+               .id = FME_FEATURE_ID_PR_MGMT,
+               .ops = &pr_mgmt_ops,
+       },
+       {
              .ops = NULL,
      },

};

@@ -194,14 +199,49 @@ static const struct file_operations fme_fops = {
      .unlocked_ioctl = fme_ioctl,

};

+static int fme_dev_init(struct platform_device *pdev)
+{
+       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+       struct fpga_fme *fme;
+
+       fme = devm_kzalloc(&pdev->dev, sizeof(*fme), GFP_KERNEL);
+       if (!fme)
+               return -ENOMEM;
+
+       fme->pdata = pdata;
+
+       mutex_lock(&pdata->lock);
+       fpga_pdata_set_private(pdata, fme);
+       mutex_unlock(&pdata->lock);
+
+       return 0;
+}
+
+static void fme_dev_destroy(struct platform_device *pdev)
+{
+       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+       struct fpga_fme *fme;
+
+       mutex_lock(&pdata->lock);
+       fme = fpga_pdata_get_private(pdata);
+       fpga_pdata_set_private(pdata, NULL);
+       mutex_unlock(&pdata->lock);
+
+       devm_kfree(&pdev->dev, fme);

Is this needed?  This is called when the device is being destroyed anyway.

quoted

+}
+
static int fme_probe(struct platform_device *pdev)
{
      int ret;

-       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
+       ret = fme_dev_init(pdev);
      if (ret)
              goto exit;

+       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
+       if (ret)
+               goto dev_destroy;
+
      ret = fpga_register_dev_ops(pdev, &fme_fops, THIS_MODULE);
      if (ret)
              goto feature_uinit;

@@ -210,6 +250,8 @@ static int fme_probe(struct platform_device *pdev)

feature_uinit:
      fpga_dev_feature_uinit(pdev);
+dev_destroy:
+       fme_dev_destroy(pdev);
exit:
      return ret;
}

@@ -218,6 +260,7 @@ static int fme_remove(struct platform_device *pdev)

{
      fpga_dev_feature_uinit(pdev);
      fpga_unregister_dev_ops(pdev);
+       fme_dev_destroy(pdev);

      return 0;
}

diff --git a/drivers/fpga/dfl-fme-pr.c b/drivers/fpga/dfl-fme-pr.c
new file mode 100644
index 0000000..526e90b
--- /dev/null
+++ b/drivers/fpga/dfl-fme-pr.c

@@ -0,0 +1,497 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Driver for FPGA Management Engine (FME) Partial Reconfiguration
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Christopher Rauer <christopher.rauer@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ */
+
+#include <linux/types.h>
+#include <linux/device.h>
+#include <linux/vmalloc.h>
+#include <linux/uaccess.h>
+#include <linux/fpga/fpga-mgr.h>
+#include <linux/fpga/fpga-bridge.h>
+#include <linux/fpga/fpga-region.h>
+#include <linux/fpga-dfl.h>
+
+#include "dfl.h"
+#include "dfl-fme.h"
+#include "dfl-fme-pr.h"
+
+static struct fme_region *
+find_fme_region_by_port_id(struct fpga_fme *fme, int port_id)
+{
+       struct fme_region *fme_region;
+
+       list_for_each_entry(fme_region, &fme->region_list, node)
+               if (fme_region->port_id == port_id)
+                       return fme_region;
+
+       return NULL;
+}
+
+static int fpga_fme_region_match(struct device *dev, const void *data)
+{
+       return dev->parent == data;
+}
+
+static struct fpga_region *
+fpga_fme_region_find(struct fpga_fme *fme, int port_id)
+{
+       struct fme_region *fme_region;
+       struct fpga_region *region;
+
+       fme_region = find_fme_region_by_port_id(fme, port_id);
+       if (!fme_region)
+               return NULL;
+
+       region = fpga_region_class_find(NULL, &fme_region->region->dev,
+                                       fpga_fme_region_match);
+       if (!region)
+               return NULL;
+
+       return region;
+}
+
+static int fme_pr(struct platform_device *pdev, unsigned long arg)
+{
+       void __user *argp = (void __user *)arg;
+       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+       struct fpga_fme *fme;
+       struct fpga_image_info *info;
+       struct fpga_region *region;
+       struct fpga_fme_port_pr port_pr;
+       unsigned long minsz;
+       void __iomem *fme_hdr;
+       void *buf = NULL;
+       int ret = 0;
+       u64 v;
+
+       minsz = offsetofend(struct fpga_fme_port_pr, buffer_address);
+
+       if (copy_from_user(&port_pr, argp, minsz))
+               return -EFAULT;
+
+       if (port_pr.argsz < minsz || port_pr.flags)
+               return -EINVAL;
+
+       if (!IS_ALIGNED(port_pr.buffer_size, 4))
+               return -EINVAL;
+
+       /* get fme header region */
+       fme_hdr = get_feature_ioaddr_by_id(&pdev->dev,
+                                          FME_FEATURE_ID_HEADER);
+
+       /* check port id */
+       v = readq(fme_hdr + FME_HDR_CAP);
+       if (port_pr.port_id >= FIELD_GET(FME_CAP_NUM_PORTS, v)) {
+               dev_dbg(&pdev->dev, "port number more than maximum\n");
+               return -EINVAL;
+       }
+
+       if (!access_ok(VERIFY_READ,
+                      (void __user *)(unsigned long)port_pr.buffer_address,
+                      port_pr.buffer_size))
+               return -EFAULT;
+
+       buf = vmalloc(port_pr.buffer_size);
+       if (!buf)
+               return -ENOMEM;
+
+       if (copy_from_user(buf,
+                          (void __user *)(unsigned long)port_pr.buffer_address,
+                          port_pr.buffer_size)) {
+               ret = -EFAULT;
+               goto free_exit;
+       }
+
+       /* prepare fpga_image_info for PR */
+       info = fpga_image_info_alloc(&pdev->dev);
+       if (!info) {
+               ret = -ENOMEM;
+               goto free_exit;
+       }
+
+       info->flags |= FPGA_MGR_PARTIAL_RECONFIG;
+
+       mutex_lock(&pdata->lock);
+       fme = fpga_pdata_get_private(pdata);
+       /* fme device has been unregistered. */
+       if (!fme) {
+               ret = -EINVAL;
+               goto unlock_exit;
+       }
+
+       region = fpga_fme_region_find(fme, port_pr.port_id);
+       if (!region) {
+               ret = -EINVAL;
+               goto unlock_exit;
+       }
+
+       fpga_image_info_free(region->info);
+
+       info->buf = buf;
+       info->count = port_pr.buffer_size;
+       info->region_id = port_pr.port_id;
+       region->info = info;
+
+       ret = fpga_region_program_fpga(region);
+
+       if (region->get_bridges)
+               fpga_bridges_put(&region->bridge_list);
+
+       put_device(&region->dev);
+unlock_exit:
+       mutex_unlock(&pdata->lock);
+free_exit:
+       vfree(buf);
+       if (copy_to_user((void __user *)arg, &port_pr, minsz))
+               return -EFAULT;
+
+       return ret;
+}
+
+/**
+ * fpga_fme_create_mgr - create fpga mgr platform device as child device
+ *
+ * @pdata: fme platform_device's pdata
+ *
+ * Return: mgr platform device if successful, and error code otherwise.
+ */
+static struct platform_device *
+fpga_fme_create_mgr(struct feature_platform_data *pdata)
+{
+       struct platform_device *mgr, *fme = pdata->dev;
+       struct feature *feature;
+       struct resource res;
+       struct resource *pres;
+       int ret = -ENOMEM;
+
+       feature = get_feature_by_id(&pdata->dev->dev, FME_FEATURE_ID_PR_MGMT);
+       if (!feature)
+               return ERR_PTR(-ENODEV);
+
+       /*
+        * Each FME has only one fpga-mgr, so allocate platform device using
+        * the same FME platform device id.
+        */
+       mgr = platform_device_alloc(FPGA_DFL_FME_MGR, fme->id);

At this point, the framework is assuming all FME's include the same
FPGA manager device which would use the driver in dfl-fme-mgr.c.

I'm thinking of two cases where the manager isn't the same as a
dfl-fme-mgr.c manager are a bit different:

(1) a FME-based FPGA manager, but different implementation, different
registers.  The constraint is that the port implementation has to be
similar enough to use the rest of the base FME code.   I am wondering
if the FPGA manager can be added to the DFL.  At this point, the DFL
would drive which FPGA manager is alloc'd.   That way the user gets to
use all this code in dfl-fme-pr.c but with their FPGA manager.

I am thinking of the case of porting to a new hardware implementation
for Partial Reconfiguration.  Since the new hardware is completely different
at the register level, we would need new a new feature id.  The
fme init code for this new feature should be able to call the generic
code here and pass in the fgpa_mgr_ops that are hardware specific.

Per my understanding, if we introduced one new private feature (with a
different feature id), we could consider to reuse dfl-fme-pr.c firstly,
e.g in the init function, it could check the feature id and take different
code path for handling (create different fpga mgr). And for sure, we could
introduce another private feature driver if existing code can't be reused
at all. It depends the actual hardware implementation I think. : )

It sounds to me like we are in aggreement that a lot of dfl-fme-pr.c could
be reused by another hardware implementation.  I am currently scoping
Partial Reconfiguration for a FPGA that is not the Arria10, and I think this
much of dfl-fme-pr.c could be reused for that different hw.

Yes, agree. : )

quoted

(2) a FPGA manager that can be added by device tree in the case of a
platform that is using device tree.  I think this will be pretty
simple and can be done later when someone is actually bringing this
framework up on a FPGA running under device tree.  I'm thinking that
the base DFL device that reads the dfl data from hardware can have a
DT property that points to the FPGA manager.  That manager can be
saved somewhere handy like the pdata and passed down to this code,
which realizes it can use that existing device and doesn't need to
alloc a platform device.  But again, that's probably best done later.

quoted

+       if (!mgr)
+               return ERR_PTR(ret);
+
+       mgr->dev.parent = &fme->dev;
+
+       pres = platform_get_resource(fme, IORESOURCE_MEM,
+                                    feature->resource_index);
+       if (!pres) {
+               ret = -ENODEV;
+               goto create_mgr_err;
+       }
+
+       memset(&res, 0, sizeof(struct resource));
+
+       res.start = pres->start;
+       res.end = pres->end;
+       res.name = pres->name;
+       res.flags = IORESOURCE_MEM;
+
+       ret = platform_device_add_resources(mgr, &res, 1);
+       if (ret)
+               goto create_mgr_err;
+
+       ret = platform_device_add(mgr);
+       if (ret)
+               goto create_mgr_err;
+
+       return mgr;
+
+create_mgr_err:
+       platform_device_put(mgr);
+       return ERR_PTR(ret);
+}
+
+/**
+ * fpga_fme_destroy_mgr - destroy fpga mgr platform device
+ * @pdata: fme platform device's pdata
+ */
+static void fpga_fme_destroy_mgr(struct feature_platform_data *pdata)
+{
+       struct fpga_fme *priv = fpga_pdata_get_private(pdata);
+
+       platform_device_unregister(priv->mgr);
+}
+
+/**
+ * fpga_fme_create_bridge - create fme fpga bridge platform device as child
+ *
+ * @pdata: fme platform device's pdata
+ * @port_id: port id for the bridge to be created.
+ *
+ * Return: bridge platform device if successful, and error code otherwise.
+ */
+static struct fme_bridge *
+fpga_fme_create_bridge(struct feature_platform_data *pdata, int port_id)
+{
+       struct device *dev = &pdata->dev->dev;
+       struct fme_br_pdata br_pdata;
+       struct fme_bridge *fme_br;
+       int ret = -ENOMEM;
+
+       fme_br = devm_kzalloc(dev, sizeof(*fme_br), GFP_KERNEL);
+       if (!fme_br)
+               return ERR_PTR(ret);
+
+       br_pdata.port = fpga_cdev_find_port(fpga_pdata_to_fpga_cdev(pdata),
+                                           &port_id, fpga_port_check_id);
+       if (!br_pdata.port)
+               return ERR_PTR(-ENODEV);
+
+       /*
+        * Each FPGA device may have more than one port, so allocate platform
+        * device using the same port platform device id.
+        */
+       fme_br->br = platform_device_alloc(FPGA_DFL_FME_BRIDGE,
+                                          br_pdata.port->id);
+       if (!fme_br->br) {
+               ret = -ENOMEM;
+               goto create_br_err;
+       }
+
+       fme_br->br->dev.parent = dev;
+
+       ret = platform_device_add_data(fme_br->br, &br_pdata, sizeof(br_pdata));
+       if (ret)
+               goto create_br_err;
+
+       ret = platform_device_add(fme_br->br);
+       if (ret)
+               goto create_br_err;
+
+       return fme_br;
+
+create_br_err:
+       platform_device_put(fme_br->br);
+       put_device(&br_pdata.port->dev);
+       return ERR_PTR(ret);
+}
+
+/**
+ * fpga_fme_destroy_bridge - destroy fpga bridge platform device
+ * @fme_br: fme bridge to destroy
+ */
+static void fpga_fme_destroy_bridge(struct fme_bridge *fme_br)
+{
+       struct fme_br_pdata *br_pdata = dev_get_platdata(&fme_br->br->dev);
+
+       put_device(&br_pdata->port->dev);
+       platform_device_unregister(fme_br->br);
+}
+
+/**
+ * fpga_fme_destroy_bridge - destroy all fpga bridge platform device
+ * @pdata: fme platform device's pdata
+ */
+static void fpga_fme_destroy_bridges(struct feature_platform_data *pdata)
+{
+       struct fpga_fme *priv = fpga_pdata_get_private(pdata);
+       struct fme_bridge *fbridge, *tmp;
+
+       list_for_each_entry_safe(fbridge, tmp, &priv->bridge_list, node) {
+               list_del(&fbridge->node);
+               fpga_fme_destroy_bridge(fbridge);
+       }
+}
+
+/**
+ * fpga_fme_create_region - create fpga region platform device as child
+ *
+ * @pdata: fme platform device's pdata
+ * @mgr: mgr platform device needed for region
+ * @br: br platform device needed for region
+ * @port_id: port id
+ *
+ * Return: fme region if successful, and error code otherwise.
+ */
+static struct fme_region *
+fpga_fme_create_region(struct feature_platform_data *pdata,
+                      struct platform_device *mgr,
+                      struct platform_device *br, int port_id)
+{
+       struct device *dev = &pdata->dev->dev;
+       struct fme_region_pdata region_pdata;
+       struct fme_region *fme_region;
+       int ret = -ENOMEM;
+
+       fme_region = devm_kzalloc(dev, sizeof(*fme_region), GFP_KERNEL);
+       if (!fme_region)
+               return ERR_PTR(ret);
+
+       region_pdata.mgr = mgr;
+       region_pdata.br = br;
+
+       /*
+        * Each FPGA device may have more than one port, so allocate platform
+        * device using the same port platform device id.
+        */
+       fme_region->region = platform_device_alloc(FPGA_DFL_FME_REGION, br->id);
+       if (!fme_region->region)
+               return ERR_PTR(ret);
+
+       fme_region->region->dev.parent = dev;
+
+       ret = platform_device_add_data(fme_region->region, &region_pdata,
+                                      sizeof(region_pdata));
+       if (ret)
+               goto create_region_err;
+
+       ret = platform_device_add(fme_region->region);
+       if (ret)
+               goto create_region_err;
+
+       fme_region->port_id = port_id;
+
+       return fme_region;
+
+create_region_err:
+       platform_device_put(fme_region->region);
+       return ERR_PTR(ret);
+}
+
+/**
+ * fpga_fme_destroy_region - destroy fme region
+ * @fme_region: fme region to destroy
+ */
+static void fpga_fme_destroy_region(struct fme_region *fme_region)
+{
+       platform_device_unregister(fme_region->region);
+}
+
+/**
+ * fpga_fme_destroy_regions - destroy all fme regions
+ * @pdata: fme platform device's pdata
+ */
+static void fpga_fme_destroy_regions(struct feature_platform_data *pdata)
+{
+       struct fpga_fme *priv = fpga_pdata_get_private(pdata);
+       struct fme_region *fme_region, *tmp;
+
+       list_for_each_entry_safe(fme_region, tmp, &priv->region_list, node) {
+               list_del(&fme_region->node);
+               fpga_fme_destroy_region(fme_region);
+       }
+}
+
+static int pr_mgmt_init(struct platform_device *pdev, struct feature *feature)
+{
+       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+       void __iomem *fme_hdr;
+       struct platform_device *mgr;
+       struct fme_region *fme_region;
+       struct fme_bridge *fme_br;
+       struct fpga_fme *priv;
+       int ret = -ENODEV, i = 0;
+       u64 fme_cap, port_offset;
+
+       fme_hdr = get_feature_ioaddr_by_id(&pdev->dev,
+                                          FME_FEATURE_ID_HEADER);
+
+       mutex_lock(&pdata->lock);
+       priv = fpga_pdata_get_private(pdata);
+
+       /* Initialize the region and bridge sub device list */
+       INIT_LIST_HEAD(&priv->region_list);
+       INIT_LIST_HEAD(&priv->bridge_list);
+
+       /* Create fpga mgr platform device */
+       mgr = fpga_fme_create_mgr(pdata);
+       if (IS_ERR(mgr)) {
+               dev_err(&pdev->dev, "fail to create fpga mgr pdev\n");
+               goto unlock;
+       }
+
+       priv->mgr = mgr;
+
+       /* Read capability register to check number of regions and bridges */
+       fme_cap = readq(fme_hdr + FME_HDR_CAP);

I don't think this capability field exists in currently deployed FPGA images
using DFH.  I believe this difference requires a new feature id
to differentiate between deployed FPGA images and images with this new
"feature".

I'm not sure about the "currently deployed FPGA images using DFH", this feature
driver is only for FME PR private feature (Yes, FME's private feature with
id = 0x5). But if they use DFH, so they must have their own feature ids as that
field is in the DFH, isn't it? Then we should be able to distinguish them based
on that.

Thanks
Hao

I am specifically thinking of the FPGA images associated with DCP 1.0.
Should this driver work with those images?  It looks to me that this
capability register is not in the DCP 1.0.

Sure, I verified this patchset on DCP 1.0, PR function works fine. This cap
register is from FME header register set for the number of implemented ports.

Thanks
Hao

quoted

+       for (; i < FIELD_GET(FME_CAP_NUM_PORTS, fme_cap); i++) {
+               port_offset = readq(fme_hdr + FME_HDR_PORT_OFST(i));
+               if (!(port_offset & FME_PORT_OFST_IMP))
+                       continue;
+
+               /* Create bridge for each port */
+               fme_br = fpga_fme_create_bridge(pdata, i);
+               if (IS_ERR(fme_br)) {
+                       ret = PTR_ERR(fme_br);
+                       goto destroy_region;
+               }
+
+               list_add(&fme_br->node, &priv->bridge_list);
+
+               /* Create region for each port */
+               fme_region = fpga_fme_create_region(pdata, mgr, fme_br->br, i);
+               if (!fme_region) {
+                       ret = PTR_ERR(fme_region);
+                       goto destroy_region;
+               }
+
+               list_add(&fme_region->node, &priv->region_list);
+       }
+       mutex_unlock(&pdata->lock);
+
+       return 0;
+
+destroy_region:
+       fpga_fme_destroy_regions(pdata);
+       fpga_fme_destroy_bridges(pdata);
+       fpga_fme_destroy_mgr(pdata);
+unlock:
+       mutex_unlock(&pdata->lock);
+       return ret;
+}
+
+static void pr_mgmt_uinit(struct platform_device *pdev, struct feature *feature)
+{
+       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+       struct fpga_fme *priv;
+
+       mutex_lock(&pdata->lock);
+       priv = fpga_pdata_get_private(pdata);
+
+       fpga_fme_destroy_regions(pdata);
+       fpga_fme_destroy_bridges(pdata);
+       fpga_fme_destroy_mgr(pdata);
+       mutex_unlock(&pdata->lock);
+}
+
+static long fme_pr_ioctl(struct platform_device *pdev, struct feature *feature,
+                        unsigned int cmd, unsigned long arg)
+{
+       long ret;
+
+       switch (cmd) {
+       case FPGA_FME_PORT_PR:
+               ret = fme_pr(pdev, arg);
+               break;
+       default:
+               ret = -ENODEV;
+       }
+
+       return ret;
+}
+
+const struct feature_ops pr_mgmt_ops = {
+       .init = pr_mgmt_init,
+       .uinit = pr_mgmt_uinit,
+       .ioctl = fme_pr_ioctl,
+};

diff --git a/drivers/fpga/dfl-fme-pr.h b/drivers/fpga/dfl-fme-pr.h
new file mode 100644
index 0000000..11bd001
--- /dev/null
+++ b/drivers/fpga/dfl-fme-pr.h

@@ -0,0 +1,113 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Header file for FPGA Management Engine (FME) Partial Reconfiguration Driver
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ */
+
+#ifndef __DFL_FME_PR_H
+#define __DFL_FME_PR_H
+
+#include <linux/platform_device.h>
+
+/**
+ * struct fme_region - FME fpga region data structure
+ *
+ * @region: platform device of the FPGA region.
+ * @node: used to link fme_region to a list.
+ * @port_id: indicate which port this region connected to.
+ */
+struct fme_region {
+       struct platform_device *region;
+       struct list_head node;
+       int port_id;
+};
+
+/**
+ * struct fme_region_pdata - platform data for FME region platform device.
+ *
+ * @mgr: platform device of the FPGA manager.
+ * @br: platform device of the FPGA bridge.
+ * @region_id: region id (same as port_id).
+ */
+struct fme_region_pdata {
+       struct platform_device *mgr;
+       struct platform_device *br;
+       int region_id;
+};
+
+/**
+ * struct fme_bridge - FME fpga bridge data structure
+ *
+ * @br: platform device of the FPGA bridge.
+ * @node: used to link fme_bridge to a list.
+ */
+struct fme_bridge {
+       struct platform_device *br;
+       struct list_head node;
+};
+
+/**
+ * struct fme_bridge_pdata - platform data for FME bridge platform device.
+ *
+ * @port: platform device of the port feature dev.
+ */
+struct fme_br_pdata {
+       struct platform_device *port;
+};
+
+#define FPGA_DFL_FME_MGR       "dfl-fme-mgr"
+#define FPGA_DFL_FME_BRIDGE    "dfl-fme-bridge"
+#define FPGA_DFL_FME_REGION    "dfl-fme-region"
+

Everything in dfl-fme-pr.h up to this point is good and general and
should remain in dfl-fme-pr.h.  The register #defines for this FME's
FPGA manager device below should be associated with the FPGA manager
driver.  Sorry if the way I stated that in the v3 review wasn't clear.

quoted

+/* FME Partial Reconfiguration Sub Feature Register Set */
+#define FME_PR_DFH             0x0
+#define FME_PR_CTRL            0x8
+#define FME_PR_STS             0x10
+#define FME_PR_DATA            0x18
+#define FME_PR_ERR             0x20
+#define FME_PR_INTFC_ID_H      0xA8
+#define FME_PR_INTFC_ID_L      0xB0
+
+/* FME PR Control Register Bitfield */
+#define FME_PR_CTRL_PR_RST     BIT(0)  /* Reset PR engine */
+#define FME_PR_CTRL_PR_RSTACK  BIT(4)  /* Ack for PR engine reset */
+#define FME_PR_CTRL_PR_RGN_ID  GENMASK_ULL(9, 7)       /* PR Region ID */
+#define FME_PR_CTRL_PR_START   BIT(12) /* Start to request for PR service */
+#define FME_PR_CTRL_PR_COMPLETE        BIT(13) /* PR data push complete notification */
+
+/* FME PR Status Register Bitfield */
+/* Number of available entries in HW queue inside the PR engine. */
+#define FME_PR_STS_PR_CREDIT   GENMASK_ULL(8, 0)
+#define FME_PR_STS_PR_STS      BIT(16) /* PR operation status */
+#define FME_PR_STS_PR_STS_IDLE 0
+#define FME_PR_STS_PR_CTRLR_STS        GENMASK_ULL(22, 20)     /* Controller status */
+#define FME_PR_STS_PR_HOST_STS GENMASK_ULL(27, 24)     /* PR host status */
+
+/* FME PR Data Register Bitfield */
+/* PR data from the raw-binary file. */
+#define FME_PR_DATA_PR_DATA_RAW        GENMASK_ULL(32, 0)
+
+/* FME PR Error Register */
+/* PR Operation errors detected. */
+#define FME_PR_ERR_OPERATION_ERR       BIT(0)
+/* CRC error detected. */
+#define FME_PR_ERR_CRC_ERR             BIT(1)
+/* Incompatible PR bitstream detected. */
+#define FME_PR_ERR_INCOMPATIBLE_BS     BIT(2)
+/* PR data push protocol violated. */
+#define FME_PR_ERR_PROTOCOL_ERR                BIT(3)
+/* PR data fifo overflow error detected */
+#define FME_PR_ERR_FIFO_OVERFLOW       BIT(4)

This stuff is specific to this FPGA manager device, so it should
either be in the dfl-fme-mgr.c or in a dfl-fme-mgr.h

quoted

+
+#endif /* __DFL_FME_PR_H */

diff --git a/drivers/fpga/dfl-fme.h b/drivers/fpga/dfl-fme.h
new file mode 100644
index 0000000..c8bd48a
--- /dev/null
+++ b/drivers/fpga/dfl-fme.h

@@ -0,0 +1,38 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Header file for FPGA Management Engine (FME) Driver
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ */
+
+#ifndef __DFL_FME_H
+#define __DFL_FME_H
+
+/**
+ * struct fpga_fme - fpga fme private data
+ *
+ * @mgr: FME's FPGA manager platform device.
+ * @region_list: link list of FME's FPGA regions.
+ * @bridge_list: link list of FME's FPGA bridges.
+ * @pdata: fme platform device's pdata.
+ */
+struct fpga_fme {
+       struct platform_device *mgr;
+       struct list_head region_list;
+       struct list_head bridge_list;
+       struct feature_platform_data *pdata;
+};
+
+extern const struct feature_ops pr_mgmt_ops;
+
+#endif /* __DFL_FME_H */

diff --git a/include/uapi/linux/fpga-dfl.h b/include/uapi/linux/fpga-dfl.h
index 9321ee9..50ee831 100644
--- a/include/uapi/linux/fpga-dfl.h
+++ b/include/uapi/linux/fpga-dfl.h

@@ -14,6 +14,8 @@

#ifndef _UAPI_LINUX_FPGA_DFL_H
#define _UAPI_LINUX_FPGA_DFL_H

+#include <linux/types.h>
+
#define FPGA_API_VERSION 0

/*

@@ -26,6 +28,7 @@

#define FPGA_MAGIC 0xB6

#define FPGA_BASE 0
+#define FME_BASE 0x80

/**
* FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)

@@ -45,4 +48,28 @@

#define FPGA_CHECK_EXTENSION   _IO(FPGA_MAGIC, FPGA_BASE + 1)

+/* IOCTLs for FME file descriptor */
+
+/**
+ * FPGA_FME_PORT_PR - _IOW(FPGA_MAGIC, FME_BASE + 0, struct fpga_fme_port_pr)
+ *
+ * Driver does Partial Reconfiguration based on Port ID and Buffer (Image)
+ * provided by caller.
+ * Return: 0 on success, -errno on failure.
+ * If FPGA_FME_PORT_PR returns -EIO, that indicates the HW has detected
+ * some errors during PR, under this case, the user can fetch HW error info
+ * from the status of FME's fpga manager.
+ */
+
+struct fpga_fme_port_pr {
+       /* Input */
+       __u32 argsz;            /* Structure length */
+       __u32 flags;            /* Zero for now */
+       __u32 port_id;
+       __u32 buffer_size;
+       __u64 buffer_address;   /* Userspace address to the buffer for PR */
+};
+
+#define FPGA_FME_PORT_PR       _IO(FPGA_MAGIC, FME_BASE + 0)
+
#endif /* _UAPI_LINUX_FPGA_DFL_H */
--
2.7.4

Thanks,
Alan
--
To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v4 08/24] fpga: add FPGA DFL PCIe device driver

From: Alan Tull <atull@kernel.org>
Date: 2018-03-13 16:06:11

On Tue, Feb 13, 2018 at 3:24 AM, Wu Hao [off-list ref] wrote:

Hi Hao,

From: Zhang Yi <redacted>

This patch implements the basic framework of the driver for FPGA PCIe
device which implements the Device Feature List (DFL) in its MMIO space.
This driver is verified on Intel(R) PCIe based FPGA DFL devices, including
both integrated (e.g Intel Server Platform with In-package FPGA) and
discrete (e.g Intel FPGA PCIe Acceleration Cards) solutions.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Zhang Yi <redacted>
Signed-off-by: Xiao Guangrong <redacted>
Signed-off-by: Wu Hao <redacted>

 Acked-by: Alan Tull [off-list ref]

Thanks,
Alan

quoted hunk

---
v2: move the code to drivers/fpga folder as suggested by Alan Tull.
    switch to GPLv2 license.
    fix comments from Moritz Fischer.
v3: switch to pci_set_dma_mask/consistent_dma_mask() function.
    remove pci_save_state() in probe function.
    rename driver to INTEL_FPGA_DFL_PCI and intel-dfl-pci.c to indicate
    this driver supports Intel FPGA PCI devices which implement DFL.
    improve Kconfig description for INTEL_FPGA_DFL_PCI
v4: rename to FPGA_DFL_PCI (dfl-pci.c) for better reuse.
    fix SPDX license issue.
---
 drivers/fpga/Kconfig   |  15 ++++++
 drivers/fpga/Makefile  |   3 ++
 drivers/fpga/dfl-pci.c | 127 +++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 145 insertions(+)
 create mode 100644 drivers/fpga/dfl-pci.c

diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
index 01ad31f..87f3d44 100644
--- a/drivers/fpga/Kconfig
+++ b/drivers/fpga/Kconfig

@@ -140,4 +140,19 @@ config FPGA_DFL
          Gate Array (FPGA) solutions which implement Device Feature List.
          It provides enumeration APIs, and feature device infrastructure.

+config FPGA_DFL_PCI
+       tristate "FPGA Device Feature List (DFL) PCIe Device Driver"
+       depends on PCI && FPGA_DFL
+       help
+         Select this option to enable PCIe driver for PCIe based
+         Field-Programmable Gate Array (FPGA) solutions which implemented
+         the Device Feature List (DFL). This driver provides interfaces
+         for userspace applications to configure, enumerate, open and access
+         FPGA accelerators on the FPGA DFL devices, enables system level
+         management functions such as FPGA partial reconfiguration, power
+         management, and virtualization with DFL framework and DFL feature
+         device drivers.
+
+         To compile this as a module, choose M here.
+
 endif # FPGA

diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index c4c62b9..4375630 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile

@@ -30,3 +30,6 @@ obj-$(CONFIG_OF_FPGA_REGION)          += of-fpga-region.o

 # FPGA Device Feature List Support
 obj-$(CONFIG_FPGA_DFL)                 += dfl.o
+
+# Drivers for FPGAs which implement DFL
+obj-$(CONFIG_FPGA_DFL_PCI)             += dfl-pci.o

diff --git a/drivers/fpga/dfl-pci.c b/drivers/fpga/dfl-pci.c
new file mode 100644
index 0000000..d91ea42
--- /dev/null
+++ b/drivers/fpga/dfl-pci.c

@@ -0,0 +1,127 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Driver for FPGA Device Feature List (DFL) PCIe device
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Zhang Yi <Yi.Z.Zhang@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ */
+
+#include <linux/pci.h>
+#include <linux/types.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/stddef.h>
+#include <linux/errno.h>
+#include <linux/aer.h>
+
+#define DRV_VERSION    "0.8"
+#define DRV_NAME       "dfl-pci"
+
+/* PCI Device ID */
+#define PCIE_DEVICE_ID_PF_INT_5_X      0xBCBD
+#define PCIE_DEVICE_ID_PF_INT_6_X      0xBCC0
+#define PCIE_DEVICE_ID_PF_DSC_1_X      0x09C4
+/* VF Device */
+#define PCIE_DEVICE_ID_VF_INT_5_X      0xBCBF
+#define PCIE_DEVICE_ID_VF_INT_6_X      0xBCC1
+#define PCIE_DEVICE_ID_VF_DSC_1_X      0x09C5
+
+static struct pci_device_id cci_pcie_id_tbl[] = {
+       {PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIE_DEVICE_ID_PF_INT_5_X),},
+       {PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIE_DEVICE_ID_VF_INT_5_X),},
+       {PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIE_DEVICE_ID_PF_INT_6_X),},
+       {PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIE_DEVICE_ID_VF_INT_6_X),},
+       {PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIE_DEVICE_ID_PF_DSC_1_X),},
+       {PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIE_DEVICE_ID_VF_DSC_1_X),},
+       {0,}
+};
+MODULE_DEVICE_TABLE(pci, cci_pcie_id_tbl);
+
+static
+int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
+{
+       int ret;
+
+       ret = pci_enable_device(pcidev);
+       if (ret < 0) {
+               dev_err(&pcidev->dev, "Failed to enable device %d.\n", ret);
+               return ret;
+       }
+
+       ret = pci_enable_pcie_error_reporting(pcidev);
+       if (ret && ret != -EINVAL)
+               dev_info(&pcidev->dev, "PCIE AER unavailable %d.\n", ret);
+
+       ret = pci_request_regions(pcidev, DRV_NAME);
+       if (ret) {
+               dev_err(&pcidev->dev, "Failed to request regions.\n");
+               goto disable_error_report_exit;
+       }
+
+       pci_set_master(pcidev);
+
+       if (!pci_set_dma_mask(pcidev, DMA_BIT_MASK(64))) {
+               ret = pci_set_consistent_dma_mask(pcidev, DMA_BIT_MASK(64));
+               if (ret)
+                       goto release_region_exit;
+       } else if (!pci_set_dma_mask(pcidev, DMA_BIT_MASK(32))) {
+               ret = pci_set_consistent_dma_mask(pcidev, DMA_BIT_MASK(32));
+               if (ret)
+                       goto release_region_exit;
+       } else {
+               ret = -EIO;
+               dev_err(&pcidev->dev, "No suitable DMA support available.\n");
+               goto release_region_exit;
+       }
+
+       /* TODO: create and add the platform device per feature list */
+       return 0;
+
+release_region_exit:
+       pci_release_regions(pcidev);
+disable_error_report_exit:
+       pci_disable_pcie_error_reporting(pcidev);
+       pci_disable_device(pcidev);
+       return ret;
+}
+
+static void cci_pci_remove(struct pci_dev *pcidev)
+{
+       pci_release_regions(pcidev);
+       pci_disable_pcie_error_reporting(pcidev);
+       pci_disable_device(pcidev);
+}
+
+static struct pci_driver cci_pci_driver = {
+       .name = DRV_NAME,
+       .id_table = cci_pcie_id_tbl,
+       .probe = cci_pci_probe,
+       .remove = cci_pci_remove,
+};
+
+static int __init ccidrv_init(void)
+{
+       pr_info("FPGA DFL PCIe Driver: Version %s\n", DRV_VERSION);
+
+       return pci_register_driver(&cci_pci_driver);
+}
+
+static void __exit ccidrv_exit(void)
+{
+       pci_unregister_driver(&cci_pci_driver);
+}
+
+module_init(ccidrv_init);
+module_exit(ccidrv_exit);
+
+MODULE_DESCRIPTION("FPGA DFL PCIe Device Driver");
+MODULE_AUTHOR("Intel Corporation");
+MODULE_LICENSE("GPL v2");
--

2.7.4

Re: [PATCH v4 09/24] fpga: dfl-pci: add enumeration for feature devices

From: Alan Tull <atull@kernel.org>
Date: 2018-03-13 18:31:07

On Tue, Feb 13, 2018 at 3:24 AM, Wu Hao [off-list ref] wrote:

Hi Hao,

Thanks again for splitting the pci part of the code from enumeration
and everything else.

One thing that may need to be fixed below, so with that fixed, adding my ack.

The Device Feature List (DFL) is implemented in MMIO, and features
are linked via the DFLs. This patch enables pcie driver to prepare
enumeration information (e.g locations of all device feature lists
in MMIO) and use common APIs provided by the Device Feature List
framework to enumerate each feature device linked.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Zhang Yi <redacted>
Signed-off-by: Xiao Guangrong <redacted>
Signed-off-by: Wu Hao <redacted>

Acked-by: Alan Tull <atull@kernel.org>

quoted hunk

---
v3: split from another patch
    use common functions from DFL framework for enumeration.
v4: rebase
---
 drivers/fpga/dfl-pci.c | 199 ++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 197 insertions(+), 2 deletions(-)

diff --git a/drivers/fpga/dfl-pci.c b/drivers/fpga/dfl-pci.c
index d91ea42..8ce8a94 100644
--- a/drivers/fpga/dfl-pci.c
+++ b/drivers/fpga/dfl-pci.c

@@ -22,9 +22,52 @@
 #include <linux/errno.h>
 #include <linux/aer.h>

+#include "dfl.h"
+
 #define DRV_VERSION    "0.8"
 #define DRV_NAME       "dfl-pci"

+struct cci_drvdata {
+       struct fpga_cdev *cdev; /* container device */
+       struct list_head regions; /* list of pci bar mapping region */
+};
+
+/* pci bar mapping info */
+struct cci_region {
+       int bar;
+       void __iomem *ioaddr;   /* pointer to mapped bar region */
+       struct list_head node;
+};
+
+static void __iomem *cci_pci_ioremap_bar(struct pci_dev *pcidev, int bar)
+{
+       struct cci_drvdata *drvdata = pci_get_drvdata(pcidev);
+       struct cci_region *region;
+
+       list_for_each_entry(region, &drvdata->regions, node)
+               if (region->bar == bar) {
+                       dev_dbg(&pcidev->dev, "BAR %d region exists\n", bar);
+                       return region->ioaddr;
+               }
+
+       region = devm_kzalloc(&pcidev->dev, sizeof(*region), GFP_KERNEL);
+       if (!region)
+               return NULL;
+
+       region->bar = bar;
+       region->ioaddr = pci_ioremap_bar(pcidev, bar);
+       if (!region->ioaddr) {
+               dev_err(&pcidev->dev, "can't ioremap memory from BAR %d.\n",
+                       bar);
+               devm_kfree(&pcidev->dev, region);
+               return NULL;
+       }
+
+       list_add(&region->node, &drvdata->regions);
+
+       return region->ioaddr;
+}
+
 /* PCI Device ID */
 #define PCIE_DEVICE_ID_PF_INT_5_X      0xBCBD
 #define PCIE_DEVICE_ID_PF_INT_6_X      0xBCC0

@@ -45,6 +88,143 @@ static struct pci_device_id cci_pcie_id_tbl[] = {
 };
 MODULE_DEVICE_TABLE(pci, cci_pcie_id_tbl);

+static int cci_init_drvdata(struct pci_dev *pcidev)
+{
+       struct cci_drvdata *drvdata;
+
+       drvdata = devm_kzalloc(&pcidev->dev, sizeof(*drvdata), GFP_KERNEL);
+       if (!drvdata)
+               return -ENOMEM;
+
+       INIT_LIST_HEAD(&drvdata->regions);
+
+       pci_set_drvdata(pcidev, drvdata);
+
+       return 0;
+}
+
+static void cci_pci_release_regions(struct pci_dev *pcidev)
+{
+       struct cci_drvdata *drvdata = pci_get_drvdata(pcidev);
+       struct cci_region *tmp, *region;
+
+       list_for_each_entry_safe(region, tmp, &drvdata->regions, node) {
+               list_del(&region->node);
+               if (region->ioaddr)
+                       pci_iounmap(pcidev, region->ioaddr);
+               devm_kfree(&pcidev->dev, region);
+       }
+}
+
+static void cci_remove_drvdata(struct pci_dev *pcidev)
+{
+       struct cci_drvdata *drvdata = pci_get_drvdata(pcidev);
+
+       cci_pci_release_regions(pcidev);

Would it make sense to call fpga_enum_info_free here?  I understand
fpga_enum_info_alloc uses devm, but it does a get_device which needs
to be put.

+       pci_set_drvdata(pcidev, NULL);
+       devm_kfree(&pcidev->dev, drvdata);
+}
+
+static void cci_remove_feature_devs(struct pci_dev *pcidev)
+{
+       struct cci_drvdata *drvdata = pci_get_drvdata(pcidev);
+
+       /* remove all children feature devices */
+       fpga_remove_feature_devs(drvdata->cdev);
+}
+
+/* enumerate feature devices under pci device */
+static int cci_enumerate_feature_devs(struct pci_dev *pcidev)
+{
+       struct cci_drvdata *drvdata = pci_get_drvdata(pcidev);
+       struct fpga_cdev *cdev;
+       struct fpga_enum_info *info;
+       resource_size_t start, len;
+       void __iomem *base;
+       int port_num, bar, i, ret = 0;
+       u32 offset;
+       u64 v;
+
+       /* allocate enumeration info via pci_dev */
+       info = fpga_enum_info_alloc(&pcidev->dev);
+       if (!info)
+               return -ENOMEM;
+
+       /* start to find Device Feature List from Bar 0 */
+       base = cci_pci_ioremap_bar(pcidev, 0);
+       if (!base) {
+               ret = -ENOMEM;
+               goto enum_info_free_exit;
+       }
+
+       /*
+        * PF device has FME and Ports/AFUs, and VF device only has 1 Port/AFU.
+        * check them and add related "Device Feature List" info for the next
+        * step enumeration.
+        */
+       if (feature_is_fme(base)) {
+               start = pci_resource_start(pcidev, 0);
+               len = pci_resource_len(pcidev, 0);
+
+               fpga_enum_info_add_dfl(info, start, len, base);
+
+               /*
+                * find more Device Feature Lists (e.g Ports) per information
+                * indicated by FME module.
+                */
+               v = readq(base + FME_HDR_CAP);
+               port_num = FIELD_GET(FME_CAP_NUM_PORTS, v);
+
+               WARN_ON(port_num > MAX_FPGA_PORT_NUM);
+
+               for (i = 0; i < port_num; i++) {
+                       v = readq(base + FME_HDR_PORT_OFST(i));
+
+                       /* skip ports which are not implemented. */
+                       if (!(v & FME_PORT_OFST_IMP))
+                               continue;
+
+                       /*
+                        * add Port's Device Feature List information for next
+                        * step enumeration.
+                        */
+                       bar = FIELD_GET(FME_PORT_OFST_BAR_ID, v);
+                       offset = FIELD_GET(FME_PORT_OFST_DFH_OFST, v);
+                       base = cci_pci_ioremap_bar(pcidev, bar);
+                       if (!base)
+                               continue;
+
+                       start = pci_resource_start(pcidev, bar) + offset;
+                       len = pci_resource_len(pcidev, bar) - offset;
+
+                       fpga_enum_info_add_dfl(info, start, len, base + offset);
+               }
+       } else if (feature_is_port(base)) {
+               start = pci_resource_start(pcidev, 0);
+               len = pci_resource_len(pcidev, 0);
+
+               fpga_enum_info_add_dfl(info, start, len, base);
+       } else {
+               ret = -ENODEV;
+               goto enum_info_free_exit;
+       }
+
+       /* start enumeration with prepared enumeration information */
+       cdev = fpga_enumerate_feature_devs(info);
+       if (IS_ERR(cdev)) {
+               dev_err(&pcidev->dev, "Enumeration failure\n");
+               ret = PTR_ERR(cdev);
+               goto enum_info_free_exit;
+       }
+
+       drvdata->cdev = cdev;
+
+enum_info_free_exit:
+       fpga_enum_info_free(info);

This is the only place I saw fpga_enum_info_free being called.

Thanks,
Alan

quoted hunk

+
+       return ret;
+}
+
 static
 int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
 {

@@ -82,9 +262,22 @@ int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
                goto release_region_exit;
        }

-       /* TODO: create and add the platform device per feature list */
-       return 0;
+       ret = cci_init_drvdata(pcidev);
+       if (ret) {
+               dev_err(&pcidev->dev, "Fail to init drvdata %d.\n", ret);
+               goto release_region_exit;
+       }
+
+       ret = cci_enumerate_feature_devs(pcidev);
+       if (ret) {
+               dev_err(&pcidev->dev, "enumeration failure %d.\n", ret);
+               goto remove_drvdata_exit;
+       }
+
+       return ret;

+remove_drvdata_exit:
+       cci_remove_drvdata(pcidev);
 release_region_exit:
        pci_release_regions(pcidev);
 disable_error_report_exit:
@@ -95,6 +288,8 @@ int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)

 static void cci_pci_remove(struct pci_dev *pcidev)
 {
+       cci_remove_feature_devs(pcidev);
+       cci_remove_drvdata(pcidev);
        pci_release_regions(pcidev);
        pci_disable_pcie_error_reporting(pcidev);
        pci_disable_device(pcidev);
--

2.7.4

Re: [PATCH v4 09/24] fpga: dfl-pci: add enumeration for feature devices

From: Wu Hao <hidden>
Date: 2018-03-14 05:31:17

On Tue, Mar 13, 2018 at 01:30:24PM -0500, Alan Tull wrote:

On Tue, Feb 13, 2018 at 3:24 AM, Wu Hao [off-list ref] wrote:

Hi Hao,

Thanks again for splitting the pci part of the code from enumeration
and everything else.

One thing that may need to be fixed below, so with that fixed, adding my ack.

Hi Alan

Thanks a lot for your review and acked-by on these patches,

please see my replies below. : )

quoted

The Device Feature List (DFL) is implemented in MMIO, and features
are linked via the DFLs. This patch enables pcie driver to prepare
enumeration information (e.g locations of all device feature lists
in MMIO) and use common APIs provided by the Device Feature List
framework to enumerate each feature device linked.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Zhang Yi <redacted>
Signed-off-by: Xiao Guangrong <redacted>
Signed-off-by: Wu Hao <redacted>

Acked-by: Alan Tull <atull@kernel.org>

quoted

---
v3: split from another patch
    use common functions from DFL framework for enumeration.
v4: rebase
---
 drivers/fpga/dfl-pci.c | 199 ++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 197 insertions(+), 2 deletions(-)

diff --git a/drivers/fpga/dfl-pci.c b/drivers/fpga/dfl-pci.c
index d91ea42..8ce8a94 100644
--- a/drivers/fpga/dfl-pci.c
+++ b/drivers/fpga/dfl-pci.c

@@ -22,9 +22,52 @@
 #include <linux/errno.h>
 #include <linux/aer.h>

+#include "dfl.h"
+
 #define DRV_VERSION    "0.8"
 #define DRV_NAME       "dfl-pci"

+struct cci_drvdata {
+       struct fpga_cdev *cdev; /* container device */
+       struct list_head regions; /* list of pci bar mapping region */
+};
+
+/* pci bar mapping info */
+struct cci_region {
+       int bar;
+       void __iomem *ioaddr;   /* pointer to mapped bar region */
+       struct list_head node;
+};
+
+static void __iomem *cci_pci_ioremap_bar(struct pci_dev *pcidev, int bar)
+{
+       struct cci_drvdata *drvdata = pci_get_drvdata(pcidev);
+       struct cci_region *region;
+
+       list_for_each_entry(region, &drvdata->regions, node)
+               if (region->bar == bar) {
+                       dev_dbg(&pcidev->dev, "BAR %d region exists\n", bar);
+                       return region->ioaddr;
+               }
+
+       region = devm_kzalloc(&pcidev->dev, sizeof(*region), GFP_KERNEL);
+       if (!region)
+               return NULL;
+
+       region->bar = bar;
+       region->ioaddr = pci_ioremap_bar(pcidev, bar);
+       if (!region->ioaddr) {
+               dev_err(&pcidev->dev, "can't ioremap memory from BAR %d.\n",
+                       bar);
+               devm_kfree(&pcidev->dev, region);
+               return NULL;
+       }
+
+       list_add(&region->node, &drvdata->regions);
+
+       return region->ioaddr;
+}
+
 /* PCI Device ID */
 #define PCIE_DEVICE_ID_PF_INT_5_X      0xBCBD
 #define PCIE_DEVICE_ID_PF_INT_6_X      0xBCC0

@@ -45,6 +88,143 @@ static struct pci_device_id cci_pcie_id_tbl[] = {
 };
 MODULE_DEVICE_TABLE(pci, cci_pcie_id_tbl);

+static int cci_init_drvdata(struct pci_dev *pcidev)
+{
+       struct cci_drvdata *drvdata;
+
+       drvdata = devm_kzalloc(&pcidev->dev, sizeof(*drvdata), GFP_KERNEL);
+       if (!drvdata)
+               return -ENOMEM;
+
+       INIT_LIST_HEAD(&drvdata->regions);
+
+       pci_set_drvdata(pcidev, drvdata);
+
+       return 0;
+}
+
+static void cci_pci_release_regions(struct pci_dev *pcidev)
+{
+       struct cci_drvdata *drvdata = pci_get_drvdata(pcidev);
+       struct cci_region *tmp, *region;
+
+       list_for_each_entry_safe(region, tmp, &drvdata->regions, node) {
+               list_del(&region->node);
+               if (region->ioaddr)
+                       pci_iounmap(pcidev, region->ioaddr);
+               devm_kfree(&pcidev->dev, region);
+       }
+}
+
+static void cci_remove_drvdata(struct pci_dev *pcidev)
+{
+       struct cci_drvdata *drvdata = pci_get_drvdata(pcidev);
+
+       cci_pci_release_regions(pcidev);

Would it make sense to call fpga_enum_info_free here?  I understand
fpga_enum_info_alloc uses devm, but it does a get_device which needs
to be put.

quoted

+       pci_set_drvdata(pcidev, NULL);
+       devm_kfree(&pcidev->dev, drvdata);
+}
+
+static void cci_remove_feature_devs(struct pci_dev *pcidev)
+{
+       struct cci_drvdata *drvdata = pci_get_drvdata(pcidev);
+
+       /* remove all children feature devices */
+       fpga_remove_feature_devs(drvdata->cdev);
+}
+
+/* enumerate feature devices under pci device */
+static int cci_enumerate_feature_devs(struct pci_dev *pcidev)
+{
+       struct cci_drvdata *drvdata = pci_get_drvdata(pcidev);
+       struct fpga_cdev *cdev;
+       struct fpga_enum_info *info;
+       resource_size_t start, len;
+       void __iomem *base;
+       int port_num, bar, i, ret = 0;
+       u32 offset;
+       u64 v;
+
+       /* allocate enumeration info via pci_dev */
+       info = fpga_enum_info_alloc(&pcidev->dev);
+       if (!info)
+               return -ENOMEM;
+
+       /* start to find Device Feature List from Bar 0 */
+       base = cci_pci_ioremap_bar(pcidev, 0);
+       if (!base) {
+               ret = -ENOMEM;
+               goto enum_info_free_exit;
+       }
+
+       /*
+        * PF device has FME and Ports/AFUs, and VF device only has 1 Port/AFU.
+        * check them and add related "Device Feature List" info for the next
+        * step enumeration.
+        */
+       if (feature_is_fme(base)) {
+               start = pci_resource_start(pcidev, 0);
+               len = pci_resource_len(pcidev, 0);
+
+               fpga_enum_info_add_dfl(info, start, len, base);
+
+               /*
+                * find more Device Feature Lists (e.g Ports) per information
+                * indicated by FME module.
+                */
+               v = readq(base + FME_HDR_CAP);
+               port_num = FIELD_GET(FME_CAP_NUM_PORTS, v);
+
+               WARN_ON(port_num > MAX_FPGA_PORT_NUM);
+
+               for (i = 0; i < port_num; i++) {
+                       v = readq(base + FME_HDR_PORT_OFST(i));
+
+                       /* skip ports which are not implemented. */
+                       if (!(v & FME_PORT_OFST_IMP))
+                               continue;
+
+                       /*
+                        * add Port's Device Feature List information for next
+                        * step enumeration.
+                        */
+                       bar = FIELD_GET(FME_PORT_OFST_BAR_ID, v);
+                       offset = FIELD_GET(FME_PORT_OFST_DFH_OFST, v);
+                       base = cci_pci_ioremap_bar(pcidev, bar);
+                       if (!base)
+                               continue;
+
+                       start = pci_resource_start(pcidev, bar) + offset;
+                       len = pci_resource_len(pcidev, bar) - offset;
+
+                       fpga_enum_info_add_dfl(info, start, len, base + offset);
+               }
+       } else if (feature_is_port(base)) {
+               start = pci_resource_start(pcidev, 0);
+               len = pci_resource_len(pcidev, 0);
+
+               fpga_enum_info_add_dfl(info, start, len, base);
+       } else {
+               ret = -ENODEV;
+               goto enum_info_free_exit;
+       }
+
+       /* start enumeration with prepared enumeration information */
+       cdev = fpga_enumerate_feature_devs(info);
+       if (IS_ERR(cdev)) {
+               dev_err(&pcidev->dev, "Enumeration failure\n");
+               ret = PTR_ERR(cdev);
+               goto enum_info_free_exit;
+       }
+
+       drvdata->cdev = cdev;
+
+enum_info_free_exit:
+       fpga_enum_info_free(info);

This is the only place I saw fpga_enum_info_free being called.

It doesn't need to keep the enumeration inforamtion data structure once
the enumeration done, so in the driver, it always did fpga_enum_info_free
once fpga_enumerate_feature_devs(info) returned in this function. so
no need to consider it in other places per my understanding. : )

Thanks
Hao

Thanks,
Alan

quoted

+
+       return ret;
+}
+
 static
 int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
 {

@@ -82,9 +262,22 @@ int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
                goto release_region_exit;
        }

-       /* TODO: create and add the platform device per feature list */
-       return 0;
+       ret = cci_init_drvdata(pcidev);
+       if (ret) {
+               dev_err(&pcidev->dev, "Fail to init drvdata %d.\n", ret);
+               goto release_region_exit;
+       }
+
+       ret = cci_enumerate_feature_devs(pcidev);
+       if (ret) {
+               dev_err(&pcidev->dev, "enumeration failure %d.\n", ret);
+               goto remove_drvdata_exit;
+       }
+
+       return ret;

+remove_drvdata_exit:
+       cci_remove_drvdata(pcidev);
 release_region_exit:
        pci_release_regions(pcidev);
 disable_error_report_exit:
@@ -95,6 +288,8 @@ int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)

 static void cci_pci_remove(struct pci_dev *pcidev)
 {
+       cci_remove_feature_devs(pcidev);
+       cci_remove_drvdata(pcidev);
        pci_release_regions(pcidev);
        pci_disable_pcie_error_reporting(pcidev);
        pci_disable_device(pcidev);
--

2.7.4

Re: [PATCH v4 09/24] fpga: dfl-pci: add enumeration for feature devices

From: Alan Tull <atull@kernel.org>
Date: 2018-03-14 14:49:18

On Wed, Mar 14, 2018 at 12:21 AM, Wu Hao [off-list ref] wrote:

quoted

+
+       drvdata->cdev = cdev;
+
+enum_info_free_exit:
+       fpga_enum_info_free(info);

This is the only place I saw fpga_enum_info_free being called.

It doesn't need to keep the enumeration inforamtion data structure once
the enumeration done, so in the driver, it always did fpga_enum_info_free
once fpga_enumerate_feature_devs(info) returned in this function. so
no need to consider it in other places per my understanding. : )

Thanks
Hao

Oh yes I see!  I was looking right at it and didn't see it!  Thanks
for the clarification.

Alan

Re: [PATCH v4 08/24] fpga: add FPGA DFL PCIe device driver

From: Moritz Fischer <mdf@kernel.org>
Date: 2018-03-15 18:50:01

Hi Hao,

On Tue, Feb 13, 2018 at 05:24:37PM +0800, Wu Hao wrote:

From: Zhang Yi <redacted>

This patch implements the basic framework of the driver for FPGA PCIe
device which implements the Device Feature List (DFL) in its MMIO space.
This driver is verified on Intel(R) PCIe based FPGA DFL devices, including
both integrated (e.g Intel Server Platform with In-package FPGA) and
discrete (e.g Intel FPGA PCIe Acceleration Cards) solutions.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Zhang Yi <redacted>
Signed-off-by: Xiao Guangrong <redacted>
Signed-off-by: Wu Hao <redacted>

with module_pci_driver() fix:

Acked-by: Moritz Fischer <mdf@kernel.org>

quoted hunk

---
v2: move the code to drivers/fpga folder as suggested by Alan Tull.
    switch to GPLv2 license.
    fix comments from Moritz Fischer.
v3: switch to pci_set_dma_mask/consistent_dma_mask() function.
    remove pci_save_state() in probe function.
    rename driver to INTEL_FPGA_DFL_PCI and intel-dfl-pci.c to indicate
    this driver supports Intel FPGA PCI devices which implement DFL.
    improve Kconfig description for INTEL_FPGA_DFL_PCI
v4: rename to FPGA_DFL_PCI (dfl-pci.c) for better reuse.
    fix SPDX license issue.
---
 drivers/fpga/Kconfig   |  15 ++++++
 drivers/fpga/Makefile  |   3 ++
 drivers/fpga/dfl-pci.c | 127 +++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 145 insertions(+)
 create mode 100644 drivers/fpga/dfl-pci.c

diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
index 01ad31f..87f3d44 100644
--- a/drivers/fpga/Kconfig
+++ b/drivers/fpga/Kconfig

@@ -140,4 +140,19 @@ config FPGA_DFL
 	  Gate Array (FPGA) solutions which implement Device Feature List.
 	  It provides enumeration APIs, and feature device infrastructure.
 
+config FPGA_DFL_PCI
+	tristate "FPGA Device Feature List (DFL) PCIe Device Driver"
+	depends on PCI && FPGA_DFL
+	help
+	  Select this option to enable PCIe driver for PCIe based
+	  Field-Programmable Gate Array (FPGA) solutions which implemented
+	  the Device Feature List (DFL). This driver provides interfaces
+	  for userspace applications to configure, enumerate, open and access
+	  FPGA accelerators on the FPGA DFL devices, enables system level
+	  management functions such as FPGA partial reconfiguration, power
+	  management, and virtualization with DFL framework and DFL feature
+	  device drivers.
+
+	  To compile this as a module, choose M here.
+
 endif # FPGA

diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index c4c62b9..4375630 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile

@@ -30,3 +30,6 @@ obj-$(CONFIG_OF_FPGA_REGION)		+= of-fpga-region.o
 
 # FPGA Device Feature List Support
 obj-$(CONFIG_FPGA_DFL)			+= dfl.o
+
+# Drivers for FPGAs which implement DFL
+obj-$(CONFIG_FPGA_DFL_PCI)		+= dfl-pci.o

diff --git a/drivers/fpga/dfl-pci.c b/drivers/fpga/dfl-pci.c
new file mode 100644
index 0000000..d91ea42
--- /dev/null
+++ b/drivers/fpga/dfl-pci.c

@@ -0,0 +1,127 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Driver for FPGA Device Feature List (DFL) PCIe device
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Zhang Yi <Yi.Z.Zhang@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ */
+
+#include <linux/pci.h>
+#include <linux/types.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/stddef.h>
+#include <linux/errno.h>
+#include <linux/aer.h>
+
+#define DRV_VERSION	"0.8"
+#define DRV_NAME	"dfl-pci"
+
+/* PCI Device ID */
+#define PCIE_DEVICE_ID_PF_INT_5_X	0xBCBD
+#define PCIE_DEVICE_ID_PF_INT_6_X	0xBCC0
+#define PCIE_DEVICE_ID_PF_DSC_1_X	0x09C4
+/* VF Device */
+#define PCIE_DEVICE_ID_VF_INT_5_X	0xBCBF
+#define PCIE_DEVICE_ID_VF_INT_6_X	0xBCC1
+#define PCIE_DEVICE_ID_VF_DSC_1_X	0x09C5
+
+static struct pci_device_id cci_pcie_id_tbl[] = {
+	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIE_DEVICE_ID_PF_INT_5_X),},
+	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIE_DEVICE_ID_VF_INT_5_X),},
+	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIE_DEVICE_ID_PF_INT_6_X),},
+	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIE_DEVICE_ID_VF_INT_6_X),},
+	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIE_DEVICE_ID_PF_DSC_1_X),},
+	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIE_DEVICE_ID_VF_DSC_1_X),},
+	{0,}
+};
+MODULE_DEVICE_TABLE(pci, cci_pcie_id_tbl);
+
+static
+int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
+{
+	int ret;
+
+	ret = pci_enable_device(pcidev);
+	if (ret < 0) {
+		dev_err(&pcidev->dev, "Failed to enable device %d.\n", ret);
+		return ret;
+	}
+
+	ret = pci_enable_pcie_error_reporting(pcidev);
+	if (ret && ret != -EINVAL)
+		dev_info(&pcidev->dev, "PCIE AER unavailable %d.\n", ret);
+
+	ret = pci_request_regions(pcidev, DRV_NAME);
+	if (ret) {
+		dev_err(&pcidev->dev, "Failed to request regions.\n");
+		goto disable_error_report_exit;
+	}
+
+	pci_set_master(pcidev);
+
+	if (!pci_set_dma_mask(pcidev, DMA_BIT_MASK(64))) {
+		ret = pci_set_consistent_dma_mask(pcidev, DMA_BIT_MASK(64));
+		if (ret)
+			goto release_region_exit;
+	} else if (!pci_set_dma_mask(pcidev, DMA_BIT_MASK(32))) {
+		ret = pci_set_consistent_dma_mask(pcidev, DMA_BIT_MASK(32));
+		if (ret)
+			goto release_region_exit;
+	} else {
+		ret = -EIO;
+		dev_err(&pcidev->dev, "No suitable DMA support available.\n");
+		goto release_region_exit;
+	}
+
+	/* TODO: create and add the platform device per feature list */
+	return 0;
+
+release_region_exit:
+	pci_release_regions(pcidev);
+disable_error_report_exit:
+	pci_disable_pcie_error_reporting(pcidev);
+	pci_disable_device(pcidev);
+	return ret;
+}
+
+static void cci_pci_remove(struct pci_dev *pcidev)
+{
+	pci_release_regions(pcidev);
+	pci_disable_pcie_error_reporting(pcidev);
+	pci_disable_device(pcidev);
+}
+
+static struct pci_driver cci_pci_driver = {
+	.name = DRV_NAME,
+	.id_table = cci_pcie_id_tbl,
+	.probe = cci_pci_probe,
+	.remove = cci_pci_remove,
+};
+
+static int __init ccidrv_init(void)
+{
+	pr_info("FPGA DFL PCIe Driver: Version %s\n", DRV_VERSION);

Not a fan of the additional output. Can you make it module_pci_driver?

+
+	return pci_register_driver(&cci_pci_driver);
+}
+
+static void __exit ccidrv_exit(void)
+{
+	pci_unregister_driver(&cci_pci_driver);
+}
+
+module_init(ccidrv_init);
+module_exit(ccidrv_exit);
+
+MODULE_DESCRIPTION("FPGA DFL PCIe Device Driver");
+MODULE_AUTHOR("Intel Corporation");
+MODULE_LICENSE("GPL v2");
-- 
2.7.4

Re: [PATCH v4 08/24] fpga: add FPGA DFL PCIe device driver

From: Wu Hao <hidden>
Date: 2018-03-16 04:39:27

On Thu, Mar 15, 2018 at 11:49:56AM -0700, Moritz Fischer wrote:

Hi Hao,

On Tue, Feb 13, 2018 at 05:24:37PM +0800, Wu Hao wrote:

quoted

From: Zhang Yi <redacted>

This patch implements the basic framework of the driver for FPGA PCIe
device which implements the Device Feature List (DFL) in its MMIO space.
This driver is verified on Intel(R) PCIe based FPGA DFL devices, including
both integrated (e.g Intel Server Platform with In-package FPGA) and
discrete (e.g Intel FPGA PCIe Acceleration Cards) solutions.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Zhang Yi <redacted>
Signed-off-by: Xiao Guangrong <redacted>
Signed-off-by: Wu Hao <redacted>

with module_pci_driver() fix:

Acked-by: Moritz Fischer <mdf@kernel.org>

quoted

---
v2: move the code to drivers/fpga folder as suggested by Alan Tull.
    switch to GPLv2 license.
    fix comments from Moritz Fischer.
v3: switch to pci_set_dma_mask/consistent_dma_mask() function.
    remove pci_save_state() in probe function.
    rename driver to INTEL_FPGA_DFL_PCI and intel-dfl-pci.c to indicate
    this driver supports Intel FPGA PCI devices which implement DFL.
    improve Kconfig description for INTEL_FPGA_DFL_PCI
v4: rename to FPGA_DFL_PCI (dfl-pci.c) for better reuse.
    fix SPDX license issue.
---
 drivers/fpga/Kconfig   |  15 ++++++
 drivers/fpga/Makefile  |   3 ++
 drivers/fpga/dfl-pci.c | 127 +++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 145 insertions(+)
 create mode 100644 drivers/fpga/dfl-pci.c

diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
index 01ad31f..87f3d44 100644
--- a/drivers/fpga/Kconfig
+++ b/drivers/fpga/Kconfig

@@ -140,4 +140,19 @@ config FPGA_DFL
 	  Gate Array (FPGA) solutions which implement Device Feature List.
 	  It provides enumeration APIs, and feature device infrastructure.
 
+config FPGA_DFL_PCI
+	tristate "FPGA Device Feature List (DFL) PCIe Device Driver"
+	depends on PCI && FPGA_DFL
+	help
+	  Select this option to enable PCIe driver for PCIe based
+	  Field-Programmable Gate Array (FPGA) solutions which implemented
+	  the Device Feature List (DFL). This driver provides interfaces
+	  for userspace applications to configure, enumerate, open and access
+	  FPGA accelerators on the FPGA DFL devices, enables system level
+	  management functions such as FPGA partial reconfiguration, power
+	  management, and virtualization with DFL framework and DFL feature
+	  device drivers.
+
+	  To compile this as a module, choose M here.
+
 endif # FPGA

diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index c4c62b9..4375630 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile

@@ -30,3 +30,6 @@ obj-$(CONFIG_OF_FPGA_REGION)		+= of-fpga-region.o
 
 # FPGA Device Feature List Support
 obj-$(CONFIG_FPGA_DFL)			+= dfl.o
+
+# Drivers for FPGAs which implement DFL
+obj-$(CONFIG_FPGA_DFL_PCI)		+= dfl-pci.o

diff --git a/drivers/fpga/dfl-pci.c b/drivers/fpga/dfl-pci.c
new file mode 100644
index 0000000..d91ea42
--- /dev/null
+++ b/drivers/fpga/dfl-pci.c

@@ -0,0 +1,127 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Driver for FPGA Device Feature List (DFL) PCIe device
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Zhang Yi <Yi.Z.Zhang@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ */
+
+#include <linux/pci.h>
+#include <linux/types.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/stddef.h>
+#include <linux/errno.h>
+#include <linux/aer.h>
+
+#define DRV_VERSION	"0.8"
+#define DRV_NAME	"dfl-pci"
+
+/* PCI Device ID */
+#define PCIE_DEVICE_ID_PF_INT_5_X	0xBCBD
+#define PCIE_DEVICE_ID_PF_INT_6_X	0xBCC0
+#define PCIE_DEVICE_ID_PF_DSC_1_X	0x09C4
+/* VF Device */
+#define PCIE_DEVICE_ID_VF_INT_5_X	0xBCBF
+#define PCIE_DEVICE_ID_VF_INT_6_X	0xBCC1
+#define PCIE_DEVICE_ID_VF_DSC_1_X	0x09C5
+
+static struct pci_device_id cci_pcie_id_tbl[] = {
+	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIE_DEVICE_ID_PF_INT_5_X),},
+	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIE_DEVICE_ID_VF_INT_5_X),},
+	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIE_DEVICE_ID_PF_INT_6_X),},
+	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIE_DEVICE_ID_VF_INT_6_X),},
+	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIE_DEVICE_ID_PF_DSC_1_X),},
+	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIE_DEVICE_ID_VF_DSC_1_X),},
+	{0,}
+};
+MODULE_DEVICE_TABLE(pci, cci_pcie_id_tbl);
+
+static
+int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
+{
+	int ret;
+
+	ret = pci_enable_device(pcidev);
+	if (ret < 0) {
+		dev_err(&pcidev->dev, "Failed to enable device %d.\n", ret);
+		return ret;
+	}
+
+	ret = pci_enable_pcie_error_reporting(pcidev);
+	if (ret && ret != -EINVAL)
+		dev_info(&pcidev->dev, "PCIE AER unavailable %d.\n", ret);
+
+	ret = pci_request_regions(pcidev, DRV_NAME);
+	if (ret) {
+		dev_err(&pcidev->dev, "Failed to request regions.\n");
+		goto disable_error_report_exit;
+	}
+
+	pci_set_master(pcidev);
+
+	if (!pci_set_dma_mask(pcidev, DMA_BIT_MASK(64))) {
+		ret = pci_set_consistent_dma_mask(pcidev, DMA_BIT_MASK(64));
+		if (ret)
+			goto release_region_exit;
+	} else if (!pci_set_dma_mask(pcidev, DMA_BIT_MASK(32))) {
+		ret = pci_set_consistent_dma_mask(pcidev, DMA_BIT_MASK(32));
+		if (ret)
+			goto release_region_exit;
+	} else {
+		ret = -EIO;
+		dev_err(&pcidev->dev, "No suitable DMA support available.\n");
+		goto release_region_exit;
+	}
+
+	/* TODO: create and add the platform device per feature list */
+	return 0;
+
+release_region_exit:
+	pci_release_regions(pcidev);
+disable_error_report_exit:
+	pci_disable_pcie_error_reporting(pcidev);
+	pci_disable_device(pcidev);
+	return ret;
+}
+
+static void cci_pci_remove(struct pci_dev *pcidev)
+{
+	pci_release_regions(pcidev);
+	pci_disable_pcie_error_reporting(pcidev);
+	pci_disable_device(pcidev);
+}
+
+static struct pci_driver cci_pci_driver = {
+	.name = DRV_NAME,
+	.id_table = cci_pcie_id_tbl,
+	.probe = cci_pci_probe,
+	.remove = cci_pci_remove,
+};
+
+static int __init ccidrv_init(void)
+{
+	pr_info("FPGA DFL PCIe Driver: Version %s\n", DRV_VERSION);

Not a fan of the additional output. Can you make it module_pci_driver?

Hi Moritz,

Thanks a lot for the review and comments.

I will use module_pci_driver and MODULE_VERSION(DRV_VERSION)
instead of the additional output in the next version. :)

Thanks
Hao

quoted

+
+	return pci_register_driver(&cci_pci_driver);
+}
+
+static void __exit ccidrv_exit(void)
+{
+	pci_unregister_driver(&cci_pci_driver);
+}
+
+module_init(ccidrv_init);
+module_exit(ccidrv_exit);
+
+MODULE_DESCRIPTION("FPGA DFL PCIe Device Driver");
+MODULE_AUTHOR("Intel Corporation");
+MODULE_LICENSE("GPL v2");
-- 
2.7.4

Re: [PATCH v4 20/24] fpga: dfl: add FPGA Accelerated Function Unit driver basic framework

From: Alan Tull <atull@kernel.org>
Date: 2018-03-19 18:40:51

On Tue, Feb 13, 2018 at 3:24 AM, Wu Hao [off-list ref] wrote:

Hi Hao,

I acked this back in v2.

On DFL FPGA devices, the Accelerated Function Unit (AFU), can be
reprogrammed for different functions. It connects to the FPGA
infrastructure("blue bistream") via a Port. Port CSRs are implemented
separately from the AFU CSRs to provide control and status of the Port.
Once valid green bitstream is programmed into the AFU, it allows access
to the AFU CSRs in the AFU MMIO space.

This patch only implements basic driver framework for AFU, including
device file operation framework.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Xiao Guangrong <redacted>
Signed-off-by: Wu Hao <redacted>

Acked-by: Alan Tull <atull@kernel.org>

Alan

Re: [PATCH v4 12/24] fpga: dfl: fme: add FPGA_GET_API_VERSION/CHECK_EXTENSION ioctls support

From: Alan Tull <atull@kernel.org>
Date: 2018-03-19 18:51:59

On Tue, Feb 13, 2018 at 3:24 AM, Wu Hao [off-list ref] wrote:

Hi Hao,

Looking at fpga-dfl.h again, there's a lot of things that start with
FPGA_.  Don't you think this is too general?  There are other FPGA_
things in the kernel.  Not just the things added by this patch, but
the whole file.

FPGA_GET_API_VERSION and FPGA_CHECK_EXTENSION ioctls are common ones which

Maybe DFL_FPGA_GET_API_VERSION?

quoted hunk

need to be supported by all feature devices drivers including FME and AFU.
Userspace application can use these ioctl interfaces to get the API info
and check if specific extension is supported or not in current driver.

This patch implements above 2 ioctls in FPGA Management Engine (FME)
driver.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Xiao Guangrong <redacted>
Signed-off-by: Wu Hao <redacted>
Acked-by: Alan Tull <atull@kernel.org>
Acked-by: Moritz Fischer <mdf@kernel.org>
---
v2: switched to GPLv2 license.
v3: rename intel-fpga.h to fpga-dfl.h and rebased.
v4: fix SPDX license issue.
    add Acked-by from Alan and Moritz.
---
 Documentation/ioctl/ioctl-number.txt |  1 +
 drivers/fpga/dfl-fme-main.c          | 12 +++++++++
 include/uapi/linux/fpga-dfl.h        | 48 ++++++++++++++++++++++++++++++++++++
 3 files changed, 61 insertions(+)
 create mode 100644 include/uapi/linux/fpga-dfl.h

diff --git a/Documentation/ioctl/ioctl-number.txt b/Documentation/ioctl/ioctl-number.txt
index 6501389..f1e7baa 100644
--- a/Documentation/ioctl/ioctl-number.txt
+++ b/Documentation/ioctl/ioctl-number.txt

@@ -324,6 +324,7 @@ Code  Seq#(hex)     Include File            Comments
 0xB3   00      linux/mmc/ioctl.h
 0xB4   00-0F   linux/gpio.h            <mailto:linux-gpio@vger.kernel.org>
 0xB5   00-0F   uapi/linux/rpmsg.h      <mailto:linux-remoteproc@vger.kernel.org>
+0xB6   all     linux/fpga-dfl.h
 0xC0   00-0F   linux/usb/iowarrior.h
 0xCA   00-0F   uapi/misc/cxl.h
 0xCA   10-2F   uapi/misc/ocxl.h

diff --git a/drivers/fpga/dfl-fme-main.c b/drivers/fpga/dfl-fme-main.c
index 056ae24..1a9929c 100644
--- a/drivers/fpga/dfl-fme-main.c
+++ b/drivers/fpga/dfl-fme-main.c

@@ -16,6 +16,7 @@

 #include <linux/kernel.h>
 #include <linux/module.h>
+#include <linux/fpga-dfl.h>

 #include "dfl.h"

@@ -114,6 +115,13 @@ static struct feature_driver fme_feature_drvs[] = {
        },
 };

+static long fme_ioctl_check_extension(struct feature_platform_data *pdata,
+                                     unsigned long arg)
+{
+       /* No extension support for now */
+       return 0;
+}
+
 static int fme_open(struct inode *inode, struct file *filp)
 {
        struct platform_device *fdev = fpga_inode_to_feature_dev(inode);

@@ -154,6 +162,10 @@ static long fme_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
        dev_dbg(&pdev->dev, "%s cmd 0x%x\n", __func__, cmd);

        switch (cmd) {
+       case FPGA_GET_API_VERSION:
+               return FPGA_API_VERSION;
+       case FPGA_CHECK_EXTENSION:
+               return fme_ioctl_check_extension(pdata, arg);
        default:
                /*
                 * Let sub-feature's ioctl function to handle the cmd

diff --git a/include/uapi/linux/fpga-dfl.h b/include/uapi/linux/fpga-dfl.h
new file mode 100644
index 0000000..9321ee9
--- /dev/null
+++ b/include/uapi/linux/fpga-dfl.h

@@ -0,0 +1,48 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Header File for FPGA DFL User API
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Zhang Yi <yi.z.zhang@intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ */
+
+#ifndef _UAPI_LINUX_FPGA_DFL_H
+#define _UAPI_LINUX_FPGA_DFL_H
+
+#define FPGA_API_VERSION 0

There will be other FPGA API's.  Maybe DFL_FPGA_... or FME_FPGA_...

+
+/*
+ * The IOCTL interface for DFL based FPGA is designed for extensibility by
+ * embedding the structure length (argsz) and flags into structures passed
+ * between kernel and userspace. This design referenced the VFIO IOCTL
+ * interface (include/uapi/linux/vfio.h).
+ */
+
+#define FPGA_MAGIC 0xB6

Same here.

+
+#define FPGA_BASE 0

And here.

+
+/**
+ * FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)
+ *
+ * Report the version of the driver API.
+ * Return: Driver API Version.
+ */
+
+#define FPGA_GET_API_VERSION   _IO(FPGA_MAGIC, FPGA_BASE + 0)
+
+/**
+ * FPGA_CHECK_EXTENSION - _IO(FPGA_MAGIC, FPGA_BASE + 1)
+ *
+ * Check whether an extension is supported.
+ * Return: 0 if not supported, otherwise the extension is supported.
+ */
+
+#define FPGA_CHECK_EXTENSION   _IO(FPGA_MAGIC, FPGA_BASE + 1)

Basically all the FPGA_* stuff here is specific to this particular DFL
FPGA API and should have a better name.

Alan

+
+#endif /* _UAPI_LINUX_FPGA_DFL_H */
--
2.7.4

Re: [PATCH v4 23/24] fpga: dfl: afu: add user afu sub feature support

From: Alan Tull <atull@kernel.org>
Date: 2018-03-19 20:11:17

On Tue, Feb 13, 2018 at 3:24 AM, Wu Hao [off-list ref] wrote:

Hi Hao,

From: Xiao Guangrong <redacted>

User Accelerated Function Unit sub feature exposes the MMIO region of

Is it 'user accelerated'?  I think it is the Accelerator interface.

the AFU. After valid green bitstream (GBS) is programmed and port is

Would it make sense to just use "partial bitstream" or "PR bitstream"
and "static bitstream" for this patchset?  I don't think that adding
this terminology makes things clearer.  In any case when someone else
uses this patchset, they may not be using this type of branding in
their terminology.

enabled, then this MMIO region could be accessed.

This patch adds support to enumerate the AFU MMIO region and expose it
to userspace via mmap file operation. Below interfaces are exposed to user:

Sysfs interface:
* /sys/class/fpga_region/<regionX>/<dfl-port.x>/afu_id
  Read-only. Indicate which green bitstream is programmed to this AFU.

Here too.

Ioctl interfaces:
* FPGA_PORT_GET_INFO
  Provide info to userspace on the number of supported region.
  Only UAFU region is supported now.

I think UAFU is really the interface for an AFU.  I'd like to not add
more terminology than is really useful here.  Unless I'm missing
something, all the UAFU stuff here is really AFU and not some
different block.

quoted hunk

* FPGA_PORT_GET_REGION_INFO
  Provide region information, including access permission, region size,
  offset from the start of device fd.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Xiao Guangrong <redacted>
Signed-off-by: Wu Hao <redacted>
---
v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
    add sysfs documentation.
    switched to GPLv2 license.
v3: rename driver to fpga-dfl-afu
    fix coding style and checkpatch issue.
    only allow afu_id to be read when port isn't in reset.
v4: rebase and add more comments in code.
    fix SPDX license issue.
---
 Documentation/ABI/testing/sysfs-platform-dfl-port |   9 +
 drivers/fpga/Makefile                             |   2 +-
 drivers/fpga/dfl-afu-main.c                       | 211 +++++++++++++++++++++-
 drivers/fpga/dfl-afu-region.c                     | 165 +++++++++++++++++
 drivers/fpga/dfl-afu.h                            |  71 ++++++++
 include/uapi/linux/fpga-dfl.h                     |  47 +++++
 6 files changed, 501 insertions(+), 4 deletions(-)
 create mode 100644 drivers/fpga/dfl-afu-region.c
 create mode 100644 drivers/fpga/dfl-afu.h

diff --git a/Documentation/ABI/testing/sysfs-platform-dfl-port b/Documentation/ABI/testing/sysfs-platform-dfl-port
index 79e8332..10bda5a 100644
--- a/Documentation/ABI/testing/sysfs-platform-dfl-port
+++ b/Documentation/ABI/testing/sysfs-platform-dfl-port

@@ -5,3 +5,12 @@ Contact:       Wu Hao <hao.wu@intel.com>
 Description:   Read-only. It returns id of this port. One DFL FPGA device
                may have more than one port. Userspace could use this id to
                distinguish different ports under same FPGA device.
+
+What:          /sys/bus/platform/devices/dfl-port.0/afu_id
+Date:          Februray 2018
+KernelVersion:  4.16
+Contact:       Wu Hao <hao.wu@intel.com>
+Description:   Read-only. User can program different green bitstreams (GBS) to
+               FPGA Accelerator Function Unit (AFU) for different functions.
+               It returns uuid which could be used to identify which GBS is
+               programmed in this AFU.

diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index 5c9607b..041e3cd1 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile

@@ -37,7 +37,7 @@ obj-$(CONFIG_FPGA_DFL_FME_REGION)     += dfl-fme-region.o
 obj-$(CONFIG_FPGA_DFL_AFU)             += dfl-afu.o

 dfl-fme-objs := dfl-fme-main.o dfl-fme-pr.o
-dfl-afu-objs := dfl-afu-main.o
+dfl-afu-objs := dfl-afu-main.o dfl-afu-region.o

 # Drivers for FPGAs which implement DFL
 obj-$(CONFIG_FPGA_DFL_PCI)             += dfl-pci.o

diff --git a/drivers/fpga/dfl-afu-main.c b/drivers/fpga/dfl-afu-main.c
index bead242..9c0e4a8 100644
--- a/drivers/fpga/dfl-afu-main.c
+++ b/drivers/fpga/dfl-afu-main.c

@@ -16,9 +16,10 @@

 #include <linux/kernel.h>
 #include <linux/module.h>
+#include <linux/uaccess.h>
 #include <linux/fpga-dfl.h>

-#include "dfl.h"
+#include "dfl-afu.h"

 static ssize_t
 id_show(struct device *dev, struct device_attribute *attr, char *buf)

@@ -78,12 +79,74 @@ static const struct feature_ops port_hdr_ops = {
        .ioctl = port_hdr_ioctl,
 };

+static ssize_t
+afu_id_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+       struct feature_platform_data *pdata = dev_get_platdata(dev);
+       void __iomem *base;
+       u64 guidl, guidh;
+
+       base = get_feature_ioaddr_by_id(dev, PORT_FEATURE_ID_AFU);
+
+       mutex_lock(&pdata->lock);
+       if (pdata->disable_count) {
+               mutex_unlock(&pdata->lock);
+               return -EBUSY;
+       }
+
+       guidl = readq(base + GUID_L);
+       guidh = readq(base + GUID_H);
+       mutex_unlock(&pdata->lock);
+
+       return scnprintf(buf, PAGE_SIZE, "%016llx%016llx\n", guidh, guidl);
+}
+static DEVICE_ATTR_RO(afu_id);
+
+static const struct attribute *port_uafu_attrs[] = {

port_afu_attrs?

+       &dev_attr_afu_id.attr,
+       NULL
+};
+
+static int port_uafu_init(struct platform_device *pdev, struct feature *feature)

port_afu_init?

so on.  The thing's userspace interface doesn't have to be named
differently from the thing itself.

quoted hunk

+{
+       struct resource *res = &pdev->resource[feature->resource_index];
+       u32 flags = FPGA_REGION_READ | FPGA_REGION_WRITE | FPGA_REGION_MMAP;
+       int ret;
+
+       dev_dbg(&pdev->dev, "PORT AFU Init.\n");
+
+       ret = afu_region_add(dev_get_platdata(&pdev->dev),
+                            FPGA_PORT_INDEX_UAFU, resource_size(res),
+                            res->start, flags);
+       if (ret)
+               return ret;
+
+       return sysfs_create_files(&pdev->dev.kobj, port_uafu_attrs);
+}
+
+static void port_uafu_uinit(struct platform_device *pdev,
+                           struct feature *feature)
+{
+       dev_dbg(&pdev->dev, "PORT AFU UInit.\n");
+
+       sysfs_remove_files(&pdev->dev.kobj, port_uafu_attrs);
+}
+
+static const struct feature_ops port_uafu_ops = {
+       .init = port_uafu_init,
+       .uinit = port_uafu_uinit,
+};
+
 static struct feature_driver port_feature_drvs[] = {
        {
                .id = PORT_FEATURE_ID_HEADER,
                .ops = &port_hdr_ops,
        },
        {
+               .id = PORT_FEATURE_ID_AFU,
+               .ops = &port_uafu_ops,
+       },
+       {
                .ops = NULL,
        }
 };

@@ -128,6 +191,64 @@ static long afu_ioctl_check_extension(struct feature_platform_data *pdata,
        return 0;
 }

+static long
+afu_ioctl_get_info(struct feature_platform_data *pdata, void __user *arg)
+{
+       struct fpga_port_info info;
+       struct fpga_afu *afu;
+       unsigned long minsz;
+
+       minsz = offsetofend(struct fpga_port_info, num_umsgs);
+
+       if (copy_from_user(&info, arg, minsz))
+               return -EFAULT;
+
+       if (info.argsz < minsz)
+               return -EINVAL;
+
+       mutex_lock(&pdata->lock);
+       afu = fpga_pdata_get_private(pdata);
+       info.flags = 0;
+       info.num_regions = afu->num_regions;
+       info.num_umsgs = afu->num_umsgs;
+       mutex_unlock(&pdata->lock);
+
+       if (copy_to_user(arg, &info, sizeof(info)))
+               return -EFAULT;
+
+       return 0;
+}
+
+static long
+afu_ioctl_get_region_info(struct feature_platform_data *pdata, void __user *arg)
+{
+       struct fpga_port_region_info rinfo;
+       struct fpga_afu_region region;
+       unsigned long minsz;
+       long ret;
+
+       minsz = offsetofend(struct fpga_port_region_info, offset);
+
+       if (copy_from_user(&rinfo, arg, minsz))
+               return -EFAULT;
+
+       if (rinfo.argsz < minsz || rinfo.padding)
+               return -EINVAL;
+
+       ret = afu_get_region_by_index(pdata, rinfo.index, &region);
+       if (ret)
+               return ret;
+
+       rinfo.flags = region.flags;
+       rinfo.size = region.size;
+       rinfo.offset = region.offset;
+
+       if (copy_to_user(arg, &rinfo, sizeof(rinfo)))
+               return -EFAULT;
+
+       return 0;
+}
+
 static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
 {
        struct platform_device *pdev = filp->private_data;
@@ -142,6 +263,10 @@ static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
                return FPGA_API_VERSION;
        case FPGA_CHECK_EXTENSION:

I commented elsewhere about the FPGA_* names.  These names such as
FPGA_CHECK_EXTENSION are specific to this DFL patchset, they weren't
written as addtions to the lower level FPGA stuff.

quoted hunk

                return afu_ioctl_check_extension(pdata, arg);
+       case FPGA_PORT_GET_INFO:
+               return afu_ioctl_get_info(pdata, (void __user *)arg);
+       case FPGA_PORT_GET_REGION_INFO:
+               return afu_ioctl_get_region_info(pdata, (void __user *)arg);
        default:
                /*
                 * Let sub-feature's ioctl function to handle the cmd

@@ -162,27 +287,106 @@ static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
        return -EINVAL;
 }

+static int afu_mmap(struct file *filp, struct vm_area_struct *vma)
+{
+       struct fpga_afu_region region;
+       struct platform_device *pdev = filp->private_data;
+       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+       u64 size = vma->vm_end - vma->vm_start;
+       u64 offset;
+       int ret;
+
+       if (!(vma->vm_flags & VM_SHARED))
+               return -EINVAL;
+
+       offset = vma->vm_pgoff << PAGE_SHIFT;
+       ret = afu_get_region_by_offset(pdata, offset, size, &region);
+       if (ret)
+               return ret;
+
+       if (!(region.flags & FPGA_REGION_MMAP))

FPGA_REGION_*?  We already have FPGA regions which are something
different.  Please call this something else.

quoted hunk

+               return -EINVAL;
+
+       if ((vma->vm_flags & VM_READ) && !(region.flags & FPGA_REGION_READ))
+               return -EPERM;
+
+       if ((vma->vm_flags & VM_WRITE) && !(region.flags & FPGA_REGION_WRITE))
+               return -EPERM;
+
+       vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
+
+       return remap_pfn_range(vma, vma->vm_start,
+                       (region.phys + (offset - region.offset)) >> PAGE_SHIFT,
+                       size, vma->vm_page_prot);
+}
+
 static const struct file_operations afu_fops = {
        .owner = THIS_MODULE,
        .open = afu_open,
        .release = afu_release,
        .unlocked_ioctl = afu_ioctl,
+       .mmap = afu_mmap,
 };

+static int afu_dev_init(struct platform_device *pdev)
+{
+       struct fpga_afu *afu;
+       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+
+       afu = devm_kzalloc(&pdev->dev, sizeof(*afu), GFP_KERNEL);
+       if (!afu)
+               return -ENOMEM;
+
+       afu->pdata = pdata;
+
+       mutex_lock(&pdata->lock);
+       fpga_pdata_set_private(pdata, afu);
+       afu_region_init(pdata);
+       mutex_unlock(&pdata->lock);
+       return 0;
+}
+
+static int afu_dev_destroy(struct platform_device *pdev)
+{
+       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+       struct fpga_afu *afu;
+
+       mutex_lock(&pdata->lock);
+       afu = fpga_pdata_get_private(pdata);
+       afu_region_destroy(pdata);
+       fpga_pdata_set_private(pdata, NULL);
+       mutex_unlock(&pdata->lock);
+
+       devm_kfree(&pdev->dev, afu);
+
+       return 0;
+}
+
 static int afu_probe(struct platform_device *pdev)
 {
        int ret;

        dev_dbg(&pdev->dev, "%s\n", __func__);

+       ret = afu_dev_init(pdev);
+       if (ret)
+               goto exit;
+
        ret = fpga_dev_feature_init(pdev, port_feature_drvs);
        if (ret)
-               return ret;
+               goto dev_destroy;

        ret = fpga_register_dev_ops(pdev, &afu_fops, THIS_MODULE);
-       if (ret)
+       if (ret) {
                fpga_dev_feature_uinit(pdev);
+               goto dev_destroy;
+       }
+
+       return 0;

+dev_destroy:
+       afu_dev_destroy(pdev);
+exit:
        return ret;
 }

@@ -192,6 +396,7 @@ static int afu_remove(struct platform_device *pdev)

        fpga_dev_feature_uinit(pdev);
        fpga_unregister_dev_ops(pdev);
+       afu_dev_destroy(pdev);

        return 0;
 }

diff --git a/drivers/fpga/dfl-afu-region.c b/drivers/fpga/dfl-afu-region.c
new file mode 100644
index 0000000..e3a467b
--- /dev/null
+++ b/drivers/fpga/dfl-afu-region.c

@@ -0,0 +1,165 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Driver for FPGA Accelerated Function Unit (AFU) Region Management
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Wu Hao <hao.wu@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ */
+#include "dfl-afu.h"
+
+/**
+ * afu_region_init - init function for afu region support
+ * @pdata: afu platform device's pdata.
+ */
+void afu_region_init(struct feature_platform_data *pdata)

These afu regions are memory regions?  We also have 'FPGA regions'
which are partial reconfiguration areas.  It probably would be really
helpful to name things to make it more clear what these are and that
they are not what drivers/fpga already calls "FPGA regions".

quoted hunk

+{
+       struct fpga_afu *afu = fpga_pdata_get_private(pdata);
+
+       INIT_LIST_HEAD(&afu->regions);
+}
+
+#define for_each_region(region, afu)   \
+       list_for_each_entry((region), &(afu)->regions, node)
+
+static struct fpga_afu_region *get_region_by_index(struct fpga_afu *afu,
+                                                  u32 region_index)
+{
+       struct fpga_afu_region *region;
+
+       for_each_region(region, afu)
+               if (region->index == region_index)
+                       return region;
+
+       return NULL;
+}
+
+/**
+ * afu_region_add - add a region to given feature dev.
+ *
+ * @region_index: region index.
+ * @region_size: region size.
+ * @phys: region's physical address of this region.
+ * @flags: region flags (access permission).
+ *
+ * Return: 0 on success, negative error code otherwise.
+ */
+int afu_region_add(struct feature_platform_data *pdata, u32 region_index,
+                  u64 region_size, u64 phys, u32 flags)
+{
+       struct fpga_afu_region *region;
+       struct fpga_afu *afu;
+       int ret = 0;
+
+       region = devm_kzalloc(&pdata->dev->dev, sizeof(*region), GFP_KERNEL);
+       if (!region)
+               return -ENOMEM;
+
+       region->index = region_index;
+       region->size = region_size;
+       region->phys = phys;
+       region->flags = flags;
+
+       mutex_lock(&pdata->lock);
+
+       afu = fpga_pdata_get_private(pdata);
+
+       /* check if @index already exists */
+       if (get_region_by_index(afu, region_index)) {
+               mutex_unlock(&pdata->lock);
+               ret = -EEXIST;
+               goto exit;
+       }
+
+       region_size = PAGE_ALIGN(region_size);
+       region->offset = afu->region_cur_offset;
+       list_add(&region->node, &afu->regions);
+
+       afu->region_cur_offset += region_size;
+       afu->num_regions++;
+       mutex_unlock(&pdata->lock);
+
+       return 0;
+
+exit:
+       devm_kfree(&pdata->dev->dev, region);
+       return ret;
+}
+
+/**
+ * afu_region_destroy - destroy all regions under given feature dev.
+ * @pdata: afu platform device's pdata.
+ */
+void afu_region_destroy(struct feature_platform_data *pdata)
+{
+       struct fpga_afu_region *tmp, *region;
+       struct fpga_afu *afu = fpga_pdata_get_private(pdata);
+
+       list_for_each_entry_safe(region, tmp, &afu->regions, node)
+               devm_kfree(&pdata->dev->dev, region);
+}
+
+/**
+ * afu_get_region_by_index - find an afu region by index.
+ * @pdata: afu platform device's pdata.
+ * @region_index: region index.
+ * @pregion: ptr to region for result.
+ *
+ * Return: 0 on success, negative error code otherwise.
+ */
+int afu_get_region_by_index(struct feature_platform_data *pdata,
+                           u32 region_index, struct fpga_afu_region *pregion)
+{
+       struct fpga_afu_region *region;
+       struct fpga_afu *afu;
+       int ret = 0;
+
+       mutex_lock(&pdata->lock);
+       afu = fpga_pdata_get_private(pdata);
+       region = get_region_by_index(afu, region_index);
+       if (!region) {
+               ret = -EINVAL;
+               goto exit;
+       }
+       *pregion = *region;
+exit:
+       mutex_unlock(&pdata->lock);
+       return ret;
+}
+
+/**
+ * afu_get_region_by_offset - find an afu region by offset and size
+ *
+ * @pdata: afu platform device's pdata.
+ * @offset: region offset from start of the device fd.
+ * @size: region size.
+ * @pregion: ptr to region for result.
+ *
+ * Find the region which fully contains the region described by input
+ * parameters (offset and size) from the feature dev's region link list.
+ *
+ * Return: 0 on success, negative error code otherwise.
+ */
+int afu_get_region_by_offset(struct feature_platform_data *pdata,
+                            u64 offset, u64 size,
+                            struct fpga_afu_region *pregion)
+{
+       struct fpga_afu_region *region;
+       struct fpga_afu *afu;
+       int ret = 0;
+
+       mutex_lock(&pdata->lock);
+       afu = fpga_pdata_get_private(pdata);
+       for_each_region(region, afu)
+               if (region->offset <= offset &&
+                   region->offset + region->size >= offset + size) {
+                       *pregion = *region;
+                       goto exit;
+               }
+       ret = -EINVAL;
+exit:
+       mutex_unlock(&pdata->lock);
+       return ret;
+}

diff --git a/drivers/fpga/dfl-afu.h b/drivers/fpga/dfl-afu.h
new file mode 100644
index 0000000..af690cc
--- /dev/null
+++ b/drivers/fpga/dfl-afu.h

@@ -0,0 +1,71 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Header file for FPGA Accelerated Function Unit (AFU) Driver
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *     Wu Hao <hao.wu@intel.com>
+ *     Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *     Joseph Grecco <joe.grecco@intel.com>
+ *     Enno Luebbers <enno.luebbers@intel.com>
+ *     Tim Whisonant <tim.whisonant@intel.com>
+ *     Ananda Ravuri <ananda.ravuri@intel.com>
+ *     Henry Mitchel <henry.mitchel@intel.com>
+ */
+
+#ifndef __DFL_AFU_H
+#define __DFL_AFU_H
+
+#include <linux/mm.h>
+
+#include "dfl.h"
+
+/**
+ * struct fpga_afu_region - afu region data structure
+ *
+ * @index: region index.
+ * @flags: region flags (access permission).
+ * @size: region size.
+ * @offset: region offset from start of the device fd.
+ * @phys: region's physical address.
+ * @node: node to add to afu feature dev's region list.
+ */
+struct fpga_afu_region {
+       u32 index;
+       u32 flags;
+       u64 size;
+       u64 offset;
+       u64 phys;
+       struct list_head node;
+};
+
+/**
+ * struct fpga_afu - afu device data structure
+ *
+ * @region_cur_offset: current region offset from start to the device fd.
+ * @num_regions: num of regions.
+ * @regions: the region link list of this afu feature device.
+ * @num_umsgs: num of umsgs.
+ * @pdata: afu platform device's pdata.
+ */
+struct fpga_afu {
+       u64 region_cur_offset;
+       int num_regions;
+       u8 num_umsgs;
+       struct list_head regions;
+
+       struct feature_platform_data *pdata;
+};
+
+void afu_region_init(struct feature_platform_data *pdata);
+int afu_region_add(struct feature_platform_data *pdata, u32 region_index,
+                  u64 region_size, u64 phys, u32 flags);
+void afu_region_destroy(struct feature_platform_data *pdata);
+int afu_get_region_by_index(struct feature_platform_data *pdata,
+                           u32 region_index, struct fpga_afu_region *pregion);
+int afu_get_region_by_offset(struct feature_platform_data *pdata,
+                            u64 offset, u64 size,
+                            struct fpga_afu_region *pregion);
+
+#endif

diff --git a/include/uapi/linux/fpga-dfl.h b/include/uapi/linux/fpga-dfl.h
index 727bec2..df41828 100644
--- a/include/uapi/linux/fpga-dfl.h
+++ b/include/uapi/linux/fpga-dfl.h

@@ -65,6 +65,53 @@

 #define FPGA_PORT_RESET                _IO(FPGA_MAGIC, PORT_BASE + 0)

+/**
+ * FPGA_PORT_GET_INFO - _IOR(FPGA_MAGIC, PORT_BASE + 1, struct fpga_port_info)
+ *
+ * Retrieve information about the fpga port.
+ * Driver fills the info in provided struct fpga_port_info.
+ * Return: 0 on success, -errno on failure.
+ */
+struct fpga_port_info {
+       /* Input */
+       __u32 argsz;            /* Structure length */
+       /* Output */
+       __u32 flags;            /* Zero for now */
+       __u32 num_regions;      /* The number of supported regions */
+       __u32 num_umsgs;        /* The number of allocated umsgs */
+};
+
+#define FPGA_PORT_GET_INFO     _IO(FPGA_MAGIC, PORT_BASE + 1)
+
+/**
+ * FPGA_PORT_GET_REGION_INFO - _IOWR(FPGA_MAGIC, PORT_BASE + 2,
+ *                                             struct fpga_port_region_info)
+ *
+ * Retrieve information about a device region.
+ * Caller provides struct fpga_port_region_info with index value set.
+ * Driver returns the region info in other fields.
+ * Return: 0 on success, -errno on failure.
+ */
+struct fpga_port_region_info {
+       /* input */
+       __u32 argsz;            /* Structure length */
+       /* Output */
+       __u32 flags;            /* Access permission */
+#define FPGA_REGION_READ       (1 << 0)        /* Region is readable */
+#define FPGA_REGION_WRITE      (1 << 1)        /* Region is writable */
+#define FPGA_REGION_MMAP       (1 << 2)        /* Can be mmaped to userspace */
+       /* Input */
+       __u32 index;            /* Region index */
+#define FPGA_PORT_INDEX_UAFU   0               /* User AFU */
+#define FPGA_PORT_INDEX_STP    1               /* Signal Tap */
+       __u32 padding;
+       /* Output */
+       __u64 size;             /* Region size (bytes) */
+       __u64 offset;           /* Region offset from start of device fd */
+};
+
+#define FPGA_PORT_GET_REGION_INFO      _IO(FPGA_MAGIC, PORT_BASE + 2)
+
 /* IOCTLs for FME file descriptor */

 /**
--

2.7.4

Thanks,
Alan

Re: [PATCH v4 12/24] fpga: dfl: fme: add FPGA_GET_API_VERSION/CHECK_EXTENSION ioctls support

From: Wu Hao <hidden>
Date: 2018-03-20 06:57:01

On Mon, Mar 19, 2018 at 01:29:52PM -0500, Alan Tull wrote:

On Tue, Feb 13, 2018 at 3:24 AM, Wu Hao [off-list ref] wrote:

Hi Hao,

Looking at fpga-dfl.h again, there's a lot of things that start with
FPGA_.  Don't you think this is too general?  There are other FPGA_
things in the kernel.  Not just the things added by this patch, but
the whole file.

Hi Alan

Thanks for the comments, I got your point here. FPGA_ things should be 
reserved for more generic things for FPGA. I will search and replace
the FPGA_xxx things used here and in other files.

quoted

FPGA_GET_API_VERSION and FPGA_CHECK_EXTENSION ioctls are common ones which

Maybe DFL_FPGA_GET_API_VERSION?

Yes, DFL_FPGA_GET_API_VERSION looks good.

quoted

need to be supported by all feature devices drivers including FME and AFU.
Userspace application can use these ioctl interfaces to get the API info
and check if specific extension is supported or not in current driver.

This patch implements above 2 ioctls in FPGA Management Engine (FME)
driver.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Xiao Guangrong <redacted>
Signed-off-by: Wu Hao <redacted>
Acked-by: Alan Tull <atull@kernel.org>
Acked-by: Moritz Fischer <mdf@kernel.org>
---
v2: switched to GPLv2 license.
v3: rename intel-fpga.h to fpga-dfl.h and rebased.
v4: fix SPDX license issue.
    add Acked-by from Alan and Moritz.
---
 Documentation/ioctl/ioctl-number.txt |  1 +
 drivers/fpga/dfl-fme-main.c          | 12 +++++++++
 include/uapi/linux/fpga-dfl.h        | 48 ++++++++++++++++++++++++++++++++++++
 3 files changed, 61 insertions(+)
 create mode 100644 include/uapi/linux/fpga-dfl.h

diff --git a/Documentation/ioctl/ioctl-number.txt b/Documentation/ioctl/ioctl-number.txt
index 6501389..f1e7baa 100644
--- a/Documentation/ioctl/ioctl-number.txt
+++ b/Documentation/ioctl/ioctl-number.txt

@@ -324,6 +324,7 @@ Code  Seq#(hex)     Include File            Comments
 0xB3   00      linux/mmc/ioctl.h
 0xB4   00-0F   linux/gpio.h            <mailto:linux-gpio@vger.kernel.org>
 0xB5   00-0F   uapi/linux/rpmsg.h      <mailto:linux-remoteproc@vger.kernel.org>
+0xB6   all     linux/fpga-dfl.h
 0xC0   00-0F   linux/usb/iowarrior.h
 0xCA   00-0F   uapi/misc/cxl.h
 0xCA   10-2F   uapi/misc/ocxl.h

diff --git a/drivers/fpga/dfl-fme-main.c b/drivers/fpga/dfl-fme-main.c
index 056ae24..1a9929c 100644
--- a/drivers/fpga/dfl-fme-main.c
+++ b/drivers/fpga/dfl-fme-main.c

@@ -16,6 +16,7 @@

 #include <linux/kernel.h>
 #include <linux/module.h>
+#include <linux/fpga-dfl.h>

 #include "dfl.h"

@@ -114,6 +115,13 @@ static struct feature_driver fme_feature_drvs[] = {
        },
 };

+static long fme_ioctl_check_extension(struct feature_platform_data *pdata,
+                                     unsigned long arg)
+{
+       /* No extension support for now */
+       return 0;
+}
+
 static int fme_open(struct inode *inode, struct file *filp)
 {
        struct platform_device *fdev = fpga_inode_to_feature_dev(inode);

@@ -154,6 +162,10 @@ static long fme_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
        dev_dbg(&pdev->dev, "%s cmd 0x%x\n", __func__, cmd);

        switch (cmd) {
+       case FPGA_GET_API_VERSION:
+               return FPGA_API_VERSION;
+       case FPGA_CHECK_EXTENSION:
+               return fme_ioctl_check_extension(pdata, arg);
        default:
                /*
                 * Let sub-feature's ioctl function to handle the cmd

diff --git a/include/uapi/linux/fpga-dfl.h b/include/uapi/linux/fpga-dfl.h
new file mode 100644
index 0000000..9321ee9
--- /dev/null
+++ b/include/uapi/linux/fpga-dfl.h

@@ -0,0 +1,48 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Header File for FPGA DFL User API
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Zhang Yi <yi.z.zhang@intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ */
+
+#ifndef _UAPI_LINUX_FPGA_DFL_H
+#define _UAPI_LINUX_FPGA_DFL_H
+
+#define FPGA_API_VERSION 0

There will be other FPGA API's.  Maybe DFL_FPGA_... or FME_FPGA_...

I will pick DFL_FPGA_... here and below places. :)

quoted

+
+/*
+ * The IOCTL interface for DFL based FPGA is designed for extensibility by
+ * embedding the structure length (argsz) and flags into structures passed
+ * between kernel and userspace. This design referenced the VFIO IOCTL
+ * interface (include/uapi/linux/vfio.h).
+ */
+
+#define FPGA_MAGIC 0xB6

Same here.

quoted

+
+#define FPGA_BASE 0

And here.

quoted

+
+/**
+ * FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)
+ *
+ * Report the version of the driver API.
+ * Return: Driver API Version.
+ */
+
+#define FPGA_GET_API_VERSION   _IO(FPGA_MAGIC, FPGA_BASE + 0)
+
+/**
+ * FPGA_CHECK_EXTENSION - _IO(FPGA_MAGIC, FPGA_BASE + 1)
+ *
+ * Check whether an extension is supported.
+ * Return: 0 if not supported, otherwise the extension is supported.
+ */
+
+#define FPGA_CHECK_EXTENSION   _IO(FPGA_MAGIC, FPGA_BASE + 1)

Basically all the FPGA_* stuff here is specific to this particular DFL
FPGA API and should have a better name.

Will update them in the next version, thanks again for the comments.

Hao

Alan

quoted

+
+#endif /* _UAPI_LINUX_FPGA_DFL_H */
--
2.7.4

Re: [PATCH v4 23/24] fpga: dfl: afu: add user afu sub feature support

From: Wu Hao <hidden>
Date: 2018-03-20 07:20:40

On Mon, Mar 19, 2018 at 03:10:28PM -0500, Alan Tull wrote:

On Tue, Feb 13, 2018 at 3:24 AM, Wu Hao [off-list ref] wrote:

Hi Hao,

quoted

From: Xiao Guangrong <redacted>

User Accelerated Function Unit sub feature exposes the MMIO region of

Is it 'user accelerated'?  I think it is the Accelerator interface.

Hi Alan,

This is only used to emphasize this is the interface to accelerator
exposed to user. But looks like this causes some confusions for user
actually from the description. I agree with you, that I will remove
this UAFU from this patchset.

quoted

the AFU. After valid green bitstream (GBS) is programmed and port is

Would it make sense to just use "partial bitstream" or "PR bitstream"
and "static bitstream" for this patchset?  I don't think that adding
this terminology makes things clearer.  In any case when someone else
uses this patchset, they may not be using this type of branding in
their terminology.

Sure, will update the commit message and also sysfs doc below.

quoted

enabled, then this MMIO region could be accessed.

This patch adds support to enumerate the AFU MMIO region and expose it
to userspace via mmap file operation. Below interfaces are exposed to user:

Sysfs interface:
* /sys/class/fpga_region/<regionX>/<dfl-port.x>/afu_id
  Read-only. Indicate which green bitstream is programmed to this AFU.

Here too.

quoted

Ioctl interfaces:
* FPGA_PORT_GET_INFO
  Provide info to userspace on the number of supported region.
  Only UAFU region is supported now.

I think UAFU is really the interface for an AFU.  I'd like to not add
more terminology than is really useful here.  Unless I'm missing
something, all the UAFU stuff here is really AFU and not some
different block.

Yes, you're right, will remove UAFU from this patchset.

quoted

* FPGA_PORT_GET_REGION_INFO
  Provide region information, including access permission, region size,
  offset from the start of device fd.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Xiao Guangrong <redacted>
Signed-off-by: Wu Hao <redacted>
---
v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
    add sysfs documentation.
    switched to GPLv2 license.
v3: rename driver to fpga-dfl-afu
    fix coding style and checkpatch issue.
    only allow afu_id to be read when port isn't in reset.
v4: rebase and add more comments in code.
    fix SPDX license issue.
---
 Documentation/ABI/testing/sysfs-platform-dfl-port |   9 +
 drivers/fpga/Makefile                             |   2 +-
 drivers/fpga/dfl-afu-main.c                       | 211 +++++++++++++++++++++-
 drivers/fpga/dfl-afu-region.c                     | 165 +++++++++++++++++
 drivers/fpga/dfl-afu.h                            |  71 ++++++++
 include/uapi/linux/fpga-dfl.h                     |  47 +++++
 6 files changed, 501 insertions(+), 4 deletions(-)
 create mode 100644 drivers/fpga/dfl-afu-region.c
 create mode 100644 drivers/fpga/dfl-afu.h

diff --git a/Documentation/ABI/testing/sysfs-platform-dfl-port b/Documentation/ABI/testing/sysfs-platform-dfl-port
index 79e8332..10bda5a 100644
--- a/Documentation/ABI/testing/sysfs-platform-dfl-port
+++ b/Documentation/ABI/testing/sysfs-platform-dfl-port

@@ -5,3 +5,12 @@ Contact:       Wu Hao <hao.wu@intel.com>
 Description:   Read-only. It returns id of this port. One DFL FPGA device
                may have more than one port. Userspace could use this id to
                distinguish different ports under same FPGA device.
+
+What:          /sys/bus/platform/devices/dfl-port.0/afu_id
+Date:          Februray 2018
+KernelVersion:  4.16
+Contact:       Wu Hao <hao.wu@intel.com>
+Description:   Read-only. User can program different green bitstreams (GBS) to
+               FPGA Accelerator Function Unit (AFU) for different functions.
+               It returns uuid which could be used to identify which GBS is
+               programmed in this AFU.

diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index 5c9607b..041e3cd1 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile

@@ -37,7 +37,7 @@ obj-$(CONFIG_FPGA_DFL_FME_REGION)     += dfl-fme-region.o
 obj-$(CONFIG_FPGA_DFL_AFU)             += dfl-afu.o

 dfl-fme-objs := dfl-fme-main.o dfl-fme-pr.o
-dfl-afu-objs := dfl-afu-main.o
+dfl-afu-objs := dfl-afu-main.o dfl-afu-region.o

 # Drivers for FPGAs which implement DFL
 obj-$(CONFIG_FPGA_DFL_PCI)             += dfl-pci.o

diff --git a/drivers/fpga/dfl-afu-main.c b/drivers/fpga/dfl-afu-main.c
index bead242..9c0e4a8 100644
--- a/drivers/fpga/dfl-afu-main.c
+++ b/drivers/fpga/dfl-afu-main.c

@@ -16,9 +16,10 @@

 #include <linux/kernel.h>
 #include <linux/module.h>
+#include <linux/uaccess.h>
 #include <linux/fpga-dfl.h>

-#include "dfl.h"
+#include "dfl-afu.h"

 static ssize_t
 id_show(struct device *dev, struct device_attribute *attr, char *buf)

@@ -78,12 +79,74 @@ static const struct feature_ops port_hdr_ops = {
        .ioctl = port_hdr_ioctl,
 };

+static ssize_t
+afu_id_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+       struct feature_platform_data *pdata = dev_get_platdata(dev);
+       void __iomem *base;
+       u64 guidl, guidh;
+
+       base = get_feature_ioaddr_by_id(dev, PORT_FEATURE_ID_AFU);
+
+       mutex_lock(&pdata->lock);
+       if (pdata->disable_count) {
+               mutex_unlock(&pdata->lock);
+               return -EBUSY;
+       }
+
+       guidl = readq(base + GUID_L);
+       guidh = readq(base + GUID_H);
+       mutex_unlock(&pdata->lock);
+
+       return scnprintf(buf, PAGE_SIZE, "%016llx%016llx\n", guidh, guidl);
+}
+static DEVICE_ATTR_RO(afu_id);
+
+static const struct attribute *port_uafu_attrs[] = {

port_afu_attrs?

quoted

+       &dev_attr_afu_id.attr,
+       NULL
+};
+
+static int port_uafu_init(struct platform_device *pdev, struct feature *feature)

port_afu_init?

so on.  The thing's userspace interface doesn't have to be named
differently from the thing itself.

Agree, will fix them.

quoted

+{
+       struct resource *res = &pdev->resource[feature->resource_index];
+       u32 flags = FPGA_REGION_READ | FPGA_REGION_WRITE | FPGA_REGION_MMAP;
+       int ret;
+
+       dev_dbg(&pdev->dev, "PORT AFU Init.\n");
+
+       ret = afu_region_add(dev_get_platdata(&pdev->dev),
+                            FPGA_PORT_INDEX_UAFU, resource_size(res),
+                            res->start, flags);
+       if (ret)
+               return ret;
+
+       return sysfs_create_files(&pdev->dev.kobj, port_uafu_attrs);
+}
+
+static void port_uafu_uinit(struct platform_device *pdev,
+                           struct feature *feature)
+{
+       dev_dbg(&pdev->dev, "PORT AFU UInit.\n");
+
+       sysfs_remove_files(&pdev->dev.kobj, port_uafu_attrs);
+}
+
+static const struct feature_ops port_uafu_ops = {
+       .init = port_uafu_init,
+       .uinit = port_uafu_uinit,
+};
+
 static struct feature_driver port_feature_drvs[] = {
        {
                .id = PORT_FEATURE_ID_HEADER,
                .ops = &port_hdr_ops,
        },
        {
+               .id = PORT_FEATURE_ID_AFU,
+               .ops = &port_uafu_ops,
+       },
+       {
                .ops = NULL,
        }
 };

@@ -128,6 +191,64 @@ static long afu_ioctl_check_extension(struct feature_platform_data *pdata,
        return 0;
 }

+static long
+afu_ioctl_get_info(struct feature_platform_data *pdata, void __user *arg)
+{
+       struct fpga_port_info info;
+       struct fpga_afu *afu;
+       unsigned long minsz;
+
+       minsz = offsetofend(struct fpga_port_info, num_umsgs);
+
+       if (copy_from_user(&info, arg, minsz))
+               return -EFAULT;
+
+       if (info.argsz < minsz)
+               return -EINVAL;
+
+       mutex_lock(&pdata->lock);
+       afu = fpga_pdata_get_private(pdata);
+       info.flags = 0;
+       info.num_regions = afu->num_regions;
+       info.num_umsgs = afu->num_umsgs;
+       mutex_unlock(&pdata->lock);
+
+       if (copy_to_user(arg, &info, sizeof(info)))
+               return -EFAULT;
+
+       return 0;
+}
+
+static long
+afu_ioctl_get_region_info(struct feature_platform_data *pdata, void __user *arg)
+{
+       struct fpga_port_region_info rinfo;
+       struct fpga_afu_region region;
+       unsigned long minsz;
+       long ret;
+
+       minsz = offsetofend(struct fpga_port_region_info, offset);
+
+       if (copy_from_user(&rinfo, arg, minsz))
+               return -EFAULT;
+
+       if (rinfo.argsz < minsz || rinfo.padding)
+               return -EINVAL;
+
+       ret = afu_get_region_by_index(pdata, rinfo.index, &region);
+       if (ret)
+               return ret;
+
+       rinfo.flags = region.flags;
+       rinfo.size = region.size;
+       rinfo.offset = region.offset;
+
+       if (copy_to_user(arg, &rinfo, sizeof(rinfo)))
+               return -EFAULT;
+
+       return 0;
+}
+
 static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
 {
        struct platform_device *pdev = filp->private_data;
@@ -142,6 +263,10 @@ static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
                return FPGA_API_VERSION;
        case FPGA_CHECK_EXTENSION:

I commented elsewhere about the FPGA_* names.  These names such as
FPGA_CHECK_EXTENSION are specific to this DFL patchset, they weren't
written as addtions to the lower level FPGA stuff.

Yes, will fix this.

quoted

                return afu_ioctl_check_extension(pdata, arg);
+       case FPGA_PORT_GET_INFO:
+               return afu_ioctl_get_info(pdata, (void __user *)arg);
+       case FPGA_PORT_GET_REGION_INFO:
+               return afu_ioctl_get_region_info(pdata, (void __user *)arg);
        default:
                /*
                 * Let sub-feature's ioctl function to handle the cmd

@@ -162,27 +287,106 @@ static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
        return -EINVAL;
 }

+static int afu_mmap(struct file *filp, struct vm_area_struct *vma)
+{
+       struct fpga_afu_region region;
+       struct platform_device *pdev = filp->private_data;
+       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+       u64 size = vma->vm_end - vma->vm_start;
+       u64 offset;
+       int ret;
+
+       if (!(vma->vm_flags & VM_SHARED))
+               return -EINVAL;
+
+       offset = vma->vm_pgoff << PAGE_SHIFT;
+       ret = afu_get_region_by_offset(pdata, offset, size, &region);
+       if (ret)
+               return ret;
+
+       if (!(region.flags & FPGA_REGION_MMAP))

FPGA_REGION_*?  We already have FPGA regions which are something
different.  Please call this something else.

Yes, will replace it with DFL_FPGA_*.

quoted

+               return -EINVAL;
+
+       if ((vma->vm_flags & VM_READ) && !(region.flags & FPGA_REGION_READ))
+               return -EPERM;
+
+       if ((vma->vm_flags & VM_WRITE) && !(region.flags & FPGA_REGION_WRITE))
+               return -EPERM;
+
+       vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
+
+       return remap_pfn_range(vma, vma->vm_start,
+                       (region.phys + (offset - region.offset)) >> PAGE_SHIFT,
+                       size, vma->vm_page_prot);
+}
+
 static const struct file_operations afu_fops = {
        .owner = THIS_MODULE,
        .open = afu_open,
        .release = afu_release,
        .unlocked_ioctl = afu_ioctl,
+       .mmap = afu_mmap,
 };

+static int afu_dev_init(struct platform_device *pdev)
+{
+       struct fpga_afu *afu;
+       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+
+       afu = devm_kzalloc(&pdev->dev, sizeof(*afu), GFP_KERNEL);
+       if (!afu)
+               return -ENOMEM;
+
+       afu->pdata = pdata;
+
+       mutex_lock(&pdata->lock);
+       fpga_pdata_set_private(pdata, afu);
+       afu_region_init(pdata);
+       mutex_unlock(&pdata->lock);
+       return 0;
+}
+
+static int afu_dev_destroy(struct platform_device *pdev)
+{
+       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+       struct fpga_afu *afu;
+
+       mutex_lock(&pdata->lock);
+       afu = fpga_pdata_get_private(pdata);
+       afu_region_destroy(pdata);
+       fpga_pdata_set_private(pdata, NULL);
+       mutex_unlock(&pdata->lock);
+
+       devm_kfree(&pdev->dev, afu);
+
+       return 0;
+}
+
 static int afu_probe(struct platform_device *pdev)
 {
        int ret;

        dev_dbg(&pdev->dev, "%s\n", __func__);

+       ret = afu_dev_init(pdev);
+       if (ret)
+               goto exit;
+
        ret = fpga_dev_feature_init(pdev, port_feature_drvs);
        if (ret)
-               return ret;
+               goto dev_destroy;

        ret = fpga_register_dev_ops(pdev, &afu_fops, THIS_MODULE);
-       if (ret)
+       if (ret) {
                fpga_dev_feature_uinit(pdev);
+               goto dev_destroy;
+       }
+
+       return 0;

+dev_destroy:
+       afu_dev_destroy(pdev);
+exit:
        return ret;
 }

@@ -192,6 +396,7 @@ static int afu_remove(struct platform_device *pdev)

        fpga_dev_feature_uinit(pdev);
        fpga_unregister_dev_ops(pdev);
+       afu_dev_destroy(pdev);

        return 0;
 }

diff --git a/drivers/fpga/dfl-afu-region.c b/drivers/fpga/dfl-afu-region.c
new file mode 100644
index 0000000..e3a467b
--- /dev/null
+++ b/drivers/fpga/dfl-afu-region.c

@@ -0,0 +1,165 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Driver for FPGA Accelerated Function Unit (AFU) Region Management
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Wu Hao <hao.wu@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ */
+#include "dfl-afu.h"
+
+/**
+ * afu_region_init - init function for afu region support
+ * @pdata: afu platform device's pdata.
+ */
+void afu_region_init(struct feature_platform_data *pdata)

These afu regions are memory regions?  We also have 'FPGA regions'
which are partial reconfiguration areas.  It probably would be really
helpful to name things to make it more clear what these are and that
they are not what drivers/fpga already calls "FPGA regions".

Yes, these afu regions are memory regions for MMIO, will fix this too.

Thanks
Hao

quoted

+{
+       struct fpga_afu *afu = fpga_pdata_get_private(pdata);
+
+       INIT_LIST_HEAD(&afu->regions);
+}
+
+#define for_each_region(region, afu)   \
+       list_for_each_entry((region), &(afu)->regions, node)
+
+static struct fpga_afu_region *get_region_by_index(struct fpga_afu *afu,
+                                                  u32 region_index)
+{
+       struct fpga_afu_region *region;
+
+       for_each_region(region, afu)
+               if (region->index == region_index)
+                       return region;
+
+       return NULL;
+}
+
+/**
+ * afu_region_add - add a region to given feature dev.
+ *
+ * @region_index: region index.
+ * @region_size: region size.
+ * @phys: region's physical address of this region.
+ * @flags: region flags (access permission).
+ *
+ * Return: 0 on success, negative error code otherwise.
+ */
+int afu_region_add(struct feature_platform_data *pdata, u32 region_index,
+                  u64 region_size, u64 phys, u32 flags)
+{
+       struct fpga_afu_region *region;
+       struct fpga_afu *afu;
+       int ret = 0;
+
+       region = devm_kzalloc(&pdata->dev->dev, sizeof(*region), GFP_KERNEL);
+       if (!region)
+               return -ENOMEM;
+
+       region->index = region_index;
+       region->size = region_size;
+       region->phys = phys;
+       region->flags = flags;
+
+       mutex_lock(&pdata->lock);
+
+       afu = fpga_pdata_get_private(pdata);
+
+       /* check if @index already exists */
+       if (get_region_by_index(afu, region_index)) {
+               mutex_unlock(&pdata->lock);
+               ret = -EEXIST;
+               goto exit;
+       }
+
+       region_size = PAGE_ALIGN(region_size);
+       region->offset = afu->region_cur_offset;
+       list_add(&region->node, &afu->regions);
+
+       afu->region_cur_offset += region_size;
+       afu->num_regions++;
+       mutex_unlock(&pdata->lock);
+
+       return 0;
+
+exit:
+       devm_kfree(&pdata->dev->dev, region);
+       return ret;
+}
+
+/**
+ * afu_region_destroy - destroy all regions under given feature dev.
+ * @pdata: afu platform device's pdata.
+ */
+void afu_region_destroy(struct feature_platform_data *pdata)
+{
+       struct fpga_afu_region *tmp, *region;
+       struct fpga_afu *afu = fpga_pdata_get_private(pdata);
+
+       list_for_each_entry_safe(region, tmp, &afu->regions, node)
+               devm_kfree(&pdata->dev->dev, region);
+}
+
+/**
+ * afu_get_region_by_index - find an afu region by index.
+ * @pdata: afu platform device's pdata.
+ * @region_index: region index.
+ * @pregion: ptr to region for result.
+ *
+ * Return: 0 on success, negative error code otherwise.
+ */
+int afu_get_region_by_index(struct feature_platform_data *pdata,
+                           u32 region_index, struct fpga_afu_region *pregion)
+{
+       struct fpga_afu_region *region;
+       struct fpga_afu *afu;
+       int ret = 0;
+
+       mutex_lock(&pdata->lock);
+       afu = fpga_pdata_get_private(pdata);
+       region = get_region_by_index(afu, region_index);
+       if (!region) {
+               ret = -EINVAL;
+               goto exit;
+       }
+       *pregion = *region;
+exit:
+       mutex_unlock(&pdata->lock);
+       return ret;
+}
+
+/**
+ * afu_get_region_by_offset - find an afu region by offset and size
+ *
+ * @pdata: afu platform device's pdata.
+ * @offset: region offset from start of the device fd.
+ * @size: region size.
+ * @pregion: ptr to region for result.
+ *
+ * Find the region which fully contains the region described by input
+ * parameters (offset and size) from the feature dev's region link list.
+ *
+ * Return: 0 on success, negative error code otherwise.
+ */
+int afu_get_region_by_offset(struct feature_platform_data *pdata,
+                            u64 offset, u64 size,
+                            struct fpga_afu_region *pregion)
+{
+       struct fpga_afu_region *region;
+       struct fpga_afu *afu;
+       int ret = 0;
+
+       mutex_lock(&pdata->lock);
+       afu = fpga_pdata_get_private(pdata);
+       for_each_region(region, afu)
+               if (region->offset <= offset &&
+                   region->offset + region->size >= offset + size) {
+                       *pregion = *region;
+                       goto exit;
+               }
+       ret = -EINVAL;
+exit:
+       mutex_unlock(&pdata->lock);
+       return ret;
+}

diff --git a/drivers/fpga/dfl-afu.h b/drivers/fpga/dfl-afu.h
new file mode 100644
index 0000000..af690cc
--- /dev/null
+++ b/drivers/fpga/dfl-afu.h

@@ -0,0 +1,71 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Header file for FPGA Accelerated Function Unit (AFU) Driver
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *     Wu Hao <hao.wu@intel.com>
+ *     Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *     Joseph Grecco <joe.grecco@intel.com>
+ *     Enno Luebbers <enno.luebbers@intel.com>
+ *     Tim Whisonant <tim.whisonant@intel.com>
+ *     Ananda Ravuri <ananda.ravuri@intel.com>
+ *     Henry Mitchel <henry.mitchel@intel.com>
+ */
+
+#ifndef __DFL_AFU_H
+#define __DFL_AFU_H
+
+#include <linux/mm.h>
+
+#include "dfl.h"
+
+/**
+ * struct fpga_afu_region - afu region data structure
+ *
+ * @index: region index.
+ * @flags: region flags (access permission).
+ * @size: region size.
+ * @offset: region offset from start of the device fd.
+ * @phys: region's physical address.
+ * @node: node to add to afu feature dev's region list.
+ */
+struct fpga_afu_region {
+       u32 index;
+       u32 flags;
+       u64 size;
+       u64 offset;
+       u64 phys;
+       struct list_head node;
+};
+
+/**
+ * struct fpga_afu - afu device data structure
+ *
+ * @region_cur_offset: current region offset from start to the device fd.
+ * @num_regions: num of regions.
+ * @regions: the region link list of this afu feature device.
+ * @num_umsgs: num of umsgs.
+ * @pdata: afu platform device's pdata.
+ */
+struct fpga_afu {
+       u64 region_cur_offset;
+       int num_regions;
+       u8 num_umsgs;
+       struct list_head regions;
+
+       struct feature_platform_data *pdata;
+};
+
+void afu_region_init(struct feature_platform_data *pdata);
+int afu_region_add(struct feature_platform_data *pdata, u32 region_index,
+                  u64 region_size, u64 phys, u32 flags);
+void afu_region_destroy(struct feature_platform_data *pdata);
+int afu_get_region_by_index(struct feature_platform_data *pdata,
+                           u32 region_index, struct fpga_afu_region *pregion);
+int afu_get_region_by_offset(struct feature_platform_data *pdata,
+                            u64 offset, u64 size,
+                            struct fpga_afu_region *pregion);
+
+#endif

diff --git a/include/uapi/linux/fpga-dfl.h b/include/uapi/linux/fpga-dfl.h
index 727bec2..df41828 100644
--- a/include/uapi/linux/fpga-dfl.h
+++ b/include/uapi/linux/fpga-dfl.h

@@ -65,6 +65,53 @@

 #define FPGA_PORT_RESET                _IO(FPGA_MAGIC, PORT_BASE + 0)

+/**
+ * FPGA_PORT_GET_INFO - _IOR(FPGA_MAGIC, PORT_BASE + 1, struct fpga_port_info)
+ *
+ * Retrieve information about the fpga port.
+ * Driver fills the info in provided struct fpga_port_info.
+ * Return: 0 on success, -errno on failure.
+ */
+struct fpga_port_info {
+       /* Input */
+       __u32 argsz;            /* Structure length */
+       /* Output */
+       __u32 flags;            /* Zero for now */
+       __u32 num_regions;      /* The number of supported regions */
+       __u32 num_umsgs;        /* The number of allocated umsgs */
+};
+
+#define FPGA_PORT_GET_INFO     _IO(FPGA_MAGIC, PORT_BASE + 1)
+
+/**
+ * FPGA_PORT_GET_REGION_INFO - _IOWR(FPGA_MAGIC, PORT_BASE + 2,
+ *                                             struct fpga_port_region_info)
+ *
+ * Retrieve information about a device region.
+ * Caller provides struct fpga_port_region_info with index value set.
+ * Driver returns the region info in other fields.
+ * Return: 0 on success, -errno on failure.
+ */
+struct fpga_port_region_info {
+       /* input */
+       __u32 argsz;            /* Structure length */
+       /* Output */
+       __u32 flags;            /* Access permission */
+#define FPGA_REGION_READ       (1 << 0)        /* Region is readable */
+#define FPGA_REGION_WRITE      (1 << 1)        /* Region is writable */
+#define FPGA_REGION_MMAP       (1 << 2)        /* Can be mmaped to userspace */
+       /* Input */
+       __u32 index;            /* Region index */
+#define FPGA_PORT_INDEX_UAFU   0               /* User AFU */
+#define FPGA_PORT_INDEX_STP    1               /* Signal Tap */
+       __u32 padding;
+       /* Output */
+       __u64 size;             /* Region size (bytes) */
+       __u64 offset;           /* Region offset from start of device fd */
+};
+
+#define FPGA_PORT_GET_REGION_INFO      _IO(FPGA_MAGIC, PORT_BASE + 2)
+
 /* IOCTLs for FME file descriptor */

 /**
--

2.7.4

Thanks,
Alan

Re: [PATCH v4 23/24] fpga: dfl: afu: add user afu sub feature support

From: Alan Tull <atull@kernel.org>
Date: 2018-03-20 18:17:57

On Tue, Mar 20, 2018 at 2:10 AM, Wu Hao [off-list ref] wrote:

On Mon, Mar 19, 2018 at 03:10:28PM -0500, Alan Tull wrote:

quoted

On Tue, Feb 13, 2018 at 3:24 AM, Wu Hao [off-list ref] wrote:

Hi Hao,

quoted

From: Xiao Guangrong <redacted>

User Accelerated Function Unit sub feature exposes the MMIO region of

Is it 'user accelerated'?  I think it is the Accelerator interface.

Hi Alan,

This is only used to emphasize this is the interface to accelerator
exposed to user. But looks like this causes some confusions for user
actually from the description. I agree with you, that I will remove
this UAFU from this patchset.

quoted

the AFU. After valid green bitstream (GBS) is programmed and port is

Would it make sense to just use "partial bitstream" or "PR bitstream"
and "static bitstream" for this patchset?  I don't think that adding
this terminology makes things clearer.  In any case when someone else
uses this patchset, they may not be using this type of branding in
their terminology.

Sure, will update the commit message and also sysfs doc below.

Yes and dfl.txt and the rest of the patchset as well, please.

Alan

Re: [PATCH v4 16/24] fpga: dfl: add fpga manager platform driver for FME

From: Alan Tull <atull@kernel.org>
Date: 2018-03-20 20:33:18

On Tue, Feb 13, 2018 at 3:24 AM, Wu Hao [off-list ref] wrote:

Hi Hao,

Elsewhere we discussed moving #defines used only in this driver either
to this .c file or to a similarly named .h file.  A couple minor
things below.

quoted hunk

This patch adds fpga manager driver for FPGA Management Engine (FME). It
implements fpga_manager_ops for FPGA Partial Reconfiguration function.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Kang Luwei <redacted>
Signed-off-by: Xiao Guangrong <redacted>
Signed-off-by: Wu Hao <redacted>
---
v3: rename driver to dfl-fpga-fme-mgr
    implemented status callback for fpga manager
    rebased due to fpga api changes
v4: rename to dfl-fme-mgr, and fix SPDX license issue
    add pr_credit comments and improve dev_err message
    remove interface_id sysfs interface
    include dfl-fme-pr.h instead of dfl.h
---
 drivers/fpga/Kconfig       |   6 +
 drivers/fpga/Makefile      |   1 +
 drivers/fpga/dfl-fme-mgr.c | 290 +++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 297 insertions(+)
 create mode 100644 drivers/fpga/dfl-fme-mgr.c

diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
index 103d5e2..89f76e8 100644
--- a/drivers/fpga/Kconfig
+++ b/drivers/fpga/Kconfig

@@ -150,6 +150,12 @@ config FPGA_DFL_FME
          FPGA platform level management features. There shall be 1 FME
          per DFL based FPGA device.

+config FPGA_DFL_FME_MGR
+       tristate "FPGA DFL FME Manager Driver"
+       depends on FPGA_DFL_FME
+       help
+         Say Y to enable FPGA Manager driver for FPGA Management Engine.
+
 config FPGA_DFL_PCI
        tristate "FPGA Device Feature List (DFL) PCIe Device Driver"
        depends on PCI && FPGA_DFL

diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index 3c44fc9..f82814a 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile

@@ -31,6 +31,7 @@ obj-$(CONFIG_OF_FPGA_REGION)          += of-fpga-region.o
 # FPGA Device Feature List Support
 obj-$(CONFIG_FPGA_DFL)                 += dfl.o
 obj-$(CONFIG_FPGA_DFL_FME)             += dfl-fme.o
+obj-$(CONFIG_FPGA_DFL_FME_MGR)         += dfl-fme-mgr.o

 dfl-fme-objs := dfl-fme-main.o dfl-fme-pr.o

diff --git a/drivers/fpga/dfl-fme-mgr.c b/drivers/fpga/dfl-fme-mgr.c
new file mode 100644
index 0000000..2f92c29
--- /dev/null
+++ b/drivers/fpga/dfl-fme-mgr.c

@@ -0,0 +1,290 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * FPGA Manager Driver for FPGA Management Engine (FME)
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Christopher Rauer <christopher.rauer@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ */
+
+#include <linux/bitfield.h>
+#include <linux/module.h>
+#include <linux/iopoll.h>
+#include <linux/fpga/fpga-mgr.h>
+
+#include "dfl-fme-pr.h"
+
+#define PR_WAIT_TIMEOUT   8000000
+#define PR_HOST_STATUS_IDLE    0
+
+struct fme_mgr_priv {
+       void __iomem *ioaddr;
+       u64 pr_error;
+};
+
+static u64 pr_error_to_mgr_status(u64 err)
+{
+       u64 status = 0;
+
+       if (err & FME_PR_ERR_OPERATION_ERR)
+               status |= FPGA_MGR_STATUS_OPERATION_ERR;
+       if (err & FME_PR_ERR_CRC_ERR)
+               status |= FPGA_MGR_STATUS_CRC_ERR;
+       if (err & FME_PR_ERR_INCOMPATIBLE_BS)
+               status |= FPGA_MGR_STATUS_INCOMPATIBLE_IMAGE_ERR;
+       if (err & FME_PR_ERR_PROTOCOL_ERR)
+               status |= FPGA_MGR_STATUS_IP_PROTOCOL_ERR;
+       if (err & FME_PR_ERR_FIFO_OVERFLOW)
+               status |= FPGA_MGR_STATUS_FIFO_OVERFLOW_ERR;
+
+       return status;
+}
+
+static u64 fme_mgr_pr_error_handle(void __iomem *fme_pr)
+{
+       u64 pr_status, pr_error;
+
+       pr_status = readq(fme_pr + FME_PR_STS);
+       if (!(pr_status & FME_PR_STS_PR_STS))
+               return 0;
+
+       pr_error = readq(fme_pr + FME_PR_ERR);
+       writeq(pr_error, fme_pr + FME_PR_ERR);
+
+       return pr_error;
+}
+
+static int fme_mgr_write_init(struct fpga_manager *mgr,
+                             struct fpga_image_info *info,
+                             const char *buf, size_t count)
+{
+       struct device *dev = &mgr->dev;
+       struct fme_mgr_priv *priv = mgr->priv;
+       void __iomem *fme_pr = priv->ioaddr;
+       u64 pr_ctrl, pr_status;
+
+       if (!(info->flags & FPGA_MGR_PARTIAL_RECONFIG)) {
+               dev_err(dev, "only supports partial reconfiguration.\n");
+               return -EINVAL;
+       }
+
+       dev_dbg(dev, "resetting PR before initiated PR\n");
+
+       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
+       pr_ctrl |= FME_PR_CTRL_PR_RST;
+       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
+
+       if (readq_poll_timeout(fme_pr + FME_PR_CTRL, pr_ctrl,
+                              pr_ctrl & FME_PR_CTRL_PR_RSTACK, 1,
+                              PR_WAIT_TIMEOUT)) {
+               dev_err(dev, "PR Reset ACK timeout\n");
+               return -ETIMEDOUT;
+       }
+
+       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
+       pr_ctrl &= ~FME_PR_CTRL_PR_RST;
+       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
+
+       dev_dbg(dev,
+               "waiting for PR resource in HW to be initialized and ready\n");
+
+       if (readq_poll_timeout(fme_pr + FME_PR_STS, pr_status,
+                              (pr_status & FME_PR_STS_PR_STS) ==
+                              FME_PR_STS_PR_STS_IDLE, 1, PR_WAIT_TIMEOUT)) {
+               dev_err(dev, "PR Status timeout\n");
+               priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
+               return -ETIMEDOUT;
+       }
+
+       dev_dbg(dev, "check and clear previous PR error\n");
+       priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
+       if (priv->pr_error)
+               dev_dbg(dev, "previous PR error detected %llx\n",
+                       (unsigned long long)priv->pr_error);
+
+       dev_dbg(dev, "set PR port ID\n");
+
+       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
+       pr_ctrl &= ~FME_PR_CTRL_PR_RGN_ID;
+       pr_ctrl |= FIELD_PREP(FME_PR_CTRL_PR_RGN_ID, info->region_id);
+       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
+
+       return 0;
+}
+
+static int fme_mgr_write(struct fpga_manager *mgr,
+                        const char *buf, size_t count)
+{
+       struct device *dev = &mgr->dev;
+       struct fme_mgr_priv *priv = mgr->priv;
+       void __iomem *fme_pr = priv->ioaddr;
+       u64 pr_ctrl, pr_status, pr_data;
+       int delay = 0, pr_credit, i = 0;
+
+       dev_dbg(dev, "start request\n");
+
+       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
+       pr_ctrl |= FME_PR_CTRL_PR_START;
+       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
+
+       dev_dbg(dev, "pushing data from bitstream to HW\n");
+
+       /*
+        * driver can push data to PR hardware using PR_DATA register once HW
+        * has enough pr_credit (> 1), pr_credit reduces one for every 32bit
+        * pr data write to PR_DATA register. If pr_credit <= 1, driver needs
+        * to wait for enough pr_credit from hardware by polling.
+        */
+       pr_status = readq(fme_pr + FME_PR_STS);
+       pr_credit = FIELD_GET(FME_PR_STS_PR_CREDIT, pr_status);
+
+       while (count > 0) {
+               while (pr_credit <= 1) {
+                       if (delay++ > PR_WAIT_TIMEOUT) {
+                               dev_err(dev, "PR_CREDIT timeout\n");
+                               return -ETIMEDOUT;
+                       }
+                       udelay(1);
+
+                       pr_status = readq(fme_pr + FME_PR_STS);
+                       pr_credit = FIELD_GET(FME_PR_STS_PR_CREDIT, pr_status);
+               }
+
+               if (count >= 4) {
+                       pr_data = 0;
+                       pr_data |= FIELD_PREP(FME_PR_DATA_PR_DATA_RAW,
+                                             *(((u32 *)buf) + i));
+                       writeq(pr_data, fme_pr + FME_PR_DATA);
+                       count -= 4;
+                       pr_credit--;
+                       i++;
+               } else {
+                       WARN_ON(1);
+                       return -EINVAL;
+               }
+       }
+
+       return 0;
+}
+
+static int fme_mgr_write_complete(struct fpga_manager *mgr,
+                                 struct fpga_image_info *info)
+{
+       struct device *dev = &mgr->dev;
+       struct fme_mgr_priv *priv = mgr->priv;
+       void __iomem *fme_pr = priv->ioaddr;
+       u64 pr_ctrl;
+
+       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
+       pr_ctrl |= FME_PR_CTRL_PR_COMPLETE;
+       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
+
+       dev_dbg(dev, "green bitstream push complete\n");
+       dev_dbg(dev, "waiting for HW to release PR resource\n");
+
+       if (readq_poll_timeout(fme_pr + FME_PR_CTRL, pr_ctrl,
+                              !(pr_ctrl & FME_PR_CTRL_PR_START), 1,
+                              PR_WAIT_TIMEOUT)) {
+               dev_err(dev, "PR Completion ACK timeout.\n");
+               return -ETIMEDOUT;
+       }
+
+       dev_dbg(dev, "PR operation complete, checking status\n");
+       priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
+       if (priv->pr_error) {
+               dev_dbg(dev, "PR error detected %llx\n",
+                       (unsigned long long)priv->pr_error);
+               return -EIO;
+       }
+
+       dev_dbg(dev, "PR done successfully\n");
+
+       return 0;
+}
+
+static enum fpga_mgr_states fme_mgr_state(struct fpga_manager *mgr)
+{
+       return FPGA_MGR_STATE_UNKNOWN;
+}
+
+static u64 fme_mgr_status(struct fpga_manager *mgr)
+{
+       struct fme_mgr_priv *priv = mgr->priv;
+
+       return pr_error_to_mgr_status(priv->pr_error);
+}
+
+static const struct fpga_manager_ops fme_mgr_ops = {
+       .write_init = fme_mgr_write_init,
+       .write = fme_mgr_write,
+       .write_complete = fme_mgr_write_complete,
+       .state = fme_mgr_state,
+       .status = fme_mgr_status,
+};
+
+static int fme_mgr_probe(struct platform_device *pdev)
+{
+       struct device *dev = &pdev->dev;
+       struct fme_mgr_priv *priv;
+       struct fpga_manager *mgr;
+       struct resource *res;
+       int ret;
+
+       priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
+       if (!priv)
+               return -ENOMEM;
+
+       res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+       priv->ioaddr = devm_ioremap(dev, res->start, resource_size(res));

How about using devm_ioremap_resourc(dev, res) here instead?

+       if (IS_ERR(priv->ioaddr))
+               return PTR_ERR(priv->ioaddr);
+
+       mgr = devm_kzalloc(dev, sizeof(*mgr), GFP_KERNEL);
+       if (!mgr)
+               return -ENOMEM;
+
+       mgr->name = "DFL FPGA Manager";
+       mgr->mops = &fme_mgr_ops;
+       mgr->priv = priv;
+       mgr->parent = dev;
+       platform_set_drvdata(pdev, mgr);
+
+       ret = fpga_mgr_register(mgr);
+       if (ret)
+               dev_err(dev, "unable to register FPGA manager\n");
+
+       return ret;

You can probably just do "return fpga_mgr_register(mgr);" here.

Thanks,
Alan

+}
+
+static int fme_mgr_remove(struct platform_device *pdev)
+{
+       struct fpga_manager *mgr = platform_get_drvdata(pdev);
+
+       fpga_mgr_unregister(mgr);
+
+       return 0;
+}
+
+static struct platform_driver fme_mgr_driver = {
+       .driver = {
+               .name    = FPGA_DFL_FME_MGR,
+       },
+       .probe   = fme_mgr_probe,
+       .remove  = fme_mgr_remove,
+};
+
+module_platform_driver(fme_mgr_driver);
+
+MODULE_DESCRIPTION("FPGA Manager for DFL FPGA Management Engine");
+MODULE_AUTHOR("Intel Corporation");
+MODULE_LICENSE("GPL v2");
+MODULE_ALIAS("platform:dfl-fme-mgr");
--
2.7.4

Re: [PATCH v4 16/24] fpga: dfl: add fpga manager platform driver for FME

From: Wu Hao <hidden>
Date: 2018-03-21 03:00:04

On Tue, Mar 20, 2018 at 03:32:34PM -0500, Alan Tull wrote:

On Tue, Feb 13, 2018 at 3:24 AM, Wu Hao [off-list ref] wrote:

Hi Hao,

Elsewhere we discussed moving #defines used only in this driver either
to this .c file or to a similarly named .h file.  A couple minor
things below.

Hi Alan,

Yes, I will move those #defines into a similarly named .h file.

quoted

This patch adds fpga manager driver for FPGA Management Engine (FME). It
implements fpga_manager_ops for FPGA Partial Reconfiguration function.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Kang Luwei <redacted>
Signed-off-by: Xiao Guangrong <redacted>
Signed-off-by: Wu Hao <redacted>
---
v3: rename driver to dfl-fpga-fme-mgr
    implemented status callback for fpga manager
    rebased due to fpga api changes
v4: rename to dfl-fme-mgr, and fix SPDX license issue
    add pr_credit comments and improve dev_err message
    remove interface_id sysfs interface
    include dfl-fme-pr.h instead of dfl.h
---
 drivers/fpga/Kconfig       |   6 +
 drivers/fpga/Makefile      |   1 +
 drivers/fpga/dfl-fme-mgr.c | 290 +++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 297 insertions(+)
 create mode 100644 drivers/fpga/dfl-fme-mgr.c

diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
index 103d5e2..89f76e8 100644
--- a/drivers/fpga/Kconfig
+++ b/drivers/fpga/Kconfig

@@ -150,6 +150,12 @@ config FPGA_DFL_FME
          FPGA platform level management features. There shall be 1 FME
          per DFL based FPGA device.

+config FPGA_DFL_FME_MGR
+       tristate "FPGA DFL FME Manager Driver"
+       depends on FPGA_DFL_FME
+       help
+         Say Y to enable FPGA Manager driver for FPGA Management Engine.
+
 config FPGA_DFL_PCI
        tristate "FPGA Device Feature List (DFL) PCIe Device Driver"
        depends on PCI && FPGA_DFL

diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index 3c44fc9..f82814a 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile

@@ -31,6 +31,7 @@ obj-$(CONFIG_OF_FPGA_REGION)          += of-fpga-region.o
 # FPGA Device Feature List Support
 obj-$(CONFIG_FPGA_DFL)                 += dfl.o
 obj-$(CONFIG_FPGA_DFL_FME)             += dfl-fme.o
+obj-$(CONFIG_FPGA_DFL_FME_MGR)         += dfl-fme-mgr.o

 dfl-fme-objs := dfl-fme-main.o dfl-fme-pr.o

diff --git a/drivers/fpga/dfl-fme-mgr.c b/drivers/fpga/dfl-fme-mgr.c
new file mode 100644
index 0000000..2f92c29
--- /dev/null
+++ b/drivers/fpga/dfl-fme-mgr.c

@@ -0,0 +1,290 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * FPGA Manager Driver for FPGA Management Engine (FME)
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Christopher Rauer <christopher.rauer@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ */
+
+#include <linux/bitfield.h>
+#include <linux/module.h>
+#include <linux/iopoll.h>
+#include <linux/fpga/fpga-mgr.h>
+
+#include "dfl-fme-pr.h"
+
+#define PR_WAIT_TIMEOUT   8000000
+#define PR_HOST_STATUS_IDLE    0
+
+struct fme_mgr_priv {
+       void __iomem *ioaddr;
+       u64 pr_error;
+};
+
+static u64 pr_error_to_mgr_status(u64 err)
+{
+       u64 status = 0;
+
+       if (err & FME_PR_ERR_OPERATION_ERR)
+               status |= FPGA_MGR_STATUS_OPERATION_ERR;
+       if (err & FME_PR_ERR_CRC_ERR)
+               status |= FPGA_MGR_STATUS_CRC_ERR;
+       if (err & FME_PR_ERR_INCOMPATIBLE_BS)
+               status |= FPGA_MGR_STATUS_INCOMPATIBLE_IMAGE_ERR;
+       if (err & FME_PR_ERR_PROTOCOL_ERR)
+               status |= FPGA_MGR_STATUS_IP_PROTOCOL_ERR;
+       if (err & FME_PR_ERR_FIFO_OVERFLOW)
+               status |= FPGA_MGR_STATUS_FIFO_OVERFLOW_ERR;
+
+       return status;
+}
+
+static u64 fme_mgr_pr_error_handle(void __iomem *fme_pr)
+{
+       u64 pr_status, pr_error;
+
+       pr_status = readq(fme_pr + FME_PR_STS);
+       if (!(pr_status & FME_PR_STS_PR_STS))
+               return 0;
+
+       pr_error = readq(fme_pr + FME_PR_ERR);
+       writeq(pr_error, fme_pr + FME_PR_ERR);
+
+       return pr_error;
+}
+
+static int fme_mgr_write_init(struct fpga_manager *mgr,
+                             struct fpga_image_info *info,
+                             const char *buf, size_t count)
+{
+       struct device *dev = &mgr->dev;
+       struct fme_mgr_priv *priv = mgr->priv;
+       void __iomem *fme_pr = priv->ioaddr;
+       u64 pr_ctrl, pr_status;
+
+       if (!(info->flags & FPGA_MGR_PARTIAL_RECONFIG)) {
+               dev_err(dev, "only supports partial reconfiguration.\n");
+               return -EINVAL;
+       }
+
+       dev_dbg(dev, "resetting PR before initiated PR\n");
+
+       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
+       pr_ctrl |= FME_PR_CTRL_PR_RST;
+       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
+
+       if (readq_poll_timeout(fme_pr + FME_PR_CTRL, pr_ctrl,
+                              pr_ctrl & FME_PR_CTRL_PR_RSTACK, 1,
+                              PR_WAIT_TIMEOUT)) {
+               dev_err(dev, "PR Reset ACK timeout\n");
+               return -ETIMEDOUT;
+       }
+
+       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
+       pr_ctrl &= ~FME_PR_CTRL_PR_RST;
+       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
+
+       dev_dbg(dev,
+               "waiting for PR resource in HW to be initialized and ready\n");
+
+       if (readq_poll_timeout(fme_pr + FME_PR_STS, pr_status,
+                              (pr_status & FME_PR_STS_PR_STS) ==
+                              FME_PR_STS_PR_STS_IDLE, 1, PR_WAIT_TIMEOUT)) {
+               dev_err(dev, "PR Status timeout\n");
+               priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
+               return -ETIMEDOUT;
+       }
+
+       dev_dbg(dev, "check and clear previous PR error\n");
+       priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
+       if (priv->pr_error)
+               dev_dbg(dev, "previous PR error detected %llx\n",
+                       (unsigned long long)priv->pr_error);
+
+       dev_dbg(dev, "set PR port ID\n");
+
+       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
+       pr_ctrl &= ~FME_PR_CTRL_PR_RGN_ID;
+       pr_ctrl |= FIELD_PREP(FME_PR_CTRL_PR_RGN_ID, info->region_id);
+       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
+
+       return 0;
+}
+
+static int fme_mgr_write(struct fpga_manager *mgr,
+                        const char *buf, size_t count)
+{
+       struct device *dev = &mgr->dev;
+       struct fme_mgr_priv *priv = mgr->priv;
+       void __iomem *fme_pr = priv->ioaddr;
+       u64 pr_ctrl, pr_status, pr_data;
+       int delay = 0, pr_credit, i = 0;
+
+       dev_dbg(dev, "start request\n");
+
+       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
+       pr_ctrl |= FME_PR_CTRL_PR_START;
+       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
+
+       dev_dbg(dev, "pushing data from bitstream to HW\n");
+
+       /*
+        * driver can push data to PR hardware using PR_DATA register once HW
+        * has enough pr_credit (> 1), pr_credit reduces one for every 32bit
+        * pr data write to PR_DATA register. If pr_credit <= 1, driver needs
+        * to wait for enough pr_credit from hardware by polling.
+        */
+       pr_status = readq(fme_pr + FME_PR_STS);
+       pr_credit = FIELD_GET(FME_PR_STS_PR_CREDIT, pr_status);
+
+       while (count > 0) {
+               while (pr_credit <= 1) {
+                       if (delay++ > PR_WAIT_TIMEOUT) {
+                               dev_err(dev, "PR_CREDIT timeout\n");
+                               return -ETIMEDOUT;
+                       }
+                       udelay(1);
+
+                       pr_status = readq(fme_pr + FME_PR_STS);
+                       pr_credit = FIELD_GET(FME_PR_STS_PR_CREDIT, pr_status);
+               }
+
+               if (count >= 4) {
+                       pr_data = 0;
+                       pr_data |= FIELD_PREP(FME_PR_DATA_PR_DATA_RAW,
+                                             *(((u32 *)buf) + i));
+                       writeq(pr_data, fme_pr + FME_PR_DATA);
+                       count -= 4;
+                       pr_credit--;
+                       i++;
+               } else {
+                       WARN_ON(1);
+                       return -EINVAL;
+               }
+       }
+
+       return 0;
+}
+
+static int fme_mgr_write_complete(struct fpga_manager *mgr,
+                                 struct fpga_image_info *info)
+{
+       struct device *dev = &mgr->dev;
+       struct fme_mgr_priv *priv = mgr->priv;
+       void __iomem *fme_pr = priv->ioaddr;
+       u64 pr_ctrl;
+
+       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
+       pr_ctrl |= FME_PR_CTRL_PR_COMPLETE;
+       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
+
+       dev_dbg(dev, "green bitstream push complete\n");
+       dev_dbg(dev, "waiting for HW to release PR resource\n");
+
+       if (readq_poll_timeout(fme_pr + FME_PR_CTRL, pr_ctrl,
+                              !(pr_ctrl & FME_PR_CTRL_PR_START), 1,
+                              PR_WAIT_TIMEOUT)) {
+               dev_err(dev, "PR Completion ACK timeout.\n");
+               return -ETIMEDOUT;
+       }
+
+       dev_dbg(dev, "PR operation complete, checking status\n");
+       priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
+       if (priv->pr_error) {
+               dev_dbg(dev, "PR error detected %llx\n",
+                       (unsigned long long)priv->pr_error);
+               return -EIO;
+       }
+
+       dev_dbg(dev, "PR done successfully\n");
+
+       return 0;
+}
+
+static enum fpga_mgr_states fme_mgr_state(struct fpga_manager *mgr)
+{
+       return FPGA_MGR_STATE_UNKNOWN;
+}
+
+static u64 fme_mgr_status(struct fpga_manager *mgr)
+{
+       struct fme_mgr_priv *priv = mgr->priv;
+
+       return pr_error_to_mgr_status(priv->pr_error);
+}
+
+static const struct fpga_manager_ops fme_mgr_ops = {
+       .write_init = fme_mgr_write_init,
+       .write = fme_mgr_write,
+       .write_complete = fme_mgr_write_complete,
+       .state = fme_mgr_state,
+       .status = fme_mgr_status,
+};
+
+static int fme_mgr_probe(struct platform_device *pdev)
+{
+       struct device *dev = &pdev->dev;
+       struct fme_mgr_priv *priv;
+       struct fpga_manager *mgr;
+       struct resource *res;
+       int ret;
+
+       priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
+       if (!priv)
+               return -ENOMEM;
+
+       res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+       priv->ioaddr = devm_ioremap(dev, res->start, resource_size(res));

How about using devm_ioremap_resourc(dev, res) here instead?

Actually the register region has already been mapped in lower level driver
(e.g pci) so I think we don't have to map the second time here. I plan to
add some code to pass the ioaddr via the platform data, and check if valid
ioaddr from the platform data firstly in this probe function. If no pdata
or no valid ioaddr, then go with devm_ioremap_resource. :)

quoted

+       if (IS_ERR(priv->ioaddr))
+               return PTR_ERR(priv->ioaddr);
+
+       mgr = devm_kzalloc(dev, sizeof(*mgr), GFP_KERNEL);
+       if (!mgr)
+               return -ENOMEM;
+
+       mgr->name = "DFL FPGA Manager";
+       mgr->mops = &fme_mgr_ops;
+       mgr->priv = priv;
+       mgr->parent = dev;
+       platform_set_drvdata(pdev, mgr);
+
+       ret = fpga_mgr_register(mgr);
+       if (ret)
+               dev_err(dev, "unable to register FPGA manager\n");
+
+       return ret;

You can probably just do "return fpga_mgr_register(mgr);" here.

Yes, it looks better, I will fix it. Thanks a lot for the review.

Hao

Thanks,
Alan

quoted

+}
+
+static int fme_mgr_remove(struct platform_device *pdev)
+{
+       struct fpga_manager *mgr = platform_get_drvdata(pdev);
+
+       fpga_mgr_unregister(mgr);
+
+       return 0;
+}
+
+static struct platform_driver fme_mgr_driver = {
+       .driver = {
+               .name    = FPGA_DFL_FME_MGR,
+       },
+       .probe   = fme_mgr_probe,
+       .remove  = fme_mgr_remove,
+};
+
+module_platform_driver(fme_mgr_driver);
+
+MODULE_DESCRIPTION("FPGA Manager for DFL FPGA Management Engine");
+MODULE_AUTHOR("Intel Corporation");
+MODULE_LICENSE("GPL v2");
+MODULE_ALIAS("platform:dfl-fme-mgr");
--
2.7.4

Re: [PATCH v4 23/24] fpga: dfl: afu: add user afu sub feature support

From: Wu Hao <hidden>
Date: 2018-03-21 03:10:39

On Tue, Mar 20, 2018 at 01:17:14PM -0500, Alan Tull wrote:

On Tue, Mar 20, 2018 at 2:10 AM, Wu Hao [off-list ref] wrote:

quoted

On Mon, Mar 19, 2018 at 03:10:28PM -0500, Alan Tull wrote:

quoted

On Tue, Feb 13, 2018 at 3:24 AM, Wu Hao [off-list ref] wrote:

Hi Hao,

quoted

From: Xiao Guangrong <redacted>

User Accelerated Function Unit sub feature exposes the MMIO region of

Is it 'user accelerated'?  I think it is the Accelerator interface.

Hi Alan,

This is only used to emphasize this is the interface to accelerator
exposed to user. But looks like this causes some confusions for user
actually from the description. I agree with you, that I will remove
this UAFU from this patchset.

quoted

the AFU. After valid green bitstream (GBS) is programmed and port is

Would it make sense to just use "partial bitstream" or "PR bitstream"
and "static bitstream" for this patchset?  I don't think that adding
this terminology makes things clearer.  In any case when someone else
uses this patchset, they may not be using this type of branding in
their terminology.

Sure, will update the commit message and also sysfs doc below.

Yes and dfl.txt and the rest of the patchset as well, please.

Sure, I understand that it may have different PR hardwares under this
framework, each PR hardware may have a different terminology for its own
bitstream. We should use common terminology in the common doc and code
to avoid confusion. Thanks for the reminder. I will fix this.

Thanks
Hao

Alan

Re: [PATCH v4 16/24] fpga: dfl: add fpga manager platform driver for FME

From: Moritz Fischer <mdf@kernel.org>
Date: 2018-03-21 16:56:00

On Wed, Mar 21, 2018 at 10:50:01AM +0800, Wu Hao wrote:

On Tue, Mar 20, 2018 at 03:32:34PM -0500, Alan Tull wrote:

quoted

On Tue, Feb 13, 2018 at 3:24 AM, Wu Hao [off-list ref] wrote:

Hi Hao,

Elsewhere we discussed moving #defines used only in this driver either
to this .c file or to a similarly named .h file.  A couple minor
things below.

Hi Alan,

Yes, I will move those #defines into a similarly named .h file.

quoted

This patch adds fpga manager driver for FPGA Management Engine (FME). It
implements fpga_manager_ops for FPGA Partial Reconfiguration function.

Signed-off-by: Tim Whisonant <redacted>
Signed-off-by: Enno Luebbers <redacted>
Signed-off-by: Shiva Rao <redacted>
Signed-off-by: Christopher Rauer <redacted>
Signed-off-by: Kang Luwei <redacted>
Signed-off-by: Xiao Guangrong <redacted>
Signed-off-by: Wu Hao <redacted>
---
v3: rename driver to dfl-fpga-fme-mgr
    implemented status callback for fpga manager
    rebased due to fpga api changes
v4: rename to dfl-fme-mgr, and fix SPDX license issue
    add pr_credit comments and improve dev_err message
    remove interface_id sysfs interface
    include dfl-fme-pr.h instead of dfl.h
---
 drivers/fpga/Kconfig       |   6 +
 drivers/fpga/Makefile      |   1 +
 drivers/fpga/dfl-fme-mgr.c | 290 +++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 297 insertions(+)
 create mode 100644 drivers/fpga/dfl-fme-mgr.c

diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
index 103d5e2..89f76e8 100644
--- a/drivers/fpga/Kconfig
+++ b/drivers/fpga/Kconfig

@@ -150,6 +150,12 @@ config FPGA_DFL_FME
          FPGA platform level management features. There shall be 1 FME
          per DFL based FPGA device.

+config FPGA_DFL_FME_MGR
+       tristate "FPGA DFL FME Manager Driver"
+       depends on FPGA_DFL_FME
+       help
+         Say Y to enable FPGA Manager driver for FPGA Management Engine.
+
 config FPGA_DFL_PCI
        tristate "FPGA Device Feature List (DFL) PCIe Device Driver"
        depends on PCI && FPGA_DFL

diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index 3c44fc9..f82814a 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile

@@ -31,6 +31,7 @@ obj-$(CONFIG_OF_FPGA_REGION)          += of-fpga-region.o
 # FPGA Device Feature List Support
 obj-$(CONFIG_FPGA_DFL)                 += dfl.o
 obj-$(CONFIG_FPGA_DFL_FME)             += dfl-fme.o
+obj-$(CONFIG_FPGA_DFL_FME_MGR)         += dfl-fme-mgr.o

 dfl-fme-objs := dfl-fme-main.o dfl-fme-pr.o

diff --git a/drivers/fpga/dfl-fme-mgr.c b/drivers/fpga/dfl-fme-mgr.c
new file mode 100644
index 0000000..2f92c29
--- /dev/null
+++ b/drivers/fpga/dfl-fme-mgr.c

@@ -0,0 +1,290 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * FPGA Manager Driver for FPGA Management Engine (FME)
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Christopher Rauer <christopher.rauer@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ */
+
+#include <linux/bitfield.h>
+#include <linux/module.h>
+#include <linux/iopoll.h>
+#include <linux/fpga/fpga-mgr.h>
+
+#include "dfl-fme-pr.h"
+
+#define PR_WAIT_TIMEOUT   8000000
+#define PR_HOST_STATUS_IDLE    0
+
+struct fme_mgr_priv {
+       void __iomem *ioaddr;
+       u64 pr_error;
+};
+
+static u64 pr_error_to_mgr_status(u64 err)
+{
+       u64 status = 0;
+
+       if (err & FME_PR_ERR_OPERATION_ERR)
+               status |= FPGA_MGR_STATUS_OPERATION_ERR;
+       if (err & FME_PR_ERR_CRC_ERR)
+               status |= FPGA_MGR_STATUS_CRC_ERR;
+       if (err & FME_PR_ERR_INCOMPATIBLE_BS)
+               status |= FPGA_MGR_STATUS_INCOMPATIBLE_IMAGE_ERR;
+       if (err & FME_PR_ERR_PROTOCOL_ERR)
+               status |= FPGA_MGR_STATUS_IP_PROTOCOL_ERR;
+       if (err & FME_PR_ERR_FIFO_OVERFLOW)
+               status |= FPGA_MGR_STATUS_FIFO_OVERFLOW_ERR;
+
+       return status;
+}
+
+static u64 fme_mgr_pr_error_handle(void __iomem *fme_pr)
+{
+       u64 pr_status, pr_error;
+
+       pr_status = readq(fme_pr + FME_PR_STS);
+       if (!(pr_status & FME_PR_STS_PR_STS))
+               return 0;
+
+       pr_error = readq(fme_pr + FME_PR_ERR);
+       writeq(pr_error, fme_pr + FME_PR_ERR);
+
+       return pr_error;
+}
+
+static int fme_mgr_write_init(struct fpga_manager *mgr,
+                             struct fpga_image_info *info,
+                             const char *buf, size_t count)
+{
+       struct device *dev = &mgr->dev;
+       struct fme_mgr_priv *priv = mgr->priv;
+       void __iomem *fme_pr = priv->ioaddr;
+       u64 pr_ctrl, pr_status;
+
+       if (!(info->flags & FPGA_MGR_PARTIAL_RECONFIG)) {
+               dev_err(dev, "only supports partial reconfiguration.\n");
+               return -EINVAL;
+       }
+
+       dev_dbg(dev, "resetting PR before initiated PR\n");
+
+       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
+       pr_ctrl |= FME_PR_CTRL_PR_RST;
+       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
+
+       if (readq_poll_timeout(fme_pr + FME_PR_CTRL, pr_ctrl,
+                              pr_ctrl & FME_PR_CTRL_PR_RSTACK, 1,
+                              PR_WAIT_TIMEOUT)) {
+               dev_err(dev, "PR Reset ACK timeout\n");
+               return -ETIMEDOUT;
+       }
+
+       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
+       pr_ctrl &= ~FME_PR_CTRL_PR_RST;
+       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
+
+       dev_dbg(dev,
+               "waiting for PR resource in HW to be initialized and ready\n");
+
+       if (readq_poll_timeout(fme_pr + FME_PR_STS, pr_status,
+                              (pr_status & FME_PR_STS_PR_STS) ==
+                              FME_PR_STS_PR_STS_IDLE, 1, PR_WAIT_TIMEOUT)) {
+               dev_err(dev, "PR Status timeout\n");
+               priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
+               return -ETIMEDOUT;
+       }
+
+       dev_dbg(dev, "check and clear previous PR error\n");
+       priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
+       if (priv->pr_error)
+               dev_dbg(dev, "previous PR error detected %llx\n",
+                       (unsigned long long)priv->pr_error);
+
+       dev_dbg(dev, "set PR port ID\n");
+
+       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
+       pr_ctrl &= ~FME_PR_CTRL_PR_RGN_ID;
+       pr_ctrl |= FIELD_PREP(FME_PR_CTRL_PR_RGN_ID, info->region_id);
+       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
+
+       return 0;
+}
+
+static int fme_mgr_write(struct fpga_manager *mgr,
+                        const char *buf, size_t count)
+{
+       struct device *dev = &mgr->dev;
+       struct fme_mgr_priv *priv = mgr->priv;
+       void __iomem *fme_pr = priv->ioaddr;
+       u64 pr_ctrl, pr_status, pr_data;
+       int delay = 0, pr_credit, i = 0;
+
+       dev_dbg(dev, "start request\n");
+
+       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
+       pr_ctrl |= FME_PR_CTRL_PR_START;
+       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
+
+       dev_dbg(dev, "pushing data from bitstream to HW\n");
+
+       /*
+        * driver can push data to PR hardware using PR_DATA register once HW
+        * has enough pr_credit (> 1), pr_credit reduces one for every 32bit
+        * pr data write to PR_DATA register. If pr_credit <= 1, driver needs
+        * to wait for enough pr_credit from hardware by polling.
+        */
+       pr_status = readq(fme_pr + FME_PR_STS);
+       pr_credit = FIELD_GET(FME_PR_STS_PR_CREDIT, pr_status);
+
+       while (count > 0) {
+               while (pr_credit <= 1) {
+                       if (delay++ > PR_WAIT_TIMEOUT) {
+                               dev_err(dev, "PR_CREDIT timeout\n");
+                               return -ETIMEDOUT;
+                       }
+                       udelay(1);
+
+                       pr_status = readq(fme_pr + FME_PR_STS);
+                       pr_credit = FIELD_GET(FME_PR_STS_PR_CREDIT, pr_status);
+               }
+
+               if (count >= 4) {
+                       pr_data = 0;
+                       pr_data |= FIELD_PREP(FME_PR_DATA_PR_DATA_RAW,
+                                             *(((u32 *)buf) + i));
+                       writeq(pr_data, fme_pr + FME_PR_DATA);
+                       count -= 4;
+                       pr_credit--;
+                       i++;
+               } else {
+                       WARN_ON(1);
+                       return -EINVAL;
+               }
+       }
+
+       return 0;
+}
+
+static int fme_mgr_write_complete(struct fpga_manager *mgr,
+                                 struct fpga_image_info *info)
+{
+       struct device *dev = &mgr->dev;
+       struct fme_mgr_priv *priv = mgr->priv;
+       void __iomem *fme_pr = priv->ioaddr;
+       u64 pr_ctrl;
+
+       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
+       pr_ctrl |= FME_PR_CTRL_PR_COMPLETE;
+       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
+
+       dev_dbg(dev, "green bitstream push complete\n");
+       dev_dbg(dev, "waiting for HW to release PR resource\n");
+
+       if (readq_poll_timeout(fme_pr + FME_PR_CTRL, pr_ctrl,
+                              !(pr_ctrl & FME_PR_CTRL_PR_START), 1,
+                              PR_WAIT_TIMEOUT)) {
+               dev_err(dev, "PR Completion ACK timeout.\n");
+               return -ETIMEDOUT;
+       }
+
+       dev_dbg(dev, "PR operation complete, checking status\n");
+       priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
+       if (priv->pr_error) {
+               dev_dbg(dev, "PR error detected %llx\n",
+                       (unsigned long long)priv->pr_error);
+               return -EIO;
+       }
+
+       dev_dbg(dev, "PR done successfully\n");
+
+       return 0;
+}
+
+static enum fpga_mgr_states fme_mgr_state(struct fpga_manager *mgr)
+{
+       return FPGA_MGR_STATE_UNKNOWN;
+}
+
+static u64 fme_mgr_status(struct fpga_manager *mgr)
+{
+       struct fme_mgr_priv *priv = mgr->priv;
+
+       return pr_error_to_mgr_status(priv->pr_error);
+}
+
+static const struct fpga_manager_ops fme_mgr_ops = {
+       .write_init = fme_mgr_write_init,
+       .write = fme_mgr_write,
+       .write_complete = fme_mgr_write_complete,
+       .state = fme_mgr_state,
+       .status = fme_mgr_status,
+};
+
+static int fme_mgr_probe(struct platform_device *pdev)
+{
+       struct device *dev = &pdev->dev;
+       struct fme_mgr_priv *priv;
+       struct fpga_manager *mgr;
+       struct resource *res;
+       int ret;
+
+       priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
+       if (!priv)
+               return -ENOMEM;
+
+       res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+       priv->ioaddr = devm_ioremap(dev, res->start, resource_size(res));

How about using devm_ioremap_resourc(dev, res) here instead?

Actually the register region has already been mapped in lower level driver
(e.g pci) so I think we don't have to map the second time here. I plan to
add some code to pass the ioaddr via the platform data, and check if valid
ioaddr from the platform data firstly in this probe function. If no pdata
or no valid ioaddr, then go with devm_ioremap_resource. :)

If you end up sharing register spaces between drivers is regmap / syscon
maybe a good idea?

quoted

+       if (IS_ERR(priv->ioaddr))
+               return PTR_ERR(priv->ioaddr);
+
+       mgr = devm_kzalloc(dev, sizeof(*mgr), GFP_KERNEL);
+       if (!mgr)
+               return -ENOMEM;
+
+       mgr->name = "DFL FPGA Manager";
+       mgr->mops = &fme_mgr_ops;
+       mgr->priv = priv;
+       mgr->parent = dev;
+       platform_set_drvdata(pdev, mgr);
+
+       ret = fpga_mgr_register(mgr);
+       if (ret)
+               dev_err(dev, "unable to register FPGA manager\n");
+
+       return ret;

You can probably just do "return fpga_mgr_register(mgr);" here.

Yes, it looks better, I will fix it. Thanks a lot for the review.

Hao

quoted

Thanks,
Alan

quoted

+}
+
+static int fme_mgr_remove(struct platform_device *pdev)
+{
+       struct fpga_manager *mgr = platform_get_drvdata(pdev);
+
+       fpga_mgr_unregister(mgr);
+
+       return 0;
+}
+
+static struct platform_driver fme_mgr_driver = {
+       .driver = {
+               .name    = FPGA_DFL_FME_MGR,
+       },
+       .probe   = fme_mgr_probe,
+       .remove  = fme_mgr_remove,
+};
+
+module_platform_driver(fme_mgr_driver);
+
+MODULE_DESCRIPTION("FPGA Manager for DFL FPGA Management Engine");
+MODULE_AUTHOR("Intel Corporation");
+MODULE_LICENSE("GPL v2");
+MODULE_ALIAS("platform:dfl-fme-mgr");
--
2.7.4

Cheers,
Moritz

Re: [PATCH v4 23/24] fpga: dfl: afu: add user afu sub feature support

From: Alan Tull <atull@kernel.org>
Date: 2018-03-21 23:51:37

On Tue, Mar 20, 2018 at 2:10 AM, Wu Hao [off-list ref] wrote:

quoted

+static int afu_mmap(struct file *filp, struct vm_area_struct *vma)
+{
+       struct fpga_afu_region region;
+       struct platform_device *pdev = filp->private_data;
+       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+       u64 size = vma->vm_end - vma->vm_start;
+       u64 offset;
+       int ret;
+
+       if (!(vma->vm_flags & VM_SHARED))
+               return -EINVAL;
+
+       offset = vma->vm_pgoff << PAGE_SHIFT;
+       ret = afu_get_region_by_offset(pdata, offset, size, &region);

Most of the functions here are afu_region_*, but there's also
afu_get_region_by_*.  Better if afu_region_get_by_* to be consistent.

quoted

+       if (ret)
+               return ret;
+
+       if (!(region.flags & FPGA_REGION_MMAP))

FPGA_REGION_*?  We already have FPGA regions which are something
different.  Please call this something else.

Yes, will replace it with DFL_FPGA_*.

Re: [PATCH v4 04/24] fpga: add device feature list support

From: Alan Tull <atull@kernel.org>
Date: 2018-03-21 23:55:41

On Tue, Feb 13, 2018 at 3:24 AM, Wu Hao [off-list ref] wrote:

Hi Hao,

+static int
+build_info_create_dev(struct build_feature_devs_info *binfo,
+                     enum fpga_id_type type, const char *name,
+                     void __iomem *ioaddr)
+{
+       struct platform_device *fdev;
+       int ret;
+
+       /* we will create a new device, commit current device first */
+       ret = build_info_commit_dev(binfo);
+       if (ret)
+               return ret;
+
+       /*
+        * we use -ENODEV as the initialization indicator which indicates
+        * whether the id need to be reclaimed
+        */
+       fdev = platform_device_alloc(name, -ENODEV);
+       if (!fdev)
+               return -ENOMEM;
+
+       binfo->feature_dev = fdev;
+       binfo->feature_num = 0;
+       binfo->ioaddr = ioaddr;
+       INIT_LIST_HEAD(&binfo->sub_features);
+
+       fdev->id = alloc_fpga_id(type, &fdev->dev);
+       if (fdev->id < 0)
+               return fdev->id;

Do we need platform_device_put if returning here?

Alan

+
+       fdev->dev.parent = &binfo->cdev->region.dev;
+
+       return 0;
+}
+

Re: [PATCH v4 04/24] fpga: add device feature list support

From: Wu Hao <hidden>
Date: 2018-03-22 04:50:09

On Wed, Mar 21, 2018 at 06:54:58PM -0500, Alan Tull wrote:

On Tue, Feb 13, 2018 at 3:24 AM, Wu Hao [off-list ref] wrote:

Hi Hao,

quoted

+static int
+build_info_create_dev(struct build_feature_devs_info *binfo,
+                     enum fpga_id_type type, const char *name,
+                     void __iomem *ioaddr)
+{
+       struct platform_device *fdev;
+       int ret;
+
+       /* we will create a new device, commit current device first */
+       ret = build_info_commit_dev(binfo);
+       if (ret)
+               return ret;
+
+       /*
+        * we use -ENODEV as the initialization indicator which indicates
+        * whether the id need to be reclaimed
+        */
+       fdev = platform_device_alloc(name, -ENODEV);
+       if (!fdev)
+               return -ENOMEM;
+
+       binfo->feature_dev = fdev;
+       binfo->feature_num = 0;
+       binfo->ioaddr = ioaddr;
+       INIT_LIST_HEAD(&binfo->sub_features);
+
+       fdev->id = alloc_fpga_id(type, &fdev->dev);
+       if (fdev->id < 0)
+               return fdev->id;

Do we need platform_device_put if returning here?

Hi Alan,

Actually if any error returned by this build_info_create_dev, it will cause
parse_feature_list returns error code and finally build_info_free is invoked.
In build_info_free function, it will do platform_device_put there, so we don't
have to add code to do put here. : )

Thanks
Hao

Alan

quoted

+
+       fdev->dev.parent = &binfo->cdev->region.dev;
+
+       return 0;
+}
+

Next 18 of 18 remaining

`h`	back out one level
`j`	next message in thread
`k`	previous message in thread
`l`	drill in
`Esc`	close help / fold thread tree
`?`	toggle this help