In order to support virtualization usage via PCIe SRIOV, this patch
adds two ioctls under FPGA Management Engine (FME) to release and
assign back the port device. In order to safely turn Port from PF
into VF and enable PCIe SRIOV, it requires user to invoke this
PORT_RELEASE ioctl to release port firstly to remove userspace
interfaces, and then configure the PF/VF access register in FME.
After disable SRIOV, it requires user to invoke this PORT_ASSIGN
ioctl to attach the port back to PF.
Ioctl interfaces:
* DFL_FPGA_FME_PORT_RELEASE
Release platform device of given port, it deletes port platform
device to remove related userspace interfaces on PF. After this
function, then it's safe to configure PF/VF access mode to VF,
and enable VFs via SRIOV.
* DFL_FPGA_FME_PORT_ASSIGN
Assign platform device of given port back to PF. After configure
PF/VF access mode to PF, this ioctl adds port platform device
back to re-enable related userspace interfaces on PF.
Signed-off-by: Zhang Yi Z <redacted>
Signed-off-by: Xu Yilun <yilun.xu@intel.com>
Signed-off-by: Wu Hao <redacted>
Acked-by: Alan Tull <atull@kernel.org>
Acked-by: Moritz Fischer <mdf@kernel.org>
Signed-off-by: Moritz Fischer <mdf@kernel.org>
---
v2: remove argsz from ioctls.
v4: split dfl_fpga_cdev_config_port to 2 functions *release/assign_port.
---
drivers/fpga/dfl-fme-main.c | 42 ++++++++++++++++
drivers/fpga/dfl.c | 113 +++++++++++++++++++++++++++++++++++++-----
drivers/fpga/dfl.h | 10 ++++
include/uapi/linux/fpga-dfl.h | 18 +++++++
4 files changed, 171 insertions(+), 12 deletions(-)
@@ -474,6 +478,7 @@ static int build_info_commit_dev(struct build_feature_devs_info *binfo)pdata->dev=fdev;pdata->num=binfo->feature_num;pdata->dfl_cdev=binfo->cdev;+pdata->id=FEATURE_DEV_ID_UNUSED;mutex_init(&pdata->lock);lockdep_set_class_and_name(&pdata->lock,&dfl_pdata_keys[type],dfl_pdata_key_strings[type]);
@@ -973,25 +978,27 @@ void dfl_fpga_feature_devs_remove(struct dfl_fpga_cdev *cdev){structdfl_feature_platform_data*pdata,*ptmp;-remove_feature_devs(cdev);-mutex_lock(&cdev->lock);-if(cdev->fme_dev){-/* the fme should be unregistered. */-WARN_ON(device_is_registered(cdev->fme_dev));+if(cdev->fme_dev)put_device(cdev->fme_dev);-}list_for_each_entry_safe(pdata,ptmp,&cdev->port_dev_list,node){structplatform_device*port_dev=pdata->dev;-/* the port should be unregistered. */-WARN_ON(device_is_registered(&port_dev->dev));+/* remove released ports */+if(!device_is_registered(&port_dev->dev)){+dfl_id_free(feature_dev_id_type(port_dev),+port_dev->id);+platform_device_put(port_dev);+}+list_del(&pdata->node);put_device(&port_dev->dev);}mutex_unlock(&cdev->lock);+remove_feature_devs(cdev);+fpga_region_unregister(cdev->region);devm_kfree(cdev->parent,cdev);}
@@ -1042,6 +1049,88 @@ static int __init dfl_fpga_init(void)returnret;}+/**+*dfl_fpga_cdev_release_port-releaseaportplatformdevice+*+*@cdev:parentcontainerdevice.+*@port_id:idoftheportplatformdevice.+*+*Thisfunctionallowsusertoreleaseaportplatformdevice.Thisisa+*mandatorystepbeforeturnaportfromPFintoVFforSRIOVsupport.+*+*Return:0onsuccess,negativeerrorcodeotherwise.+*/+intdfl_fpga_cdev_release_port(structdfl_fpga_cdev*cdev,intport_id)+{+structplatform_device*port_pdev;+intret=-ENODEV;++mutex_lock(&cdev->lock);+port_pdev=__dfl_fpga_cdev_find_port(cdev,&port_id,+dfl_fpga_check_port_id);+if(!port_pdev)+gotounlock_exit;++if(!device_is_registered(&port_pdev->dev)){+ret=-EBUSY;+gotoput_dev_exit;+}++ret=dfl_feature_dev_use_begin(dev_get_platdata(&port_pdev->dev));+if(ret)+gotoput_dev_exit;++platform_device_del(port_pdev);+cdev->released_port_num++;+put_dev_exit:+put_device(&port_pdev->dev);+unlock_exit:+mutex_unlock(&cdev->lock);+returnret;+}+EXPORT_SYMBOL_GPL(dfl_fpga_cdev_release_port);++/**+*dfl_fpga_cdev_assign_port-assignaportplatformdeviceback+*+*@cdev:parentcontainerdevice.+*@port_id:idoftheportplatformdevice.+*+*Thisfunctionallowsusertoassignaportplatformdeviceback.Thisis+*amandatorystepafterdisableSRIOVsupport.+*+*Return:0onsuccess,negativeerrorcodeotherwise.+*/+intdfl_fpga_cdev_assign_port(structdfl_fpga_cdev*cdev,intport_id)+{+structplatform_device*port_pdev;+intret=-ENODEV;++mutex_lock(&cdev->lock);+port_pdev=__dfl_fpga_cdev_find_port(cdev,&port_id,+dfl_fpga_check_port_id);+if(!port_pdev)+gotounlock_exit;++if(device_is_registered(&port_pdev->dev)){+ret=-EBUSY;+gotoput_dev_exit;+}++ret=platform_device_add(port_pdev);+if(ret)+gotoput_dev_exit;++dfl_feature_dev_use_end(dev_get_platdata(&port_pdev->dev));+cdev->released_port_num--;+put_dev_exit:+put_device(&port_pdev->dev);+unlock_exit:+mutex_unlock(&cdev->lock);+returnret;+}+EXPORT_SYMBOL_GPL(dfl_fpga_cdev_assign_port);+staticvoid__exitdfl_fpga_exit(void){dfl_chardev_uinit();
This patch enables the standard sriov support. It allows user to
enable SRIOV (and VFs), then user could pass through accelerators
(VFs) into virtual machine or use VFs directly in host.
Signed-off-by: Zhang Yi Z <redacted>
Signed-off-by: Xu Yilun <yilun.xu@intel.com>
Signed-off-by: Wu Hao <redacted>
Acked-by: Alan Tull <atull@kernel.org>
Acked-by: Moritz Fischer <mdf@kernel.org>
Signed-off-by: Moritz Fischer <mdf@kernel.org>
---
v2: remove DRV/MODULE_VERSION modifications.
v4: split __dfl_fpga_cdev_config_port_vf into 2 functions with
locking added.
---
drivers/fpga/dfl-pci.c | 36 ++++++++++++++++++++++
drivers/fpga/dfl.c | 82 ++++++++++++++++++++++++++++++++++++++++++++++++++
drivers/fpga/dfl.h | 3 +-
3 files changed, 120 insertions(+), 1 deletion(-)
This patch introduces userclock sysfs interfaces for AFU, user
could use these interfaces for clock setting to AFU.
Please note that, this is only working for port header feature
with revision 0, for later revisions, userclock setting is moved
to a separated private feature, so one revision sysfs interface
is exposed to userspace application for this purpose too.
Signed-off-by: Ananda Ravuri <redacted>
Signed-off-by: Russ Weight <redacted>
Signed-off-by: Xu Yilun <yilun.xu@intel.com>
Signed-off-by: Wu Hao <redacted>
Acked-by: Alan Tull <atull@kernel.org>
Signed-off-by: Moritz Fischer <mdf@kernel.org>
---
v2: rebased, and switched to use device_add/remove_groups for sysfs
v3: update kernel version and date in sysfs doc
v4: rebased.
---
Documentation/ABI/testing/sysfs-platform-dfl-port | 35 +++++++
drivers/fpga/dfl-afu-main.c | 114 +++++++++++++++++++++-
drivers/fpga/dfl.h | 9 ++
3 files changed, 157 insertions(+), 1 deletion(-)
@@ -46,3 +46,38 @@ Contact: Wu Hao <hao.wu@intel.com> Description: Read-write. Read or set AFU latency tolerance reporting value. Set ltr to 1 if the AFU can tolerate latency >= 40us or set it to 0 if it is latency sensitive.++What: /sys/bus/platform/devices/dfl-port.0/revision+Date: August 2019+KernelVersion: 5.4+Contact: Wu Hao <hao.wu@intel.com>+Description: Read-only. Read this file to get the revision of port header+ feature.++What: /sys/bus/platform/devices/dfl-port.0/userclk_freqcmd+Date: August 2019+KernelVersion: 5.4+Contact: Wu Hao <hao.wu@intel.com>+Description: Write-only. User writes command to this interface to set+ userclock to AFU.++What: /sys/bus/platform/devices/dfl-port.0/userclk_freqsts+Date: August 2019+KernelVersion: 5.4+Contact: Wu Hao <hao.wu@intel.com>+Description: Read-only. Read this file to get the status of issued command+ to userclck_freqcmd.++What: /sys/bus/platform/devices/dfl-port.0/userclk_freqcntrcmd+Date: August 2019+KernelVersion: 5.4+Contact: Wu Hao <hao.wu@intel.com>+Description: Write-only. User writes command to this interface to set+ userclock counter.++What: /sys/bus/platform/devices/dfl-port.0/userclk_freqcntrsts+Date: August 2019+KernelVersion: 5.4+Contact: Wu Hao <hao.wu@intel.com>+Description: Read-only. Read this file to get the status of issued command+ to userclck_freqcntrcmd.
As these two functions are used by other private features. e.g.
in error reporting private feature, it requires to check port status
and reset port for error clearing.
Signed-off-by: Xu Yilun <yilun.xu@intel.com>
Signed-off-by: Wu Hao <redacted>
Acked-by: Moritz Fischer <mdf@kernel.org>
Acked-by: Alan Tull <atull@kernel.org>
Signed-off-by: Moritz Fischer <mdf@kernel.org>
---
v2: rebased
---
drivers/fpga/dfl-afu-main.c | 25 ++++++++++++++-----------
drivers/fpga/dfl-afu.h | 3 +++
2 files changed, 17 insertions(+), 11 deletions(-)
Error reporting is one important private feature, it reports error
detected on port and accelerated function unit (AFU). It introduces
several sysfs interfaces to allow userspace to check and clear
errors detected by hardware.
Signed-off-by: Xu Yilun <yilun.xu@intel.com>
Signed-off-by: Wu Hao <redacted>
Acked-by: Alan Tull <atull@kernel.org>
Signed-off-by: Moritz Fischer <mdf@kernel.org>
---
v2: switch to device_add/remove_group for sysfs.
v3: update kernel version and date in sysfs doc
v4: remove dev_dbg in init/uinit callback function.
---
Documentation/ABI/testing/sysfs-platform-dfl-port | 39 ++++
drivers/fpga/Makefile | 1 +
drivers/fpga/dfl-afu-error.c | 221 ++++++++++++++++++++++
drivers/fpga/dfl-afu-main.c | 4 +
drivers/fpga/dfl-afu.h | 4 +
5 files changed, 269 insertions(+)
create mode 100644 drivers/fpga/dfl-afu-error.c
@@ -81,3 +81,42 @@ KernelVersion: 5.4 Contact: Wu Hao <hao.wu@intel.com> Description: Read-only. Read this file to get the status of issued command to userclck_freqcntrcmd.++What: /sys/bus/platform/devices/dfl-port.0/errors/revision+Date: August 2019+KernelVersion: 5.4+Contact: Wu Hao <hao.wu@intel.com>+Description: Read-only. Read this file to get the revision of this error+ reporting private feature.++What: /sys/bus/platform/devices/dfl-port.0/errors/errors+Date: August 2019+KernelVersion: 5.4+Contact: Wu Hao <hao.wu@intel.com>+Description: Read-only. Read this file to get errors detected on port and+ Accelerated Function Unit (AFU).++What: /sys/bus/platform/devices/dfl-port.0/errors/first_error+Date: August 2019+KernelVersion: 5.4+Contact: Wu Hao <hao.wu@intel.com>+Description: Read-only. Read this file to get the first error detected by+ hardware.++What: /sys/bus/platform/devices/dfl-port.0/errors/first_malformed_req+Date: August 2019+KernelVersion: 5.4+Contact: Wu Hao <hao.wu@intel.com>+Description: Read-only. Read this file to get the first malformed request+ captured by hardware.++What: /sys/bus/platform/devices/dfl-port.0/errors/clear+Date: August 2019+KernelVersion: 5.4+Contact: Wu Hao <hao.wu@intel.com>+Description: Write-only. Write error code to this file to clear errors.+ Write fails with -EINVAL if input parsing fails or input error+ code doesn't match.+ Write fails with -EBUSY or -ETIMEDOUT if error can't be cleared+ as hardware is in low power state (-EBUSY) or not responding+ (-ETIMEDOUT).
@@ -0,0 +1,221 @@+// SPDX-License-Identifier: GPL-2.0+/*+*DriverforFPGAAcceleratedFunctionUnit(AFU)ErrorReporting+*+*Copyright2019IntelCorporation,Inc.+*+*Authors:+*WuHao<hao.wu@linux.intel.com>+*XiaoGuangrong<guangrong.xiao@linux.intel.com>+*JosephGrecco<joe.grecco@intel.com>+*EnnoLuebbers<enno.luebbers@intel.com>+*TimWhisonant<tim.whisonant@intel.com>+*AnandaRavuri<ananda.ravuri@intel.com>+*MitchelHenry<henry.mitchel@intel.com>+*/++#include<linux/uaccess.h>++#include"dfl-afu.h"++#define PORT_ERROR_MASK 0x8+#define PORT_ERROR 0x10+#define PORT_FIRST_ERROR 0x18+#define PORT_MALFORMED_REQ0 0x20+#define PORT_MALFORMED_REQ1 0x28++#define ERROR_MASK GENMASK_ULL(63, 0)++/* mask or unmask port errors by the error mask register. */+staticvoid__port_err_mask(structdevice*dev,boolmask)+{+void__iomem*base;++base=dfl_get_feature_ioaddr_by_id(dev,PORT_FEATURE_ID_ERROR);++writeq(mask?ERROR_MASK:0,base+PORT_ERROR_MASK);+}++/* clear port errors. */+staticint__port_err_clear(structdevice*dev,u64err)+{+structplatform_device*pdev=to_platform_device(dev);+void__iomem*base_err,*base_hdr;+intret;+u64v;++base_err=dfl_get_feature_ioaddr_by_id(dev,PORT_FEATURE_ID_ERROR);+base_hdr=dfl_get_feature_ioaddr_by_id(dev,PORT_FEATURE_ID_HEADER);++/*+*clearPortErrors+*+*-CheckforAP6State+*-HaltPortbykeepingPortinreset+*-SetPORTErrormasktoall1tomaskerrors+*-Clearallerrors+*-SetPortmasktoall0toenableerrors+*-Allerrorsstartcapturingnewerrors+*-EnablePortbypullingtheportoutofreset+*/++/* if device is still in AP6 power state, can not clear any error. */+v=readq(base_hdr+PORT_HDR_STS);+if(FIELD_GET(PORT_STS_PWR_STATE,v)==PORT_STS_PWR_STATE_AP6){+dev_err(dev,"Could not clear errors, device in AP6 state.\n");+return-EBUSY;+}++/* Halt Port by keeping Port in reset */+ret=__port_disable(pdev);+if(ret)+returnret;++/* Mask all errors */+__port_err_mask(dev,true);++/* Clear errors if err input matches with current port errors.*/+v=readq(base_err+PORT_ERROR);++if(v==err){+writeq(v,base_err+PORT_ERROR);++v=readq(base_err+PORT_FIRST_ERROR);+writeq(v,base_err+PORT_FIRST_ERROR);+}else{+ret=-EINVAL;+}++/* Clear mask */+__port_err_mask(dev,false);++/* Enable the Port by clear the reset */+__port_enable(pdev);++returnret;+}++staticssize_trevision_show(structdevice*dev,structdevice_attribute*attr,+char*buf)+{+void__iomem*base;++base=dfl_get_feature_ioaddr_by_id(dev,PORT_FEATURE_ID_ERROR);++returnsprintf(buf,"%u\n",dfl_feature_revision(base));+}+staticDEVICE_ATTR_RO(revision);++staticssize_terrors_show(structdevice*dev,structdevice_attribute*attr,+char*buf)+{+structdfl_feature_platform_data*pdata=dev_get_platdata(dev);+void__iomem*base;+u64error;++base=dfl_get_feature_ioaddr_by_id(dev,PORT_FEATURE_ID_ERROR);++mutex_lock(&pdata->lock);+error=readq(base+PORT_ERROR);+mutex_unlock(&pdata->lock);++returnsprintf(buf,"0x%llx\n",(unsignedlonglong)error);+}+staticDEVICE_ATTR_RO(errors);++staticssize_tfirst_error_show(structdevice*dev,+structdevice_attribute*attr,char*buf)+{+structdfl_feature_platform_data*pdata=dev_get_platdata(dev);+void__iomem*base;+u64error;++base=dfl_get_feature_ioaddr_by_id(dev,PORT_FEATURE_ID_ERROR);++mutex_lock(&pdata->lock);+error=readq(base+PORT_FIRST_ERROR);+mutex_unlock(&pdata->lock);++returnsprintf(buf,"0x%llx\n",(unsignedlonglong)error);+}+staticDEVICE_ATTR_RO(first_error);++staticssize_tfirst_malformed_req_show(structdevice*dev,+structdevice_attribute*attr,+char*buf)+{+structdfl_feature_platform_data*pdata=dev_get_platdata(dev);+void__iomem*base;+u64req0,req1;++base=dfl_get_feature_ioaddr_by_id(dev,PORT_FEATURE_ID_ERROR);++mutex_lock(&pdata->lock);+req0=readq(base+PORT_MALFORMED_REQ0);+req1=readq(base+PORT_MALFORMED_REQ1);+mutex_unlock(&pdata->lock);++returnsprintf(buf,"0x%016llx%016llx\n",+(unsignedlonglong)req1,(unsignedlonglong)req0);+}+staticDEVICE_ATTR_RO(first_malformed_req);++staticssize_tclear_store(structdevice*dev,structdevice_attribute*attr,+constchar*buff,size_tcount)+{+structdfl_feature_platform_data*pdata=dev_get_platdata(dev);+u64value;+intret;++if(kstrtou64(buff,0,&value))+return-EINVAL;++mutex_lock(&pdata->lock);+ret=__port_err_clear(dev,value);+mutex_unlock(&pdata->lock);++returnret?ret:count;+}+staticDEVICE_ATTR_WO(clear);++staticstructattribute*port_err_attrs[]={+&dev_attr_revision.attr,+&dev_attr_errors.attr,+&dev_attr_first_error.attr,+&dev_attr_first_malformed_req.attr,+&dev_attr_clear.attr,+NULL,+};++staticstructattribute_groupport_err_attr_group={+.attrs=port_err_attrs,+.name="errors",+};++staticintport_err_init(structplatform_device*pdev,+structdfl_feature*feature)+{+structdfl_feature_platform_data*pdata=dev_get_platdata(&pdev->dev);++mutex_lock(&pdata->lock);+__port_err_mask(&pdev->dev,false);+mutex_unlock(&pdata->lock);++returndevice_add_group(&pdev->dev,&port_err_attr_group);+}++staticvoidport_err_uinit(structplatform_device*pdev,+structdfl_feature*feature)+{+device_remove_group(&pdev->dev,&port_err_attr_group);+}++conststructdfl_feature_idport_err_id_table[]={+{.id=PORT_FEATURE_ID_ERROR,},+{0,}+};++conststructdfl_feature_opsport_err_ops={+.init=port_err_init,+.uinit=port_err_uinit,+};
This patch makes uinit callback of sub features optional. With
this change, people don't need to prepare any empty uinit callback.
Signed-off-by: Wu Hao <redacted>
---
drivers/fpga/dfl.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
STP (SignalTap) is one of the private features under the port for
debugging. This patch adds private feature driver support for it
to allow userspace applications to mmap related mmio region and
provide STP service.
Signed-off-by: Xu Yilun <yilun.xu@intel.com>
Signed-off-by: Wu Hao <redacted>
Acked-by: Moritz Fischer <mdf@kernel.org>
Acked-by: Alan Tull <atull@kernel.org>
Signed-off-by: Moritz Fischer <mdf@kernel.org>
---
v4: remove uinit callback which does nothing.
remove dev_dbg in init callback function.
---
drivers/fpga/dfl-afu-main.c | 25 +++++++++++++++++++++++++
1 file changed, 25 insertions(+)
@@ -21,3 +21,26 @@ Contact: Wu Hao <hao.wu@intel.com> Description: Read-only. It returns Bitstream (static FPGA region) meta data, which includes the synthesis date, seed and other information of this static FPGA region.++What: /sys/bus/platform/devices/dfl-fme.0/cache_size+Date: August 2019+KernelVersion: 5.4+Contact: Wu Hao <hao.wu@intel.com>+Description: Read-only. It returns cache size of this FPGA device.++What: /sys/bus/platform/devices/dfl-fme.0/fabric_version+Date: August 2019+KernelVersion: 5.4+Contact: Wu Hao <hao.wu@intel.com>+Description: Read-only. It returns fabric version of this FPGA device.+ Userspace applications need this information to select+ best data channels per different fabric design.++What: /sys/bus/platform/devices/dfl-fme.0/socket_id+Date: August 2019+KernelVersion: 5.4+Contact: Wu Hao <hao.wu@intel.com>+Description: Read-only. It returns socket_id to indicate which socket+ this FPGA belongs to, only valid for integrated solution.+ User only needs this information, in case standard numa node+ can't provide correct information.
This patch adds support for global error reporting for FPGA
Management Engine (FME), it introduces sysfs interfaces to
report different error detected by the hardware, and allow
user to clear errors or inject error for testing purpose.
Signed-off-by: Luwei Kang <redacted>
Signed-off-by: Ananda Ravuri <redacted>
Signed-off-by: Xu Yilun <yilun.xu@intel.com>
Signed-off-by: Wu Hao <redacted>
Acked-by: Alan Tull <atull@kernel.org>
Signed-off-by: Moritz Fischer <mdf@kernel.org>
---
v2: switch to device_add/remove_groups for sysfs.
v3: update kernel version and date in sysfs doc
v4: rebase, remove dev_dbg in init/uinit callback.
---
Documentation/ABI/testing/sysfs-platform-dfl-fme | 75 +++++
drivers/fpga/Makefile | 2 +-
drivers/fpga/dfl-fme-error.c | 381 +++++++++++++++++++++++
drivers/fpga/dfl-fme-main.c | 4 +
drivers/fpga/dfl-fme.h | 2 +
drivers/fpga/dfl.h | 2 +
6 files changed, 465 insertions(+), 1 deletion(-)
create mode 100644 drivers/fpga/dfl-fme-error.c
@@ -44,3 +44,78 @@ Description: Read-only. It returns socket_id to indicate which socket this FPGA belongs to, only valid for integrated solution. User only needs this information, in case standard numa node can't provide correct information.++What: /sys/bus/platform/devices/dfl-fme.0/errors/revision+Date: August 2019+KernelVersion: 5.4+Contact: Wu Hao <hao.wu@intel.com>+Description: Read-only. Read this file to get the revision of this global+ error reporting private feature.++What: /sys/bus/platform/devices/dfl-fme.0/errors/pcie0_errors+Date: August 2019+KernelVersion: 5.4+Contact: Wu Hao <hao.wu@intel.com>+Description: Read-Write. Read this file for errors detected on pcie0 link.+ Write this file to clear errors logged in pcie0_errors. Write+ fails with -EINVAL if input parsing fails or input error code+ doesn't match.++What: /sys/bus/platform/devices/dfl-fme.0/errors/pcie1_errors+Date: August 2019+KernelVersion: 5.4+Contact: Wu Hao <hao.wu@intel.com>+Description: Read-Write. Read this file for errors detected on pcie1 link.+ Write this file to clear errors logged in pcie1_errors. Write+ fails with -EINVAL if input parsing fails or input error code+ doesn't match.++What: /sys/bus/platform/devices/dfl-fme.0/errors/nonfatal_errors+Date: August 2019+KernelVersion: 5.4+Contact: Wu Hao <hao.wu@intel.com>+Description: Read-only. It returns non-fatal errors detected.++What: /sys/bus/platform/devices/dfl-fme.0/errors/catfatal_errors+Date: August 2019+KernelVersion: 5.4+Contact: Wu Hao <hao.wu@intel.com>+Description: Read-only. It returns catastrophic and fatal errors detected.++What: /sys/bus/platform/devices/dfl-fme.0/errors/inject_error+Date: August 2019+KernelVersion: 5.4+Contact: Wu Hao <hao.wu@intel.com>+Description: Read-Write. Read this file to check errors injected. Write this+ file to inject errors for testing purpose. Write fails with+ -EINVAL if input parsing fails or input inject error code isn't+ supported.++What: /sys/bus/platform/devices/dfl-fme.0/errors/fme-errors/errors+Date: August 2019+KernelVersion: 5.4+Contact: Wu Hao <hao.wu@intel.com>+Description: Read-only. Read this file to get errors detected by hardware.++What: /sys/bus/platform/devices/dfl-fme.0/errors/fme-errors/first_error+Date: August 2019+KernelVersion: 5.4+Contact: Wu Hao <hao.wu@intel.com>+Description: Read-only. Read this file to get the first error detected by+ hardware.++What: /sys/bus/platform/devices/dfl-fme.0/errors/fme-errors/next_error+Date: August 2019+KernelVersion: 5.4+Contact: Wu Hao <hao.wu@intel.com>+Description: Read-only. Read this file to get the second error detected by+ hardware.++What: /sys/bus/platform/devices/dfl-fme.0/errors/fme-errors/clear+Date: August 2019+KernelVersion: 5.4+Contact: Wu Hao <hao.wu@intel.com>+Description: Write-only. Write error code to this file to clear all errors+ logged in errors, first_error and next_error. Write fails with+ -EINVAL if input parsing fails or input error code doesn't+ match.
This patch adds virtualization support description for DFL based
FPGA devices (based on PCIe SRIOV), and introductions to new
interfaces added by new dfl private feature drivers.
[mdf@kernel.org: Fixed up to make it work with new reStructuredText docs]
Signed-off-by: Xu Yilun <yilun.xu@intel.com>
Signed-off-by: Wu Hao <redacted>
Acked-by: Alan Tull <atull@kernel.org>
Signed-off-by: Moritz Fischer <mdf@kernel.org>
---
Documentation/fpga/dfl.rst | 105 +++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 105 insertions(+)
@@ -87,6 +87,8 @@ The following functions are exposed through ioctls:- Get driver API version (DFL_FPGA_GET_API_VERSION)- Check for extensions (DFL_FPGA_CHECK_EXTENSION)- Program bitstream (DFL_FPGA_FME_PORT_PR)+- Assign port to PF (DFL_FPGA_FME_PORT_ASSIGN)+- Release port from PF (DFL_FPGA_FME_PORT_RELEASE) More functions are exposed through sysfs (/sys/class/fpga_region/regionX/dfl-fme.n/):
@@ -102,6 +104,10 @@ More functions are exposed through sysfs one FPGA device may have more than one port, this sysfs interface indicates how many ports the FPGA device has.+ Global error reporting management (errors/)+ error reporting sysfs interfaces allow user to read errors detected by the+ hardware, and clear the logged errors.+ FIU - PORT ==========
@@ -143,6 +149,10 @@ More functions are exposed through sysfs: Read Accelerator GUID (afu_id) afu_id indicates which PR bitstream is programmed to this AFU.+ Error reporting (errors/)+ error reporting sysfs interfaces allow user to read port/afu errors+ detected by the hardware, and clear the logged errors.+ DFL Framework Overview ======================
@@ -218,6 +228,101 @@ the compat_id exposed by the target FPGA region. This check is usually done by userspace before calling the reconfiguration IOCTL.+FPGA virtualization - PCIe SRIOV+================================+This section describes the virtualization support on DFL based FPGA device to+enable accessing an accelerator from applications running in a virtual machine+(VM). This section only describes the PCIe based FPGA device with SRIOV support.++Features supported by the particular FPGA device are exposed through Device+Feature Lists, as illustrated below:++::++ +-------------------------------+ +-------------++| PF | | VF |+ +-------------------------------+ +-------------++ ^ ^ ^ ^+| | | |+ +-----|------------|---------|--------------|-------++| | | | | |+| +-----+ +-------+ +-------+ +-------+ |+| | FME | | Port0 | | Port1 | | Port2 | |+| +-----+ +-------+ +-------+ +-------+ |+| ^ ^ ^ |+| | | | |+| +-------+ +------+ +-------+ |+| | AFU | | AFU | | AFU | |+| +-------+ +------+ +-------+ |+| |+| DFL based FPGA PCIe Device |+ +---------------------------------------------------+++FME is always accessed through the physical function (PF).++Ports (and related AFUs) are accessed via PF by default, but could be exposed+through virtual function (VF) devices via PCIe SRIOV. Each VF only contains+1 Port and 1 AFU for isolation. Users could assign individual VFs (accelerators)+created via PCIe SRIOV interface, to virtual machines.++The driver organization in virtualization case is illustrated below:+::++ +-------++------++------+ |+| FME || FME || FME | |+| FPGA || FPGA || FPGA | |+ |Manager||Bridge||Region| |+ +-------++------++------+ |+ +-----------------------+ +--------+ | +--------++| FME | | AFU | | | AFU |+| Module | | Module | | | Module |+ +-----------------------+ +--------+ | +--------++ +-----------------------+ | +-----------------------++| FPGA Container Device | | | FPGA Container Device |+| (FPGA Base Region) | | | (FPGA Base Region) |+ +-----------------------+ | +-----------------------++ +------------------+ | +------------------++| FPGA PCIE Module | | Virtual | FPGA PCIE Module |+ +------------------+ Host | Machine +------------------++ -------------------------------------- | ------------------------------+ +---------------+ | +---------------++| PCI PF Device | | | PCI VF Device |+ +---------------+ | +---------------+++FPGA PCIe device driver is always loaded first once a FPGA PCIe PF or VF device+is detected. It:++* Finishes enumeration on both FPGA PCIe PF and VF device using common+ interfaces from DFL framework.+* Supports SRIOV.++The FME device driver plays a management role in this driver architecture, it+provides ioctls to release Port from PF and assign Port to PF. After release+a port from PF, then it's safe to expose this port through a VF via PCIe SRIOV+sysfs interface.++To enable accessing an accelerator from applications running in a VM, the+respective AFU's port needs to be assigned to a VF using the following steps:++#. The PF owns all AFU ports by default. Any port that needs to be+ reassigned to a VF must first be released through the+ DFL_FPGA_FME_PORT_RELEASE ioctl on the FME device.++#. Once N ports are released from PF, then user can use command below+ to enable SRIOV and VFs. Each VF owns only one Port with AFU.++ ::++ echo N > $PCI_DEVICE_PATH/sriov_numvfs++#. Pass through the VFs to VMs++#. The AFU under VF is accessible from applications in VM (using the+ same driver inside the VF).++Note that an FME can't be assigned to a VF, thus PR and other management+functions are only available via the PF.+ Device enumeration ================== This section introduces how applications enumerate the fpga device from
@@ -281,6 +281,21 @@ static int dfl_feature_instance_init(struct platform_device *pdev,returnret;}+staticbooldfl_feature_drv_match(structdfl_feature*feature,+structdfl_feature_driver*driver)+{+conststructdfl_feature_id*ids=driver->id_table;++if(ids){+while(ids->id){+if(ids->id==feature->id)+returntrue;+ids++;+}+}+returnfalse;+}+/***dfl_fpga_dev_feature_init-initforsubfeaturesofdflfeaturedevice*@pdev:featuredevice.
@@ -301,8 +316,7 @@ int dfl_fpga_dev_feature_init(struct platform_device *pdev,while(drv->ops){dfl_fpga_dev_for_each_feature(pdata,feature){-/* match feature and drv using id */-if(feature->id==drv->id){+if(dfl_feature_drv_match(feature,drv)){ret=dfl_feature_instance_init(pdev,pdata,feature,drv);if(ret)
@@ -30,8 +30,8 @@/* plus one for fme device */#define MAX_DFL_FEATURE_DEV_NUM (MAX_DFL_FPGA_PORT_NUM + 1)-/* Reserved 0x0 for Header Group Register and 0xff for AFU */-#define FEATURE_ID_FIU_HEADER 0x0+/* Reserved 0xfe for Header Group Register and 0xff for AFU */+#define FEATURE_ID_FIU_HEADER 0xfe#define FEATURE_ID_AFU 0xff#define FME_FEATURE_ID_HEADER FEATURE_ID_FIU_HEADER
This patch introduces more sysfs interfaces for Accelerated
Function Unit (AFU). These interfaces allow users to read
current AFU Power State (APx), read / clear AFU Power (APx)
events which are sticky to identify transient APx state,
and manage AFU's LTR (latency tolerance reporting).
Signed-off-by: Ananda Ravuri <redacted>
Signed-off-by: Xu Yilun <yilun.xu@intel.com>
Signed-off-by: Wu Hao <redacted>
Acked-by: Alan Tull <atull@kernel.org>
Signed-off-by: Moritz Fischer <mdf@kernel.org>
---
v2: rebased, and remove DRV/MODULE_VERSION modifications
v3: update kernel version and date in sysfs doc
v4: improve description in sysfs doc avoid misunderstanding.
switch to kstrtobool in sysfs entry store function.
---
Documentation/ABI/testing/sysfs-platform-dfl-port | 32 +++++
drivers/fpga/dfl-afu-main.c | 137 ++++++++++++++++++++++
drivers/fpga/dfl.h | 11 ++
3 files changed, 180 insertions(+)
@@ -14,3 +14,35 @@ Description: Read-only. User can program different PR bitstreams to FPGA Accelerator Function Unit (AFU) for different functions. It returns uuid which could be used to identify which PR bitstream is programmed in this AFU.++What: /sys/bus/platform/devices/dfl-port.0/power_state+Date: August 2019+KernelVersion: 5.4+Contact: Wu Hao <hao.wu@intel.com>+Description: Read-only. It reports the APx (AFU Power) state, different APx+ means different throttling level. When reading this file, it+ returns "0" - Normal / "1" - AP1 / "2" - AP2 / "6" - AP6.++What: /sys/bus/platform/devices/dfl-port.0/ap1_event+Date: August 2019+KernelVersion: 5.4+Contact: Wu Hao <hao.wu@intel.com>+Description: Read-write. Read this file for AP1 (AFU Power State 1) event.+ It's used to indicate transient AP1 state. Write 1 to this+ file to clear AP1 event.++What: /sys/bus/platform/devices/dfl-port.0/ap2_event+Date: August 2019+KernelVersion: 5.4+Contact: Wu Hao <hao.wu@intel.com>+Description: Read-write. Read this file for AP2 (AFU Power State 2) event.+ It's used to indicate transient AP2 state. Write 1 to this+ file to clear AP2 event.++What: /sys/bus/platform/devices/dfl-port.0/ltr+Date: August 2019+KernelVersion: 5.4+Contact: Wu Hao <hao.wu@intel.com>+Description: Read-write. Read or set AFU latency tolerance reporting value.+ Set ltr to 1 if the AFU can tolerate latency >= 40us or set it+ to 0 if it is latency sensitive.
On Sun, Aug 04, 2019 at 06:20:14PM +0800, Wu Hao wrote:
quoted hunk
This patch introduces userclock sysfs interfaces for AFU, user
could use these interfaces for clock setting to AFU.
Please note that, this is only working for port header feature
with revision 0, for later revisions, userclock setting is moved
to a separated private feature, so one revision sysfs interface
is exposed to userspace application for this purpose too.
Signed-off-by: Ananda Ravuri <redacted>
Signed-off-by: Russ Weight <redacted>
Signed-off-by: Xu Yilun <yilun.xu@intel.com>
Signed-off-by: Wu Hao <redacted>
Acked-by: Alan Tull <atull@kernel.org>
Signed-off-by: Moritz Fischer <mdf@kernel.org>
---
v2: rebased, and switched to use device_add/remove_groups for sysfs
v3: update kernel version and date in sysfs doc
v4: rebased.
---
Documentation/ABI/testing/sysfs-platform-dfl-port | 35 +++++++
drivers/fpga/dfl-afu-main.c | 114 +++++++++++++++++++++-
drivers/fpga/dfl.h | 9 ++
3 files changed, 157 insertions(+), 1 deletion(-)
@@ -46,3 +46,38 @@ Contact: Wu Hao <hao.wu@intel.com> Description: Read-write. Read or set AFU latency tolerance reporting value. Set ltr to 1 if the AFU can tolerate latency >= 40us or set it to 0 if it is latency sensitive.++What: /sys/bus/platform/devices/dfl-port.0/revision+Date: August 2019+KernelVersion: 5.4+Contact: Wu Hao <hao.wu@intel.com>+Description: Read-only. Read this file to get the revision of port header+ feature.
What does "revision" mean?
It feels like you are creating a different set of sysfs files depending
on the revision field. Which is fine, sysfs is one-value-per-file and
userspace needs to handle if the file is present or not. So why not
just rely on that and not have to mess with 'revision' at all? What is
userspace going to do with that information?
quoted hunk
+
+What: /sys/bus/platform/devices/dfl-port.0/userclk_freqcmd
+Date: August 2019
+KernelVersion: 5.4
+Contact: Wu Hao [off-list ref]
+Description: Write-only. User writes command to this interface to set
+ userclock to AFU.
+
+What: /sys/bus/platform/devices/dfl-port.0/userclk_freqsts
+Date: August 2019
+KernelVersion: 5.4
+Contact: Wu Hao [off-list ref]
+Description: Read-only. Read this file to get the status of issued command
+ to userclck_freqcmd.
+
+What: /sys/bus/platform/devices/dfl-port.0/userclk_freqcntrcmd
+Date: August 2019
+KernelVersion: 5.4
+Contact: Wu Hao [off-list ref]
+Description: Write-only. User writes command to this interface to set
+ userclock counter.
+
+What: /sys/bus/platform/devices/dfl-port.0/userclk_freqcntrsts
+Date: August 2019
+KernelVersion: 5.4
+Contact: Wu Hao [off-list ref]
+Description: Read-only. Read this file to get the status of issued command
+ to userclck_freqcntrcmd.
This all needs to be reworked based on the ability for devices to
properly add groups when they are bound on probe (the core does it for
you, no need for the driver to do it.) But until then, you should at
least consider:
+ if (ret)
+ return ret;
+
+ /*
+ * if revision > 0, the userclock will be moved from port hdr register
+ * region to a separated private feature.
+ */
+ if (dfl_feature_revision(feature->ioaddr) > 0)
+ return 0;
+
+ ret = device_add_groups(&pdev->dev, port_hdr_userclk_groups);
+ if (ret)
+ device_remove_groups(&pdev->dev, port_hdr_groups);
struct attribute_group has is_visible() as a callback to have the core
show or not show, individual attributes when they are created. So no
need for a second group of attributes and you needing to add/remove
them, just add them all and let the callback handle the "is visible"
logic. Makes cleanup _so_ much easier (i.e. you don't have to do it.)
thanks,
greg k-h
On Sun, Aug 04, 2019 at 06:20:16PM +0800, Wu Hao wrote:
quoted hunk
As these two functions are used by other private features. e.g.
in error reporting private feature, it requires to check port status
and reset port for error clearing.
Signed-off-by: Xu Yilun <yilun.xu@intel.com>
Signed-off-by: Wu Hao <redacted>
Acked-by: Moritz Fischer <mdf@kernel.org>
Acked-by: Alan Tull <atull@kernel.org>
Signed-off-by: Moritz Fischer <mdf@kernel.org>
---
v2: rebased
---
drivers/fpga/dfl-afu-main.c | 25 ++++++++++++++-----------
drivers/fpga/dfl-afu.h | 3 +++
2 files changed, 17 insertions(+), 11 deletions(-)
worst global function name ever.
Don't polute the global namespace like this for a single driver. If you
REALLY need it, then use a prefix that shows it is your individual
dfl_special_sauce_platform_device_only type thing.
thanks,
greg k-h
On Sun, Aug 04, 2019 at 06:20:17PM +0800, Wu Hao wrote:
quoted hunk
Error reporting is one important private feature, it reports error
detected on port and accelerated function unit (AFU). It introduces
several sysfs interfaces to allow userspace to check and clear
errors detected by hardware.
Signed-off-by: Xu Yilun <yilun.xu@intel.com>
Signed-off-by: Wu Hao <redacted>
Acked-by: Alan Tull <atull@kernel.org>
Signed-off-by: Moritz Fischer <mdf@kernel.org>
---
v2: switch to device_add/remove_group for sysfs.
v3: update kernel version and date in sysfs doc
v4: remove dev_dbg in init/uinit callback function.
---
Documentation/ABI/testing/sysfs-platform-dfl-port | 39 ++++
drivers/fpga/Makefile | 1 +
drivers/fpga/dfl-afu-error.c | 221 ++++++++++++++++++++++
drivers/fpga/dfl-afu-main.c | 4 +
drivers/fpga/dfl-afu.h | 4 +
5 files changed, 269 insertions(+)
create mode 100644 drivers/fpga/dfl-afu-error.c
@@ -81,3 +81,42 @@ KernelVersion: 5.4 Contact: Wu Hao <hao.wu@intel.com> Description: Read-only. Read this file to get the status of issued command to userclck_freqcntrcmd.++What: /sys/bus/platform/devices/dfl-port.0/errors/revision+Date: August 2019+KernelVersion: 5.4+Contact: Wu Hao <hao.wu@intel.com>+Description: Read-only. Read this file to get the revision of this error+ reporting private feature.
Same revision question here that I had on an earlier patch.
quoted hunk
+
+What: /sys/bus/platform/devices/dfl-port.0/errors/errors
+Date: August 2019
+KernelVersion: 5.4
+Contact: Wu Hao [off-list ref]
+Description: Read-only. Read this file to get errors detected on port and
+ Accelerated Function Unit (AFU).
+
+What: /sys/bus/platform/devices/dfl-port.0/errors/first_error
+Date: August 2019
+KernelVersion: 5.4
+Contact: Wu Hao [off-list ref]
+Description: Read-only. Read this file to get the first error detected by
+ hardware.
+
+What: /sys/bus/platform/devices/dfl-port.0/errors/first_malformed_req
+Date: August 2019
+KernelVersion: 5.4
+Contact: Wu Hao [off-list ref]
+Description: Read-only. Read this file to get the first malformed request
+ captured by hardware.
+
+What: /sys/bus/platform/devices/dfl-port.0/errors/clear
+Date: August 2019
+KernelVersion: 5.4
+Contact: Wu Hao [off-list ref]
+Description: Write-only. Write error code to this file to clear errors.
+ Write fails with -EINVAL if input parsing fails or input error
+ code doesn't match.
+ Write fails with -EBUSY or -ETIMEDOUT if error can't be cleared
+ as hardware is in low power state (-EBUSY) or not responding
+ (-ETIMEDOUT).
@@ -0,0 +1,221 @@+// SPDX-License-Identifier: GPL-2.0+/*+*DriverforFPGAAcceleratedFunctionUnit(AFU)ErrorReporting+*+*Copyright2019IntelCorporation,Inc.+*+*Authors:+*WuHao<hao.wu@linux.intel.com>+*XiaoGuangrong<guangrong.xiao@linux.intel.com>+*JosephGrecco<joe.grecco@intel.com>+*EnnoLuebbers<enno.luebbers@intel.com>+*TimWhisonant<tim.whisonant@intel.com>+*AnandaRavuri<ananda.ravuri@intel.com>+*MitchelHenry<henry.mitchel@intel.com>+*/++#include<linux/uaccess.h>++#include"dfl-afu.h"++#define PORT_ERROR_MASK 0x8+#define PORT_ERROR 0x10+#define PORT_FIRST_ERROR 0x18+#define PORT_MALFORMED_REQ0 0x20+#define PORT_MALFORMED_REQ1 0x28++#define ERROR_MASK GENMASK_ULL(63, 0)++/* mask or unmask port errors by the error mask register. */+staticvoid__port_err_mask(structdevice*dev,boolmask)+{+void__iomem*base;++base=dfl_get_feature_ioaddr_by_id(dev,PORT_FEATURE_ID_ERROR);++writeq(mask?ERROR_MASK:0,base+PORT_ERROR_MASK);+}++/* clear port errors. */+staticint__port_err_clear(structdevice*dev,u64err)+{+structplatform_device*pdev=to_platform_device(dev);+void__iomem*base_err,*base_hdr;+intret;+u64v;++base_err=dfl_get_feature_ioaddr_by_id(dev,PORT_FEATURE_ID_ERROR);+base_hdr=dfl_get_feature_ioaddr_by_id(dev,PORT_FEATURE_ID_HEADER);++/*+*clearPortErrors+*+*-CheckforAP6State+*-HaltPortbykeepingPortinreset+*-SetPORTErrormasktoall1tomaskerrors+*-Clearallerrors+*-SetPortmasktoall0toenableerrors+*-Allerrorsstartcapturingnewerrors+*-EnablePortbypullingtheportoutofreset+*/++/* if device is still in AP6 power state, can not clear any error. */+v=readq(base_hdr+PORT_HDR_STS);+if(FIELD_GET(PORT_STS_PWR_STATE,v)==PORT_STS_PWR_STATE_AP6){+dev_err(dev,"Could not clear errors, device in AP6 state.\n");+return-EBUSY;+}++/* Halt Port by keeping Port in reset */+ret=__port_disable(pdev);+if(ret)+returnret;++/* Mask all errors */+__port_err_mask(dev,true);++/* Clear errors if err input matches with current port errors.*/+v=readq(base_err+PORT_ERROR);++if(v==err){+writeq(v,base_err+PORT_ERROR);++v=readq(base_err+PORT_FIRST_ERROR);+writeq(v,base_err+PORT_FIRST_ERROR);+}else{+ret=-EINVAL;+}++/* Clear mask */+__port_err_mask(dev,false);++/* Enable the Port by clear the reset */+__port_enable(pdev);++returnret;+}++staticssize_trevision_show(structdevice*dev,structdevice_attribute*attr,+char*buf)+{+void__iomem*base;++base=dfl_get_feature_ioaddr_by_id(dev,PORT_FEATURE_ID_ERROR);++returnsprintf(buf,"%u\n",dfl_feature_revision(base));+}+staticDEVICE_ATTR_RO(revision);++staticssize_terrors_show(structdevice*dev,structdevice_attribute*attr,+char*buf)+{+structdfl_feature_platform_data*pdata=dev_get_platdata(dev);+void__iomem*base;+u64error;++base=dfl_get_feature_ioaddr_by_id(dev,PORT_FEATURE_ID_ERROR);++mutex_lock(&pdata->lock);+error=readq(base+PORT_ERROR);+mutex_unlock(&pdata->lock);++returnsprintf(buf,"0x%llx\n",(unsignedlonglong)error);+}+staticDEVICE_ATTR_RO(errors);++staticssize_tfirst_error_show(structdevice*dev,+structdevice_attribute*attr,char*buf)+{+structdfl_feature_platform_data*pdata=dev_get_platdata(dev);+void__iomem*base;+u64error;++base=dfl_get_feature_ioaddr_by_id(dev,PORT_FEATURE_ID_ERROR);++mutex_lock(&pdata->lock);+error=readq(base+PORT_FIRST_ERROR);+mutex_unlock(&pdata->lock);++returnsprintf(buf,"0x%llx\n",(unsignedlonglong)error);+}+staticDEVICE_ATTR_RO(first_error);++staticssize_tfirst_malformed_req_show(structdevice*dev,+structdevice_attribute*attr,+char*buf)+{+structdfl_feature_platform_data*pdata=dev_get_platdata(dev);+void__iomem*base;+u64req0,req1;++base=dfl_get_feature_ioaddr_by_id(dev,PORT_FEATURE_ID_ERROR);++mutex_lock(&pdata->lock);+req0=readq(base+PORT_MALFORMED_REQ0);+req1=readq(base+PORT_MALFORMED_REQ1);+mutex_unlock(&pdata->lock);++returnsprintf(buf,"0x%016llx%016llx\n",+(unsignedlonglong)req1,(unsignedlonglong)req0);+}+staticDEVICE_ATTR_RO(first_malformed_req);++staticssize_tclear_store(structdevice*dev,structdevice_attribute*attr,+constchar*buff,size_tcount)+{+structdfl_feature_platform_data*pdata=dev_get_platdata(dev);+u64value;+intret;++if(kstrtou64(buff,0,&value))+return-EINVAL;++mutex_lock(&pdata->lock);+ret=__port_err_clear(dev,value);+mutex_unlock(&pdata->lock);++returnret?ret:count;+}+staticDEVICE_ATTR_WO(clear);++staticstructattribute*port_err_attrs[]={+&dev_attr_revision.attr,+&dev_attr_errors.attr,+&dev_attr_first_error.attr,+&dev_attr_first_malformed_req.attr,+&dev_attr_clear.attr,+NULL,+};++staticstructattribute_groupport_err_attr_group={+.attrs=port_err_attrs,+.name="errors",+};++staticintport_err_init(structplatform_device*pdev,+structdfl_feature*feature)+{+structdfl_feature_platform_data*pdata=dev_get_platdata(&pdev->dev);++mutex_lock(&pdata->lock);+__port_err_mask(&pdev->dev,false);+mutex_unlock(&pdata->lock);
Locking one data structure and then modifying another one is up there
with "things never to do in the kernel unless you want a huge
headache!".
On Sun, Aug 04, 2019 at 06:20:21PM +0800, Wu Hao wrote:
+static int fme_global_err_init(struct platform_device *pdev,
+ struct dfl_feature *feature)
+{
+ struct device *dev;
+ int ret = 0;
+
+ dev = kzalloc(sizeof(*dev), GFP_KERNEL);
+ if (!dev)
+ return -ENOMEM;
+
+ dev->parent = &pdev->dev;
+ dev->release = err_dev_release;
+ dev_set_name(dev, "errors");
+
+ fme_error_enable(feature);
+
+ ret = device_register(dev);
+ if (ret) {
+ put_device(dev);
+ return ret;
+ }
+
+ ret = device_add_groups(dev, error_groups);
cute, but no, you do not create a whole struct device for a subdir. Use
the attribute group name like you did on earlier patches.
And again, you raced userspace and lost :(
thanks,
greg k-h
On Mon, Aug 05, 2019 at 05:51:13PM +0200, Greg KH wrote:
On Sun, Aug 04, 2019 at 06:20:14PM +0800, Wu Hao wrote:
quoted
This patch introduces userclock sysfs interfaces for AFU, user
could use these interfaces for clock setting to AFU.
Please note that, this is only working for port header feature
with revision 0, for later revisions, userclock setting is moved
to a separated private feature, so one revision sysfs interface
is exposed to userspace application for this purpose too.
Signed-off-by: Ananda Ravuri <redacted>
Signed-off-by: Russ Weight <redacted>
Signed-off-by: Xu Yilun <yilun.xu@intel.com>
Signed-off-by: Wu Hao <redacted>
Acked-by: Alan Tull <atull@kernel.org>
Signed-off-by: Moritz Fischer <mdf@kernel.org>
---
v2: rebased, and switched to use device_add/remove_groups for sysfs
v3: update kernel version and date in sysfs doc
v4: rebased.
---
Documentation/ABI/testing/sysfs-platform-dfl-port | 35 +++++++
drivers/fpga/dfl-afu-main.c | 114 +++++++++++++++++++++-
drivers/fpga/dfl.h | 9 ++
3 files changed, 157 insertions(+), 1 deletion(-)
@@ -46,3 +46,38 @@ Contact: Wu Hao <hao.wu@intel.com> Description: Read-write. Read or set AFU latency tolerance reporting value. Set ltr to 1 if the AFU can tolerate latency >= 40us or set it to 0 if it is latency sensitive.++What: /sys/bus/platform/devices/dfl-port.0/revision+Date: August 2019+KernelVersion: 5.4+Contact: Wu Hao <hao.wu@intel.com>+Description: Read-only. Read this file to get the revision of port header+ feature.
What does "revision" mean?
It feels like you are creating a different set of sysfs files depending
on the revision field. Which is fine, sysfs is one-value-per-file and
userspace needs to handle if the file is present or not. So why not
just rely on that and not have to mess with 'revision' at all? What is
userspace going to do with that information?
Hi Greg
Thanks for the review and comments,
Yes, different revision of private feature may have different hardware
features, so driver will expose different set of sysfs entries. revision
here is used to help userspace to distinguish them. I think it makes
sense to just rely on if sysfs entry exists or not, manage revision in
userspace code may be quit difficult. Plan to remove this entry in the
next version.
quoted
+
+What: /sys/bus/platform/devices/dfl-port.0/userclk_freqcmd
+Date: August 2019
+KernelVersion: 5.4
+Contact: Wu Hao [off-list ref]
+Description: Write-only. User writes command to this interface to set
+ userclock to AFU.
+
+What: /sys/bus/platform/devices/dfl-port.0/userclk_freqsts
+Date: August 2019
+KernelVersion: 5.4
+Contact: Wu Hao [off-list ref]
+Description: Read-only. Read this file to get the status of issued command
+ to userclck_freqcmd.
+
+What: /sys/bus/platform/devices/dfl-port.0/userclk_freqcntrcmd
+Date: August 2019
+KernelVersion: 5.4
+Contact: Wu Hao [off-list ref]
+Description: Write-only. User writes command to this interface to set
+ userclock counter.
+
+What: /sys/bus/platform/devices/dfl-port.0/userclk_freqcntrsts
+Date: August 2019
+KernelVersion: 5.4
+Contact: Wu Hao [off-list ref]
+Description: Read-only. Read this file to get the status of issued command
+ to userclck_freqcntrcmd.
This all needs to be reworked based on the ability for devices to
properly add groups when they are bound on probe (the core does it for
you, no need for the driver to do it.) But until then, you should at
least consider:
quoted
+ if (ret)
+ return ret;
+
+ /*
+ * if revision > 0, the userclock will be moved from port hdr register
+ * region to a separated private feature.
+ */
+ if (dfl_feature_revision(feature->ioaddr) > 0)
+ return 0;
+
+ ret = device_add_groups(&pdev->dev, port_hdr_userclk_groups);
+ if (ret)
+ device_remove_groups(&pdev->dev, port_hdr_groups);
struct attribute_group has is_visible() as a callback to have the core
show or not show, individual attributes when they are created. So no
need for a second group of attributes and you needing to add/remove
them, just add them all and let the callback handle the "is visible"
logic. Makes cleanup _so_ much easier (i.e. you don't have to do it.)
Sure, will use is_visible() here instead in the next version, it does
make thing more clear. Thanks a lot of the comments.
Hao
On Mon, Aug 05, 2019 at 05:52:40PM +0200, Greg KH wrote:
On Sun, Aug 04, 2019 at 06:20:16PM +0800, Wu Hao wrote:
quoted
As these two functions are used by other private features. e.g.
in error reporting private feature, it requires to check port status
and reset port for error clearing.
Signed-off-by: Xu Yilun <yilun.xu@intel.com>
Signed-off-by: Wu Hao <redacted>
Acked-by: Moritz Fischer <mdf@kernel.org>
Acked-by: Alan Tull <atull@kernel.org>
Signed-off-by: Moritz Fischer <mdf@kernel.org>
---
v2: rebased
---
drivers/fpga/dfl-afu-main.c | 25 ++++++++++++++-----------
drivers/fpga/dfl-afu.h | 3 +++
2 files changed, 17 insertions(+), 11 deletions(-)
worst global function name ever.
Don't polute the global namespace like this for a single driver. If you
REALLY need it, then use a prefix that shows it is your individual
dfl_special_sauce_platform_device_only type thing.
Oh.. Sure.. Let me fix the naming in the next version.
Thanks
Hao
On Mon, Aug 05, 2019 at 05:54:37PM +0200, Greg KH wrote:
On Sun, Aug 04, 2019 at 06:20:17PM +0800, Wu Hao wrote:
quoted
Error reporting is one important private feature, it reports error
detected on port and accelerated function unit (AFU). It introduces
several sysfs interfaces to allow userspace to check and clear
errors detected by hardware.
Signed-off-by: Xu Yilun <yilun.xu@intel.com>
Signed-off-by: Wu Hao <redacted>
Acked-by: Alan Tull <atull@kernel.org>
Signed-off-by: Moritz Fischer <mdf@kernel.org>
---
v2: switch to device_add/remove_group for sysfs.
v3: update kernel version and date in sysfs doc
v4: remove dev_dbg in init/uinit callback function.
---
Documentation/ABI/testing/sysfs-platform-dfl-port | 39 ++++
drivers/fpga/Makefile | 1 +
drivers/fpga/dfl-afu-error.c | 221 ++++++++++++++++++++++
drivers/fpga/dfl-afu-main.c | 4 +
drivers/fpga/dfl-afu.h | 4 +
5 files changed, 269 insertions(+)
create mode 100644 drivers/fpga/dfl-afu-error.c
@@ -81,3 +81,42 @@ KernelVersion: 5.4 Contact: Wu Hao <hao.wu@intel.com> Description: Read-only. Read this file to get the status of issued command to userclck_freqcntrcmd.++What: /sys/bus/platform/devices/dfl-port.0/errors/revision+Date: August 2019+KernelVersion: 5.4+Contact: Wu Hao <hao.wu@intel.com>+Description: Read-only. Read this file to get the revision of this error+ reporting private feature.
Same revision question here that I had on an earlier patch.
quoted
+
+What: /sys/bus/platform/devices/dfl-port.0/errors/errors
+Date: August 2019
+KernelVersion: 5.4
+Contact: Wu Hao [off-list ref]
+Description: Read-only. Read this file to get errors detected on port and
+ Accelerated Function Unit (AFU).
+
+What: /sys/bus/platform/devices/dfl-port.0/errors/first_error
+Date: August 2019
+KernelVersion: 5.4
+Contact: Wu Hao [off-list ref]
+Description: Read-only. Read this file to get the first error detected by
+ hardware.
+
+What: /sys/bus/platform/devices/dfl-port.0/errors/first_malformed_req
+Date: August 2019
+KernelVersion: 5.4
+Contact: Wu Hao [off-list ref]
+Description: Read-only. Read this file to get the first malformed request
+ captured by hardware.
+
+What: /sys/bus/platform/devices/dfl-port.0/errors/clear
+Date: August 2019
+KernelVersion: 5.4
+Contact: Wu Hao [off-list ref]
+Description: Write-only. Write error code to this file to clear errors.
+ Write fails with -EINVAL if input parsing fails or input error
+ code doesn't match.
+ Write fails with -EBUSY or -ETIMEDOUT if error can't be cleared
+ as hardware is in low power state (-EBUSY) or not responding
+ (-ETIMEDOUT).
@@ -0,0 +1,221 @@+// SPDX-License-Identifier: GPL-2.0+/*+*DriverforFPGAAcceleratedFunctionUnit(AFU)ErrorReporting+*+*Copyright2019IntelCorporation,Inc.+*+*Authors:+*WuHao<hao.wu@linux.intel.com>+*XiaoGuangrong<guangrong.xiao@linux.intel.com>+*JosephGrecco<joe.grecco@intel.com>+*EnnoLuebbers<enno.luebbers@intel.com>+*TimWhisonant<tim.whisonant@intel.com>+*AnandaRavuri<ananda.ravuri@intel.com>+*MitchelHenry<henry.mitchel@intel.com>+*/++#include<linux/uaccess.h>++#include"dfl-afu.h"++#define PORT_ERROR_MASK 0x8+#define PORT_ERROR 0x10+#define PORT_FIRST_ERROR 0x18+#define PORT_MALFORMED_REQ0 0x20+#define PORT_MALFORMED_REQ1 0x28++#define ERROR_MASK GENMASK_ULL(63, 0)++/* mask or unmask port errors by the error mask register. */+staticvoid__port_err_mask(structdevice*dev,boolmask)+{+void__iomem*base;++base=dfl_get_feature_ioaddr_by_id(dev,PORT_FEATURE_ID_ERROR);++writeq(mask?ERROR_MASK:0,base+PORT_ERROR_MASK);+}++/* clear port errors. */+staticint__port_err_clear(structdevice*dev,u64err)+{+structplatform_device*pdev=to_platform_device(dev);+void__iomem*base_err,*base_hdr;+intret;+u64v;++base_err=dfl_get_feature_ioaddr_by_id(dev,PORT_FEATURE_ID_ERROR);+base_hdr=dfl_get_feature_ioaddr_by_id(dev,PORT_FEATURE_ID_HEADER);++/*+*clearPortErrors+*+*-CheckforAP6State+*-HaltPortbykeepingPortinreset+*-SetPORTErrormasktoall1tomaskerrors+*-Clearallerrors+*-SetPortmasktoall0toenableerrors+*-Allerrorsstartcapturingnewerrors+*-EnablePortbypullingtheportoutofreset+*/++/* if device is still in AP6 power state, can not clear any error. */+v=readq(base_hdr+PORT_HDR_STS);+if(FIELD_GET(PORT_STS_PWR_STATE,v)==PORT_STS_PWR_STATE_AP6){+dev_err(dev,"Could not clear errors, device in AP6 state.\n");+return-EBUSY;+}++/* Halt Port by keeping Port in reset */+ret=__port_disable(pdev);+if(ret)+returnret;++/* Mask all errors */+__port_err_mask(dev,true);++/* Clear errors if err input matches with current port errors.*/+v=readq(base_err+PORT_ERROR);++if(v==err){+writeq(v,base_err+PORT_ERROR);++v=readq(base_err+PORT_FIRST_ERROR);+writeq(v,base_err+PORT_FIRST_ERROR);+}else{+ret=-EINVAL;+}++/* Clear mask */+__port_err_mask(dev,false);++/* Enable the Port by clear the reset */+__port_enable(pdev);++returnret;+}++staticssize_trevision_show(structdevice*dev,structdevice_attribute*attr,+char*buf)+{+void__iomem*base;++base=dfl_get_feature_ioaddr_by_id(dev,PORT_FEATURE_ID_ERROR);++returnsprintf(buf,"%u\n",dfl_feature_revision(base));+}+staticDEVICE_ATTR_RO(revision);++staticssize_terrors_show(structdevice*dev,structdevice_attribute*attr,+char*buf)+{+structdfl_feature_platform_data*pdata=dev_get_platdata(dev);+void__iomem*base;+u64error;++base=dfl_get_feature_ioaddr_by_id(dev,PORT_FEATURE_ID_ERROR);++mutex_lock(&pdata->lock);+error=readq(base+PORT_ERROR);+mutex_unlock(&pdata->lock);++returnsprintf(buf,"0x%llx\n",(unsignedlonglong)error);+}+staticDEVICE_ATTR_RO(errors);++staticssize_tfirst_error_show(structdevice*dev,+structdevice_attribute*attr,char*buf)+{+structdfl_feature_platform_data*pdata=dev_get_platdata(dev);+void__iomem*base;+u64error;++base=dfl_get_feature_ioaddr_by_id(dev,PORT_FEATURE_ID_ERROR);++mutex_lock(&pdata->lock);+error=readq(base+PORT_FIRST_ERROR);+mutex_unlock(&pdata->lock);++returnsprintf(buf,"0x%llx\n",(unsignedlonglong)error);+}+staticDEVICE_ATTR_RO(first_error);++staticssize_tfirst_malformed_req_show(structdevice*dev,+structdevice_attribute*attr,+char*buf)+{+structdfl_feature_platform_data*pdata=dev_get_platdata(dev);+void__iomem*base;+u64req0,req1;++base=dfl_get_feature_ioaddr_by_id(dev,PORT_FEATURE_ID_ERROR);++mutex_lock(&pdata->lock);+req0=readq(base+PORT_MALFORMED_REQ0);+req1=readq(base+PORT_MALFORMED_REQ1);+mutex_unlock(&pdata->lock);++returnsprintf(buf,"0x%016llx%016llx\n",+(unsignedlonglong)req1,(unsignedlonglong)req0);+}+staticDEVICE_ATTR_RO(first_malformed_req);++staticssize_tclear_store(structdevice*dev,structdevice_attribute*attr,+constchar*buff,size_tcount)+{+structdfl_feature_platform_data*pdata=dev_get_platdata(dev);+u64value;+intret;++if(kstrtou64(buff,0,&value))+return-EINVAL;++mutex_lock(&pdata->lock);+ret=__port_err_clear(dev,value);+mutex_unlock(&pdata->lock);++returnret?ret:count;+}+staticDEVICE_ATTR_WO(clear);++staticstructattribute*port_err_attrs[]={+&dev_attr_revision.attr,+&dev_attr_errors.attr,+&dev_attr_first_error.attr,+&dev_attr_first_malformed_req.attr,+&dev_attr_clear.attr,+NULL,+};++staticstructattribute_groupport_err_attr_group={+.attrs=port_err_attrs,+.name="errors",+};++staticintport_err_init(structplatform_device*pdev,+structdfl_feature*feature)+{+structdfl_feature_platform_data*pdata=dev_get_platdata(&pdev->dev);++mutex_lock(&pdata->lock);+__port_err_mask(&pdev->dev,false);+mutex_unlock(&pdata->lock);
Locking one data structure and then modifying another one is up there
with "things never to do in the kernel unless you want a huge
headache!".
Actually we always use the same lock for protection as other places, but
the code may cause some misunderstanding, let me improve this part in
the next version.
Do you mind giving some more hints on this one? I guess I didn't fully
understand this. :( Add handling if device_add_group failed here, or
something else I should fix?
Thanks
Hao
On Wed, Aug 07, 2019 at 10:45:22AM +0800, Wu Hao wrote:
On Mon, Aug 05, 2019 at 05:56:26PM +0200, Greg KH wrote:
quoted
On Sun, Aug 04, 2019 at 06:20:21PM +0800, Wu Hao wrote:
quoted
+static int fme_global_err_init(struct platform_device *pdev,
+ struct dfl_feature *feature)
+{
+ struct device *dev;
+ int ret = 0;
+
+ dev = kzalloc(sizeof(*dev), GFP_KERNEL);
+ if (!dev)
+ return -ENOMEM;
+
+ dev->parent = &pdev->dev;
+ dev->release = err_dev_release;
+ dev_set_name(dev, "errors");
+
+ fme_error_enable(feature);
+
+ ret = device_register(dev);
+ if (ret) {
+ put_device(dev);
+ return ret;
+ }
+
+ ret = device_add_groups(dev, error_groups);
cute, but no, you do not create a whole struct device for a subdir. Use
the attribute group name like you did on earlier patches.
Sure, let me fix it in the next version.
quoted
And again, you raced userspace and lost :(
Same here, could you please give some more hints here?
Oh.. I see..
I should follow [1] as this is a platform driver. I will fix it. Thanks!
[PATCH 00/11] Platform drivers, provide a way to add sysfs groups easily
[1]https://lkml.org/lkml/2019/7/4/181
Hao
On Wed, Aug 07, 2019 at 04:08:25PM +0800, Wu Hao wrote:
On Wed, Aug 07, 2019 at 10:45:22AM +0800, Wu Hao wrote:
quoted
On Mon, Aug 05, 2019 at 05:56:26PM +0200, Greg KH wrote:
quoted
On Sun, Aug 04, 2019 at 06:20:21PM +0800, Wu Hao wrote:
quoted
+static int fme_global_err_init(struct platform_device *pdev,
+ struct dfl_feature *feature)
+{
+ struct device *dev;
+ int ret = 0;
+
+ dev = kzalloc(sizeof(*dev), GFP_KERNEL);
+ if (!dev)
+ return -ENOMEM;
+
+ dev->parent = &pdev->dev;
+ dev->release = err_dev_release;
+ dev_set_name(dev, "errors");
+
+ fme_error_enable(feature);
+
+ ret = device_register(dev);
+ if (ret) {
+ put_device(dev);
+ return ret;
+ }
+
+ ret = device_add_groups(dev, error_groups);
cute, but no, you do not create a whole struct device for a subdir. Use
the attribute group name like you did on earlier patches.
Sure, let me fix it in the next version.
quoted
And again, you raced userspace and lost :(
Same here, could you please give some more hints here?
Oh.. I see..
I should follow [1] as this is a platform driver. I will fix it. Thanks!
[PATCH 00/11] Platform drivers, provide a way to add sysfs groups easily
[1]https://lkml.org/lkml/2019/7/4/181
Yes, that is the correct thing to do.
thanks,
greg k-h
From: Moritz Fischer <mdf@kernel.org> Date: 2019-08-09 19:51:53
On Sun, Aug 04, 2019 at 06:20:18PM +0800, Wu Hao wrote:
This patch makes uinit callback of sub features optional. With
this change, people don't need to prepare any empty uinit callback.
Signed-off-by: Wu Hao <redacted>