[RFC PATCH v12 00/17] dlb: introduce DLB device driver

@@ -0,0 +1,323 @@
+.. SPDX-License-Identifier: GPL-2.0-only
+
+===========================================
+Intel(R) Dynamic Load Balancer Overview
+===========================================
+
+:Authors: Gage Eads and Mike Ximing Chen
+
+Contents
+========
+
+- Introduction
+- Scheduling
+- Queue Entry
+- Port
+- Queue
+- Credits
+- Scheduling Domain
+- Interrupts
+- Power Management
+- User Interface
+- Reset
+
+Introduction
+============
+
+The Intel(r) Dynamic Load Balancer (Intel(r) DLB) is a PCIe device and hardware
+accelerator that provides load-balanced, prioritized scheduling for event based
+workloads across CPU cores. It can be used to save CPU resources in high
+throughput pipelines by replacing software based distribution and synchronization
+schemes. Using Intel DLB in place of software based methods has been
+demonstrated to reduce CPU utilization up to 2-3 CPU cores per DLB device,
+while also improving packet processing pipeline performance.
+
+In many applications running on processors with a large number of cores,
+workloads must be distributed (load-balanced) across a number of cores.
+In packet processing applications, for example, streams of incoming packets can
+exceed the capacity of any single core. So they have to be divided between
+available worker cores. The workload can be split by either breaking the
+processing flow into stages and places distinct stages on separate cores in a
+daisy chain fashion (a pipeline), or spraying packets across multiple workers
+that may be executing the same processing stage. Many systems employ a hybrid
+approach whereby each packet encounters multiple pipelined stages with
+distribution across multiple workers at each individual stage.
+
+The following diagram shows a typical packet processing pipeline with the Intel DLB.
+
+                              WC1              WC4
+ +-----+   +----+   +---+  /      \  +---+  /      \  +---+   +----+   +-----+
+ |NIC  |   |Rx  |   |DLB| /        \ |DLB| /        \ |DLB|   |Tx  |   |NIC  |
+ |Ports|---|Core|---|   |-----WC2----|   |-----WC5----|   |---|Core|---|Ports|
+ +-----+   -----+   +---+ \        / +---+ \        / +---+   +----+   ------+
+                           \      /         \      /
+                              WC3              WC6
+
+WCs are the worker cores which process packets distributed by DLB. Without
+hardware accelerators (such as DLB) the distribution and load balancing are normally
+carried out by software running on CPU cores. Using Intel DLB in this case not only
+saves the CPU resources but also improves the system performance.
+
+The Intel DLB consists of queues and arbiters that connect producer cores (which
+enqueues events to DLB) and consumer cores (which dequeue events from DLB). The
+device implements load-balanced queueing features including:
+
+- Lock-free multi-producer/multi-consumer operation.
+- Multiple priority levels for varying traffic types.
+- Direct traffic (i.e. multi-producer/single-consumer)
+- Simple unordered load-balanced distribution.
+- Atomic lock free load balancing across multiple consumers.
+- Queue element reordering feature allowing ordered load-balanced distribution.
+
+Note: this document uses 'DLB' when discussing the device hardware and 'dlb' when
+discussing the driver implementation.
+
+Following diagram illustrates the functional blocks of an Intel DLB device.
+
+                                       +----+
+                                       |    |
+                        +----------+   |    |   +-------+
+                       /|   IQ     |---|----|--/|       |
+                      / +----------+   |    | / |  CP   |
+                     /                 |    |/  +-------+
+        +--------+  /                  |    |
+        |        | /    +----------+   |   /|   +-------+
+        |  PP    |------|   IQ     |---|----|---|       |
+        +--------+ \    +----------+   | /  |   |  CP   |
+                    \                  |/   |   +-------+
+           ...       \     ...         |    |
+        +--------+    \               /|    |   +-------+
+        |        |     \+----------+ / |    |   |       |
+        |  PP    |------|   IQ     |/--|----|---|  CP   |
+        +--------+      +----------+   |    |   +-------+
+                                       |    |
+                                       +----+     ...
+PP: Producer Port                        |
+CP: Consumer Port                        |
+IQ: Internal Queue                   DLB Scheduler
+
+
+As shown in the diagram, the high-level Intel DLB data flow is as follows:
+ - Software threads interact with the hardware by enqueuing and dequeuing Queue
+   Elements (QEs).
+ - QEs are sent through a Producer Port (PP) to the Intel DLB internal QE
+   storage (internal queues), optionally being reordered along the way.
+ - The Intel DLB schedules QEs from internal queues to a consumer according to
+   a two-stage priority arbiter (DLB Scheduler).
+ - Once scheduled, the Intel DLB writes the QE to a memory-based Consumer Port
+   (CP), which the software thread reads and processes.
+
+
+Scheduling Types
+================
+
+Intel DLB supports four types of scheduling of 'events' (i.e., queue elements),
+where an event can represent any type of data (e.g. a network packet). The
+first, `directed`, is multi-producer/single-consumer style scheduling. The
+remaining three are multi-producer/multi-consumer, and support load-balancing
+across the consumers.
+
+- `Directed`: events are scheduled to a single consumer.
+
+- `Unordered`: events are load-balanced across consumers without any ordering
+                 guarantees.
+
+- `Ordered`: events are load-balanced across consumers, and the consumer can
+               re-enqueue its events so the device re-orders them into the
+               original order. This scheduling type allows software to
+               parallelize ordered event processing without the synchronization
+               cost of re-ordering packets.
+
+- `Atomic`: events are load-balanced across consumers, with the guarantee that
+              events from a particular 'flow' are only scheduled to a single
+              consumer at a time (but can migrate over time). This allows, for
+              example, packet processing applications to parallelize while
+              avoiding locks on per-flow data and maintaining ordering within a
+              flow.
+
+Intel DLB provides hierarchical priority scheduling, with eight priority
+levels within each. Each consumer selects up to eight queues to receive events
+from, and assigns a priority to each of these 'connected' queues. To schedule
+an event to a consumer, the device selects the highest priority non-empty queue
+of the (up to) eight connected queues. Within that queue, the device selects
+the highest priority event available (selecting a lower priority event for
+starvation avoidance 1% of the time, by default).
+
+The device also supports four load-balanced scheduler classes of service. Each
+class of service receives a (user-configurable) guaranteed percentage of the
+scheduler bandwidth, and any unreserved bandwidth is divided evenly among the
+four classes.
+
+Queue Element
+===========
+
+Each event is contained in a queue element (QE), the fundamental unit of
+communication through the device, which consists of 8B of data and 8B of
+metadata, as depicted below.
+
+QE structure format
+::
+    data     :64
+    opaque   :16
+    qid      :8
+    sched    :2
+    priority :3
+    msg_type :3
+    lock_id  :16
+    rsvd     :8
+    cmd      :8
+
+The `data` field can be any type that fits within 8B (pointer, integer,
+etc.); DLB merely copies this field from producer to consumer. The
+`opaque` and `msg_type` fields behave the same way.
+
+`qid` is set by the producer to specify to which DLB internal queue it wishes
+to enqueue this QE. The ID spaces for load-balanced and directed queues are both
+zero-based.
+
+`sched` controls the scheduling type: atomic, unordered, ordered, or
+directed. The first three scheduling types are only valid for load-balanced
+queues, and the directed scheduling type is only valid for directed queues.
+This field distinguishes whether `qid` is load-balanced or directed, since
+their ID spaces overlap.
+
+`priority` is the priority with which this QE should be scheduled.
+
+`lock_id`, used for atomic scheduling and ignored for ordered and unordered
+scheduling, identifies the atomic flow to which the QE belongs. When sending a
+directed event, `lock_id` is simply copied like the `data`, `opaque`, and
+`msg_type` fields.
+
+`cmd` specifies the operation, such as:
+- Enqueue a new QE
+- Forward a QE that was dequeued
+- Complete/terminate a QE that was dequeued
+- Return one or more consumer queue tokens.
+- Arm the port's consumer queue interrupt.
+
+Port
+====
+
+A core's interface to the DLB is called a "port", and consists of an MMIO
+region (producer port) through which the core enqueues a queue element, and an
+in-memory queue (the "consumer queue" or consumer port) to which the device
+schedules QEs. A core enqueues a QE to a device queue, then the device
+schedules the event to a port. Software specifies the connection of queues
+and ports; i.e. for each queue, to which ports the device is allowed to
+schedule its events. The device uses a credit scheme to prevent overflow of
+the on-device queue storage.
+
+Applications interface directly with the device by mapping the port's memory
+and MMIO regions into the application's address space for enqueue and dequeue
+operations, but call into the kernel driver for configuration operations. An
+application can be polling- or interrupt-driven; DLB supports both modes
+of operation.
+
+Internal Queue
+==============
+
+A DLB device supports an implementation specific and runtime discoverable
+number of load-balanced (i.e. capable of atomic, ordered, and unordered
+scheduling) and directed queues. Each internal queue supports a set of
+priority levels.
+
+A load-balanced queue is capable of scheduling its events to any combination
+of load-balanced ports, whereas each directed queue can only haveone-to-one
+mapping with any directed port. There is no restriction on port or queue types
+when a port enqueues an event to a queue; that is, a load-balanced port can
+enqueue to a directed queue and vice versa.
+
+Credits
+=======
+
+The Intel DLB uses a credit scheme to prevent overflow of the on-device
+queue storage, with separate credits for load-balanced and directed queues. A
+port spends one credit when it enqueues a QE, and one credit is replenished
+when a QE is dequeued from a consumer queue. Each scheduling domain has one pool
+of load-balanced credits and one pool of directed credits; software is
+responsible for managing the allocation and replenishment of these credits among
+the scheduling domain's ports.
+
+Scheduling Domain
+=================
+
+Device resources -- including ports, queues, and credits -- are contained
+within a scheduling domain. Scheduling domains are isolated from one another; a
+port can only enqueue to and dequeue from queues within its scheduling domain.
+
+The scheduling domain with a set of resources is created through configfs, and
+can be accessed/shared by multiple processes.
+
+Consumer Queue Interrupts
+=========================
+
+Each port has its own interrupt which fires, if armed, when the consumer queue
+depth becomes non-zero. Software arms an interrupt by enqueueing a special
+'interrupt arm' command to the device through the port's MMIO window.
+
+Power Management
+================
+
+The kernel driver keeps the device in D3Hot (power save mode) when not in use.
+The driver transitions the device to D0 when the first device file is opened,
+and keeps it there until there are no open device files or memory mappings.
+
+User Interface
+==============
+
+The dlb driver uses configfs and sysfs as its primary user interfaces. While
+the sysfs is used to configure and inquire device-wide operation and
+resources, the configfs provides domain/queue/port level configuration and
+resource management.
+
+The dlb device level sysfs files are created during driver probe and is located
+at /sys/class/dlb/dlb<N>/device, where N is the zero-based device ID. The
+configfs directories/files can be created by user applications at
+/sys/kernel/config/dlb/dlb<N> using 'mkdir'. For example, 'mkdir domain0' will
+create a /domain0 directory and associated files in the configfs. Within the
+domain directory, directories for queues and ports can be created. An example of
+a DLB configfs structure is shown in the following diagram.
+
+                              config
+                                |
+                               dlb
+                                |
+                        +------+------+------+---
+                        |      |      |      |
+                       dlb0   dlb1   dlb2   dlb3
+                        |
+                +-----------+--+--------+-------
+                |           |           |
+             domain0     domain1     domain2
+                |
+        +-------+-----+------------+---------------+------------+------------
+        |             |            |               |            |
+ num_ldb_queues     port0         port1   ...    queue0       queue1   ...
+ num_ldb_ports        |                            |
+ ...                is_ldb                   num_sequence_numbers
+ create             cq_depth                 num_qid_inflights
+ start              ...                      num_atomic_iflights
+                    enable                   ...
+                    create                   create
+
+
+To create a domain/queue/port in DLB, an application can configure the resources
+by writing to corresponding files, and then write '1' to the 'create' file to
+trigger the action in the driver.
+
+The driver also exports an mmap interface through port files, which are
+acquired through port configfs. This mmap interface is used to map
+a port's memory and MMIO window into the process's address space. Once the
+ports are mapped, applications may use 64-byte direct-store instructions such
+as movdir64b to enqueue the events for better performance.
+
+Reset
+=====
+
+The dlb driver currently supports scheduling domain reset.
+
+Scheduling domain reset occurs when an application stops using its domain.
+Specifically, when no more file references or memory mappings exist. At this
+time, the driver resets all the domain's resources (flushes its queues and
+ports) and puts them in their respective available-resource lists for later
+use.

@@ -17,6 +17,7 @@ fit into other categories.
    ad525x_dpot
    apds990x
    bh1770glc
+   dlb
    eeprom
    c2port
    dw-xdata-pcie

@@ -9335,6 +9335,13 @@ L:	linux-kernel@vger.kernel.org
 S:	Supported
 F:	arch/x86/include/asm/intel-family.h
 
+INTEL DYNAMIC LOAD BALANCER DRIVER
+M:	Mike Ximing Chen <mike.ximing.chen@intel.com>
+S:	Maintained
+F:	Documentation/ABI/testing/sysfs-driver-dlb
+F:	drivers/misc/dlb/
+F:	include/uapi/linux/dlb.h
+
 INTEL DRM DRIVERS (excluding Poulsbo, Moorestown and derivative chipsets)
 M:	Jani Nikula <jani.nikula@linux.intel.com>
 M:	Joonas Lahtinen <joonas.lahtinen@linux.intel.com>

@@ -487,4 +487,5 @@ source "drivers/misc/cardreader/Kconfig"
 source "drivers/misc/habanalabs/Kconfig"
 source "drivers/misc/uacce/Kconfig"
 source "drivers/misc/pvpanic/Kconfig"
+source "drivers/misc/dlb/Kconfig"
 endmenu

@@ -59,3 +59,4 @@ obj-$(CONFIG_UACCE)		+= uacce/
 obj-$(CONFIG_XILINX_SDFEC)	+= xilinx_sdfec.o
 obj-$(CONFIG_HISI_HIKEY_USB)	+= hisi_hikey_usb.o
 obj-$(CONFIG_HI6421V600_IRQ)	+= hi6421v600-irq.o
+obj-$(CONFIG_INTEL_DLB)		+= dlb/

@@ -0,0 +1,18 @@
+# SPDX-License-Identifier: GPL-2.0-only
+
+config INTEL_DLB
+	tristate "Intel Dynamic Load Balancer Driver"
+	depends on 64BIT && PCI && X86
+	help
+	  This driver supports the Intel Dynamic Load Balancer (DLB), a
+	  PCIe device (PCI ID 8086:27xx) that provides load-balanced,
+	  prioritized scheduling of core-to-core communication and improves
+	  DPDK Event Device library performance.
+
+	  The user-space interface is described in
+	  include/uapi/linux/dlb_user.h
+
+	  To compile this driver as a module, choose M here. The module
+	  will be called dlb.
+
+	  If unsure, select N.

@@ -0,0 +1,5 @@
+# SPDX-License-Identifier: GPL-2.0-only
+
+obj-$(CONFIG_INTEL_DLB) := dlb.o
+
+dlb-objs := dlb_main.o

@@ -0,0 +1,156 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright(C) 2016-2020 Intel Corporation. All rights reserved. */
+
+#include <linux/aer.h>
+#include <linux/cdev.h>
+#include <linux/delay.h>
+#include <linux/fs.h>
+#include <linux/init.h>
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/uaccess.h>
+
+#include "dlb_main.h"
+
+MODULE_LICENSE("GPL v2");
+MODULE_DESCRIPTION("Intel(R) Dynamic Load Balancer (DLB) Driver");
+
+static struct class *dlb_class;
+static dev_t dlb_devt;
+static DEFINE_IDR(dlb_ids);
+static DEFINE_MUTEX(dlb_ids_lock);
+
+/**********************************/
+/****** PCI driver callbacks ******/
+/**********************************/
+
+static int dlb_probe(struct pci_dev *pdev, const struct pci_device_id *pdev_id)
+{
+	struct dlb *dlb;
+	int ret;
+
+	dlb = devm_kzalloc(&pdev->dev, sizeof(*dlb), GFP_KERNEL);
+	if (!dlb)
+		return -ENOMEM;
+
+	pci_set_drvdata(pdev, dlb);
+
+	dlb->pdev = pdev;
+
+	mutex_lock(&dlb_ids_lock);
+	dlb->id = idr_alloc(&dlb_ids, (void *)dlb, 0, DLB_MAX_NUM_DEVICES - 1,
+			    GFP_KERNEL);
+	mutex_unlock(&dlb_ids_lock);
+
+	if (dlb->id < 0) {
+		dev_err(&pdev->dev, "device ID allocation failed\n");
+
+		ret = dlb->id;
+		goto alloc_id_fail;
+	}
+
+	ret = pcim_enable_device(pdev);
+	if (ret != 0) {
+		dev_err(&pdev->dev, "failed to enable: %d\n", ret);
+
+		goto pci_enable_device_fail;
+	}
+
+	ret = pcim_iomap_regions(pdev,
+				 (1U << DLB_CSR_BAR) | (1U << DLB_FUNC_BAR),
+				 "dlb");
+	if (ret != 0) {
+		dev_err(&pdev->dev, "failed to map: %d\n", ret);
+
+		goto pci_enable_device_fail;
+	}
+
+	pci_set_master(pdev);
+
+	ret = pci_enable_pcie_error_reporting(pdev);
+	if (ret != 0)
+		dev_info(&pdev->dev, "AER is not supported\n");
+
+	return 0;
+
+pci_enable_device_fail:
+	mutex_lock(&dlb_ids_lock);
+	idr_remove(&dlb_ids, dlb->id);
+	mutex_unlock(&dlb_ids_lock);
+alloc_id_fail:
+	return ret;
+}
+
+static void dlb_remove(struct pci_dev *pdev)
+{
+	struct dlb *dlb = pci_get_drvdata(pdev);
+
+	pci_disable_pcie_error_reporting(pdev);
+
+	mutex_lock(&dlb_ids_lock);
+	idr_remove(&dlb_ids, dlb->id);
+	mutex_unlock(&dlb_ids_lock);
+}
+
+static struct pci_device_id dlb_id_table[] = {
+	{ PCI_DEVICE_DATA(INTEL, DLB_PF, DLB_PF) },
+	{ 0 }
+};
+MODULE_DEVICE_TABLE(pci, dlb_id_table);
+
+static struct pci_driver dlb_pci_driver = {
+	.name		 = "dlb",
+	.id_table	 = dlb_id_table,
+	.probe		 = dlb_probe,
+	.remove		 = dlb_remove,
+};
+
+static int __init dlb_init_module(void)
+{
+	int err;
+
+	dlb_class = class_create(THIS_MODULE, "dlb");
+
+	if (IS_ERR(dlb_class)) {
+		pr_err("dlb: class_create() returned %ld\n",
+		       PTR_ERR(dlb_class));
+
+		return PTR_ERR(dlb_class);
+	}
+
+	err = alloc_chrdev_region(&dlb_devt, 0, DLB_MAX_NUM_DEVICES, "dlb");
+
+	if (err < 0) {
+		pr_err("dlb: alloc_chrdev_region() returned %d\n", err);
+
+		goto alloc_chrdev_fail;
+	}
+
+	err = pci_register_driver(&dlb_pci_driver);
+	if (err < 0) {
+		pr_err("dlb: pci_register_driver() returned %d\n", err);
+
+		goto pci_register_fail;
+	}
+
+	return 0;
+
+pci_register_fail:
+	unregister_chrdev_region(dlb_devt, DLB_MAX_NUM_DEVICES);
+alloc_chrdev_fail:
+	class_destroy(dlb_class);
+
+	return err;
+}
+
+static void __exit dlb_exit_module(void)
+{
+	pci_unregister_driver(&dlb_pci_driver);
+
+	unregister_chrdev_region(dlb_devt, DLB_MAX_NUM_DEVICES);
+
+	class_destroy(dlb_class);
+}
+
+module_init(dlb_init_module);
+module_exit(dlb_exit_module);

@@ -0,0 +1,64 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright(C) 2016-2020 Intel Corporation. All rights reserved. */
+
+#ifndef __DLB_MAIN_H
+#define __DLB_MAIN_H
+
+#include <linux/cdev.h>
+#include <linux/device.h>
+#include <linux/ktime.h>
+#include <linux/list.h>
+#include <linux/mutex.h>
+#include <linux/pci.h>
+#include <linux/types.h>
+
+/*
+ * Hardware related #defines and data structures.
+ *
+ */
+#define DLB_MAX_NUM_VDEVS			16
+#define DLB_MAX_NUM_DOMAINS			32
+#define DLB_MAX_NUM_LDB_QUEUES			32 /* LDB == load-balanced */
+#define DLB_MAX_NUM_DIR_QUEUES			64 /* DIR == directed */
+#define DLB_MAX_NUM_LDB_PORTS			64
+#define DLB_MAX_NUM_DIR_PORTS			DLB_MAX_NUM_DIR_QUEUES
+#define DLB_MAX_NUM_LDB_CREDITS			8192
+#define DLB_MAX_NUM_DIR_CREDITS			2048
+#define DLB_MAX_NUM_HIST_LIST_ENTRIES		2048
+#define DLB_MAX_NUM_AQED_ENTRIES		2048
+#define DLB_MAX_NUM_QIDS_PER_LDB_CQ		8
+#define DLB_MAX_NUM_SEQUENCE_NUMBER_GROUPS	2
+#define DLB_MAX_NUM_SEQUENCE_NUMBER_MODES	5
+#define DLB_QID_PRIORITIES			8
+#define DLB_NUM_ARB_WEIGHTS			8
+#define DLB_MAX_WEIGHT				255
+#define DLB_NUM_COS_DOMAINS			4
+#define DLB_MAX_CQ_COMP_CHECK_LOOPS		409600
+#define DLB_MAX_QID_EMPTY_CHECK_LOOPS		(32 * 64 * 1024 * (800 / 30))
+#define DLB_HZ					800000000
+#define DLB_FUNC_BAR				0
+#define DLB_CSR_BAR				2
+
+#define PCI_DEVICE_ID_INTEL_DLB_PF		0x2710
+
+/*
+ * The dlb driver uses a different minor number for each device file, of which
+ * there are:
+ * - 33 per device (PF or VF/VDEV): 1 for the device, 32 for scheduling domains
+ * - Up to 17 devices per PF: 1 PF and up to 16 VFs/VDEVs
+ * - Up to 16 PFs per system
+ */
+#define DLB_MAX_NUM_PFS	  16
+#define DLB_NUM_FUNCS_PER_DEVICE (1 + DLB_MAX_NUM_VDEVS)
+#define DLB_MAX_NUM_DEVICES	 (DLB_MAX_NUM_PFS * DLB_NUM_FUNCS_PER_DEVICE)
+
+enum dlb_device_type {
+	DLB_PF,
+};
+
+struct dlb {
+	struct pci_dev *pdev;
+	int id;
+};
+
+#endif /* __DLB_MAIN_H */

@@ -3,3 +3,4 @@
 obj-$(CONFIG_INTEL_DLB) := dlb.o
 
 dlb-objs := dlb_main.o
+dlb-objs += dlb_pf_ops.o

@@ -16,10 +16,38 @@ MODULE_LICENSE("GPL v2");
 MODULE_DESCRIPTION("Intel(R) Dynamic Load Balancer (DLB) Driver");
 
 static struct class *dlb_class;
+static struct cdev dlb_cdev;
 static dev_t dlb_devt;
 static DEFINE_IDR(dlb_ids);
 static DEFINE_MUTEX(dlb_ids_lock);
 
+static int dlb_device_create(struct dlb *dlb, struct pci_dev *pdev)
+{
+	/*
+	 * Create a new device in order to create a /dev/dlb node. This device
+	 * is a child of the DLB PCI device.
+	 */
+	dlb->dev_number = MKDEV(MAJOR(dlb_devt), dlb->id);
+	dlb->dev = device_create(dlb_class, &pdev->dev, dlb->dev_number, dlb,
+				 "dlb%d", dlb->id);
+	if (IS_ERR(dlb->dev)) {
+		dev_err(dlb->dev, "device_create() returned %ld\n",
+			PTR_ERR(dlb->dev));
+
+		return PTR_ERR(dlb->dev);
+	}
+
+	return 0;
+}
+
+/********************************/
+/****** Char dev callbacks ******/
+/********************************/
+
+static const struct file_operations dlb_fops = {
+	.owner   = THIS_MODULE,
+};
+
 /**********************************/
 /****** PCI driver callbacks ******/
 /**********************************/

@@ -71,8 +99,24 @@ static int dlb_probe(struct pci_dev *pdev, const struct pci_device_id *pdev_id)
 	if (ret != 0)
 		dev_info(&pdev->dev, "AER is not supported\n");
 
+	ret = dlb_pf_map_pci_bar_space(dlb, pdev);
+	if (ret)
+		goto map_pci_bar_fail;
+
+	ret = dlb_device_create(dlb, pdev);
+	if (ret)
+		goto map_pci_bar_fail;
+
+	ret = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(64));
+	if (ret)
+		goto dma_set_mask_fail;
+
 	return 0;
 
+dma_set_mask_fail:
+	device_destroy(dlb_class, dlb->dev_number);
+map_pci_bar_fail:
+	pci_disable_pcie_error_reporting(pdev);
 pci_enable_device_fail:
 	mutex_lock(&dlb_ids_lock);
 	idr_remove(&dlb_ids, dlb->id);

@@ -85,6 +129,8 @@ static void dlb_remove(struct pci_dev *pdev)
 {
 	struct dlb *dlb = pci_get_drvdata(pdev);
 
+	device_destroy(dlb_class, dlb->dev_number);
+
 	pci_disable_pcie_error_reporting(pdev);
 
 	mutex_lock(&dlb_ids_lock);

@@ -107,6 +153,7 @@ static struct pci_driver dlb_pci_driver = {
 
 static int __init dlb_init_module(void)
 {
+	int dlb_major;
 	int err;
 
 	dlb_class = class_create(THIS_MODULE, "dlb");

@@ -126,6 +173,12 @@ static int __init dlb_init_module(void)
 		goto alloc_chrdev_fail;
 	}
 
+	dlb_major = MAJOR(dlb_devt);
+	cdev_init(&dlb_cdev, &dlb_fops);
+	err = cdev_add(&dlb_cdev, MKDEV(dlb_major, 0), DLB_MAX_NUM_DEVICES);
+	if (err)
+		goto cdev_add_fail;
+
 	err = pci_register_driver(&dlb_pci_driver);
 	if (err < 0) {
 		pr_err("dlb: pci_register_driver() returned %d\n", err);

@@ -136,6 +189,8 @@ static int __init dlb_init_module(void)
 	return 0;
 
 pci_register_fail:
+	cdev_del(&dlb_cdev);
+cdev_add_fail:
 	unregister_chrdev_region(dlb_devt, DLB_MAX_NUM_DEVICES);
 alloc_chrdev_fail:
 	class_destroy(dlb_class);

@@ -147,6 +202,8 @@ static void __exit dlb_exit_module(void)
 {
 	pci_unregister_driver(&dlb_pci_driver);
 
+	cdev_del(&dlb_cdev);
+
 	unregister_chrdev_region(dlb_devt, DLB_MAX_NUM_DEVICES);
 
 	class_destroy(dlb_class);

@@ -16,6 +16,7 @@
  * Hardware related #defines and data structures.
  *
  */
+
 #define DLB_MAX_NUM_VDEVS			16
 #define DLB_MAX_NUM_DOMAINS			32
 #define DLB_MAX_NUM_LDB_QUEUES			32 /* LDB == load-balanced */

@@ -41,6 +42,15 @@
 
 #define PCI_DEVICE_ID_INTEL_DLB_PF		0x2710
 
+struct dlb_hw {
+	/* BAR 0 address */
+	void __iomem *csr_kva;
+	unsigned long csr_phys_addr;
+	/* BAR 2 address */
+	void __iomem *func_kva;
+	unsigned long func_phys_addr;
+};
+
 /*
  * The dlb driver uses a different minor number for each device file, of which
  * there are:

@@ -56,9 +66,18 @@ enum dlb_device_type {
 	DLB_PF,
 };
 
+struct dlb;
+
+int dlb_pf_map_pci_bar_space(struct dlb *dlb, struct pci_dev *pdev);
+void dlb_pf_unmap_pci_bar_space(struct dlb *dlb, struct pci_dev *pdev);
+
 struct dlb {
 	struct pci_dev *pdev;
+	struct dlb_hw hw;
+	struct device *dev;
+	enum dlb_device_type type;
 	int id;
+	dev_t dev_number;
 };
 
 #endif /* __DLB_MAIN_H */

@@ -0,0 +1,33 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright(C) 2016-2020 Intel Corporation. All rights reserved. */
+
+#include "dlb_main.h"
+
+/********************************/
+/****** PCI BAR management ******/
+/********************************/
+
+int dlb_pf_map_pci_bar_space(struct dlb *dlb, struct pci_dev *pdev)
+{
+	dlb->hw.func_kva = pcim_iomap_table(pdev)[DLB_FUNC_BAR];
+	dlb->hw.func_phys_addr = pci_resource_start(pdev, DLB_FUNC_BAR);
+
+	if (!dlb->hw.func_kva) {
+		dev_err(&pdev->dev, "Cannot iomap BAR 0 (size %llu)\n",
+			pci_resource_len(pdev, 0));
+
+		return -EIO;
+	}
+
+	dlb->hw.csr_kva = pcim_iomap_table(pdev)[DLB_CSR_BAR];
+	dlb->hw.csr_phys_addr = pci_resource_start(pdev, DLB_CSR_BAR);
+
+	if (!dlb->hw.csr_kva) {
+		dev_err(&pdev->dev, "Cannot iomap BAR 2 (size %llu)\n",
+			pci_resource_len(pdev, 2));
+
+		return -EIO;
+	}
+
+	return 0;
+}

@@ -3,4 +3,4 @@
 obj-$(CONFIG_INTEL_DLB) := dlb.o
 
 dlb-objs := dlb_main.o
-dlb-objs += dlb_pf_ops.o
+dlb-objs += dlb_pf_ops.o dlb_resource.o

@@ -21,6 +21,23 @@ static dev_t dlb_devt;
 static DEFINE_IDR(dlb_ids);
 static DEFINE_MUTEX(dlb_ids_lock);
 
+static int dlb_reset_device(struct pci_dev *pdev)
+{
+	int ret;
+
+	ret = pci_save_state(pdev);
+	if (ret)
+		return ret;
+
+	ret = __pci_reset_function_locked(pdev);
+	if (ret)
+		return ret;
+
+	pci_restore_state(pdev);
+
+	return 0;
+}
+
 static int dlb_device_create(struct dlb *dlb, struct pci_dev *pdev)
 {
 	/*

@@ -111,8 +128,35 @@ static int dlb_probe(struct pci_dev *pdev, const struct pci_device_id *pdev_id)
 	if (ret)
 		goto dma_set_mask_fail;
 
+	/*
+	 * PM enable must be done before any other MMIO accesses, and this
+	 * setting is persistent across device reset.
+	 */
+	dlb_pf_enable_pm(dlb);
+
+	ret = dlb_pf_wait_for_device_ready(dlb, pdev);
+	if (ret)
+		goto wait_for_device_ready_fail;
+
+	ret = dlb_reset_device(pdev);
+	if (ret)
+		goto dlb_reset_fail;
+
+	ret = dlb_resource_init(&dlb->hw);
+	if (ret)
+		goto resource_init_fail;
+
+	ret = dlb_pf_init_driver_state(dlb);
+	if (ret)
+		goto init_driver_state_fail;
+
 	return 0;
 
+init_driver_state_fail:
+	dlb_resource_free(&dlb->hw);
+resource_init_fail:
+dlb_reset_fail:
+wait_for_device_ready_fail:
 dma_set_mask_fail:
 	device_destroy(dlb_class, dlb->dev_number);
 map_pci_bar_fail:

@@ -129,6 +173,8 @@ static void dlb_remove(struct pci_dev *pdev)
 {
 	struct dlb *dlb = pci_get_drvdata(pdev);
 
+	dlb_resource_free(&dlb->hw);
+
 	device_destroy(dlb_class, dlb->dev_number);
 
 	pci_disable_pcie_error_reporting(pdev);

@@ -11,11 +11,23 @@
 #include <linux/mutex.h>
 #include <linux/pci.h>
 #include <linux/types.h>
+#include <linux/bitfield.h>
 
 /*
  * Hardware related #defines and data structures.
  *
  */
+/* Read/write register 'reg' in the CSR BAR space */
+#define DLB_CSR_REG_ADDR(a, reg)   ((a)->csr_kva + (reg))
+#define DLB_CSR_RD(hw, reg)	    ioread32(DLB_CSR_REG_ADDR((hw), (reg)))
+#define DLB_CSR_WR(hw, reg, value) iowrite32((value), \
+					      DLB_CSR_REG_ADDR((hw), (reg)))
+
+/* Read/write register 'reg' in the func BAR space */
+#define DLB_FUNC_REG_ADDR(a, reg)   ((a)->func_kva + (reg))
+#define DLB_FUNC_RD(hw, reg)	     ioread32(DLB_FUNC_REG_ADDR((hw), (reg)))
+#define DLB_FUNC_WR(hw, reg, value) iowrite32((value), \
+					       DLB_FUNC_REG_ADDR((hw), (reg)))
 
 #define DLB_MAX_NUM_VDEVS			16
 #define DLB_MAX_NUM_DOMAINS			32

@@ -42,6 +54,159 @@
 
 #define PCI_DEVICE_ID_INTEL_DLB_PF		0x2710
 
+struct dlb_ldb_queue {
+	struct list_head domain_list;
+	struct list_head func_list;
+	u32 id;
+	u32 domain_id;
+	u32 num_qid_inflights;
+	u32 aqed_limit;
+	u32 sn_group; /* sn == sequence number */
+	u32 sn_slot;
+	u32 num_mappings;
+	u8 sn_cfg_valid;
+	u8 num_pending_additions;
+	u8 owned;
+	u8 configured;
+};
+
+/*
+ * Directed ports and queues are paired by nature, so the driver tracks them
+ * with a single data structure.
+ */
+struct dlb_dir_pq_pair {
+	struct list_head domain_list;
+	struct list_head func_list;
+	u32 id;
+	u32 domain_id;
+	u32 ref_cnt;
+	u8 init_tkn_cnt;
+	u8 queue_configured;
+	u8 port_configured;
+	u8 owned;
+	u8 enabled;
+};
+
+enum dlb_qid_map_state {
+	/* The slot doesn't contain a valid queue mapping */
+	DLB_QUEUE_UNMAPPED,
+	/* The slot contains a valid queue mapping */
+	DLB_QUEUE_MAPPED,
+	/* The driver is mapping a queue into this slot */
+	DLB_QUEUE_MAP_IN_PROG,
+	/* The driver is unmapping a queue from this slot */
+	DLB_QUEUE_UNMAP_IN_PROG,
+	/*
+	 * The driver is unmapping a queue from this slot, and once complete
+	 * will replace it with another mapping.
+	 */
+	DLB_QUEUE_UNMAP_IN_PROG_PENDING_MAP,
+};
+
+struct dlb_ldb_port_qid_map {
+	enum dlb_qid_map_state state;
+	u16 qid;
+	u16 pending_qid;
+	u8 priority;
+	u8 pending_priority;
+};
+
+struct dlb_ldb_port {
+	struct list_head domain_list;
+	struct list_head func_list;
+	u32 id;
+	u32 domain_id;
+	/* The qid_map represents the hardware QID mapping state. */
+	struct dlb_ldb_port_qid_map qid_map[DLB_MAX_NUM_QIDS_PER_LDB_CQ];
+	u32 hist_list_entry_base;
+	u32 hist_list_entry_limit;
+	u32 ref_cnt;
+	u8 init_tkn_cnt;
+	u8 num_pending_removals;
+	u8 num_mappings;
+	u8 owned;
+	u8 enabled;
+	u8 configured;
+};
+
+struct dlb_sn_group {
+	u32 mode;
+	u32 sequence_numbers_per_queue;
+	u32 slot_use_bitmap;
+	u32 id;
+};
+
+/*
+ * Scheduling domain level resource data structure.
+ *
+ */
+struct dlb_hw_domain {
+	struct dlb_function_resources *parent_func;
+	struct list_head func_list;
+	struct list_head used_ldb_queues;
+	struct list_head used_ldb_ports[DLB_NUM_COS_DOMAINS];
+	struct list_head used_dir_pq_pairs;
+	struct list_head avail_ldb_queues;
+	struct list_head avail_ldb_ports[DLB_NUM_COS_DOMAINS];
+	struct list_head avail_dir_pq_pairs;
+	u32 total_hist_list_entries;
+	u32 avail_hist_list_entries;
+	u32 hist_list_entry_base;
+	u32 hist_list_entry_offset;
+	u32 num_ldb_credits;
+	u32 num_dir_credits;
+	u32 num_avail_aqed_entries;
+	u32 num_used_aqed_entries;
+	u32 id;
+	int num_pending_removals;
+	int num_pending_additions;
+	u8 configured;
+	u8 started;
+};
+
+/*
+ * Device function (either PF or VF) level resource data structure.
+ *
+ */
+struct dlb_function_resources {
+	struct list_head avail_domains;
+	struct list_head used_domains;
+	struct list_head avail_ldb_queues;
+	struct list_head avail_ldb_ports[DLB_NUM_COS_DOMAINS];
+	struct list_head avail_dir_pq_pairs;
+	struct dlb_bitmap *avail_hist_list_entries;
+	u32 num_avail_domains;
+	u32 num_avail_ldb_queues;
+	u32 num_avail_ldb_ports[DLB_NUM_COS_DOMAINS];
+	u32 num_avail_dir_pq_pairs;
+	u32 num_avail_qed_entries;
+	u32 num_avail_dqed_entries;
+	u32 num_avail_aqed_entries;
+	u8 locked; /* (VDEV only) */
+};
+
+/*
+ * After initialization, each resource in dlb_hw_resources is located in one
+ * of the following lists:
+ * -- The PF's available resources list. These are unconfigured resources owned
+ *	by the PF and not allocated to a dlb scheduling domain.
+ * -- A VDEV's available resources list. These are VDEV-owned unconfigured
+ *	resources not allocated to a dlb scheduling domain.
+ * -- A domain's available resources list. These are domain-owned unconfigured
+ *	resources.
+ * -- A domain's used resources list. These are domain-owned configured
+ *	resources.
+ *
+ * A resource moves to a new list when a VDEV or domain is created or destroyed,
+ * or when the resource is configured.
+ */
+struct dlb_hw_resources {
+	struct dlb_ldb_queue ldb_queues[DLB_MAX_NUM_LDB_QUEUES];
+	struct dlb_ldb_port ldb_ports[DLB_MAX_NUM_LDB_PORTS];
+	struct dlb_dir_pq_pair dir_pq_pairs[DLB_MAX_NUM_DIR_PORTS];
+	struct dlb_sn_group sn_groups[DLB_MAX_NUM_SEQUENCE_NUMBER_GROUPS];
+};
+
 struct dlb_hw {
 	/* BAR 0 address */
 	void __iomem *csr_kva;

@@ -49,6 +214,13 @@ struct dlb_hw {
 	/* BAR 2 address */
 	void __iomem *func_kva;
 	unsigned long func_phys_addr;
+
+	/* Resource tracking */
+	struct dlb_hw_resources rsrcs;
+	struct dlb_function_resources pf;
+	struct dlb_function_resources vdev[DLB_MAX_NUM_VDEVS];
+	struct dlb_hw_domain domains[DLB_MAX_NUM_DOMAINS];
+	u8 cos_reservation[DLB_NUM_COS_DOMAINS];
 };
 
 /*

@@ -70,14 +242,91 @@ struct dlb;
 
 int dlb_pf_map_pci_bar_space(struct dlb *dlb, struct pci_dev *pdev);
 void dlb_pf_unmap_pci_bar_space(struct dlb *dlb, struct pci_dev *pdev);
+int dlb_pf_init_driver_state(struct dlb *dlb);
+void dlb_pf_enable_pm(struct dlb *dlb);
+int dlb_pf_wait_for_device_ready(struct dlb *dlb, struct pci_dev *pdev);
 
 struct dlb {
 	struct pci_dev *pdev;
 	struct dlb_hw hw;
 	struct device *dev;
+	/*
+	 * The resource mutex serializes access to driver data structures and
+	 * hardware registers.
+	 */
+	struct mutex resource_mutex;
 	enum dlb_device_type type;
 	int id;
 	dev_t dev_number;
 };
 
+/*************************/
+/*** Bitmap operations ***/
+/*************************/
+struct dlb_bitmap {
+	unsigned long *map;
+	unsigned int len;
+};
+
+/**
+ * dlb_bitmap_alloc() - alloc a bitmap data structure
+ * @bitmap: pointer to dlb_bitmap structure pointer.
+ * @len: number of entries in the bitmap.
+ *
+ * This function allocates a bitmap and initializes it with length @len. All
+ * entries are initially zero.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or len is 0.
+ * ENOMEM - could not allocate memory for the bitmap data structure.
+ */
+static inline int dlb_bitmap_alloc(struct dlb_bitmap **bitmap,
+				   unsigned int len)
+{
+	struct dlb_bitmap *bm;
+
+	if (!bitmap || len == 0)
+		return -EINVAL;
+
+	bm = kzalloc(sizeof(*bm), GFP_KERNEL);
+	if (!bm)
+		return -ENOMEM;
+
+	bm->map = bitmap_zalloc(len, GFP_KERNEL);
+	if (!bm->map) {
+		kfree(bm);
+		return -ENOMEM;
+	}
+
+	bm->len = len;
+
+	*bitmap = bm;
+
+	return 0;
+}
+
+/**
+ * dlb_bitmap_free() - free a previously allocated bitmap data structure
+ * @bitmap: pointer to dlb_bitmap structure.
+ *
+ * This function frees a bitmap that was allocated with dlb_bitmap_alloc().
+ */
+static inline void dlb_bitmap_free(struct dlb_bitmap *bitmap)
+{
+	if (!bitmap)
+		return;
+
+	bitmap_free(bitmap->map);
+
+	kfree(bitmap);
+}
+
+/* Prototypes for dlb_resource.c */
+int dlb_resource_init(struct dlb_hw *hw);
+void dlb_resource_free(struct dlb_hw *hw);
+void dlb_clr_pmcsr_disable(struct dlb_hw *hw);
+
 #endif /* __DLB_MAIN_H */

@@ -1,7 +1,10 @@
 // SPDX-License-Identifier: GPL-2.0-only
 /* Copyright(C) 2016-2020 Intel Corporation. All rights reserved. */
 
+#include <linux/delay.h>
+
 #include "dlb_main.h"
+#include "dlb_regs.h"
 
 /********************************/
 /****** PCI BAR management ******/

@@ -31,3 +34,56 @@ int dlb_pf_map_pci_bar_space(struct dlb *dlb, struct pci_dev *pdev)
 
 	return 0;
 }
+
+/*******************************/
+/****** Driver management ******/
+/*******************************/
+
+int dlb_pf_init_driver_state(struct dlb *dlb)
+{
+	mutex_init(&dlb->resource_mutex);
+
+	return 0;
+}
+
+void dlb_pf_enable_pm(struct dlb *dlb)
+{
+	/*
+	 * Clear the power-management-disable register to power on the bulk of
+	 * the device's hardware.
+	 */
+	dlb_clr_pmcsr_disable(&dlb->hw);
+}
+
+#define DLB_READY_RETRY_LIMIT 1000
+int dlb_pf_wait_for_device_ready(struct dlb *dlb, struct pci_dev *pdev)
+{
+	u32 retries = DLB_READY_RETRY_LIMIT;
+
+	/* Allow at least 1s for the device to become active after power-on */
+	do {
+		u32 idle, pm_st, addr;
+
+		addr = CM_CFG_PM_STATUS;
+
+		pm_st = DLB_CSR_RD(&dlb->hw, addr);
+
+		addr = CM_CFG_DIAGNOSTIC_IDLE_STATUS;
+
+		idle = DLB_CSR_RD(&dlb->hw, addr);
+
+		if (FIELD_GET(CM_CFG_PM_STATUS_PMSM, pm_st) == 1 &&
+		    FIELD_GET(CM_CFG_DIAGNOSTIC_IDLE_STATUS_DLB_FUNC_IDLE, idle)
+		    == 1)
+			break;
+
+		usleep_range(1000, 2000);
+	} while (--retries);
+
+	if (!retries) {
+		dev_err(&pdev->dev, "Device idle test failed\n");
+		return -EIO;
+	}
+
+	return 0;
+}

@@ -0,0 +1,119 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright(C) 2016-2020 Intel Corporation. All rights reserved. */
+
+#ifndef __DLB_REGS_H
+#define __DLB_REGS_H
+
+#include <linux/types.h>
+
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS 0xb4000004
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_RST 0x9d0fffff
+
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_CHP_PIPEIDLE		0x00000001
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_ROP_PIPEIDLE		0x00000002
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_LSP_PIPEIDLE		0x00000004
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_NALB_PIPEIDLE		0x00000008
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_AP_PIPEIDLE		0x00000010
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_DP_PIPEIDLE		0x00000020
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_QED_PIPEIDLE		0x00000040
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_DQED_PIPEIDLE		0x00000080
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_AQED_PIPEIDLE		0x00000100
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_SYS_PIPEIDLE		0x00000200
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_CHP_UNIT_IDLE		0x00000400
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_ROP_UNIT_IDLE		0x00000800
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_LSP_UNIT_IDLE		0x00001000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_NALB_UNIT_IDLE		0x00002000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_AP_UNIT_IDLE		0x00004000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_DP_UNIT_IDLE		0x00008000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_QED_UNIT_IDLE		0x00010000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_DQED_UNIT_IDLE		0x00020000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_AQED_UNIT_IDLE		0x00040000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_SYS_UNIT_IDLE		0x00080000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_RSVD1			0x00F00000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_MSTR_CFG_RING_IDLE	0x01000000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_MSTR_CFG_MSTR_IDLE	0x02000000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_MSTR_FLR_CLKREQ_B		0x04000000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_MSTR_PROC_IDLE		0x08000000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_MSTR_PROC_IDLE_MASKED	0x10000000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_RSVD0			0x60000000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_DLB_FUNC_IDLE		0x80000000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_CHP_PIPEIDLE_LOC		0
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_ROP_PIPEIDLE_LOC		1
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_LSP_PIPEIDLE_LOC		2
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_NALB_PIPEIDLE_LOC		3
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_AP_PIPEIDLE_LOC		4
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_DP_PIPEIDLE_LOC		5
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_QED_PIPEIDLE_LOC		6
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_DQED_PIPEIDLE_LOC		7
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_AQED_PIPEIDLE_LOC		8
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_SYS_PIPEIDLE_LOC		9
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_CHP_UNIT_IDLE_LOC		10
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_ROP_UNIT_IDLE_LOC		11
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_LSP_UNIT_IDLE_LOC		12
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_NALB_UNIT_IDLE_LOC	13
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_AP_UNIT_IDLE_LOC		14
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_DP_UNIT_IDLE_LOC		15
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_QED_UNIT_IDLE_LOC		16
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_DQED_UNIT_IDLE_LOC	17
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_AQED_UNIT_IDLE_LOC	18
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_SYS_UNIT_IDLE_LOC		19
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_RSVD1_LOC			20
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_MSTR_CFG_RING_IDLE_LOC	24
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_MSTR_CFG_MSTR_IDLE_LOC	25
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_MSTR_FLR_CLKREQ_B_LOC	26
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_MSTR_PROC_IDLE_LOC	27
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_MSTR_PROC_IDLE_MASKED_LOC	28
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_RSVD0_LOC			29
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_DLB_FUNC_IDLE_LOC		31
+
+#define CM_CFG_PM_STATUS 0xb4000014
+#define CM_CFG_PM_STATUS_RST 0x100403e
+
+#define CM_CFG_PM_STATUS_PROCHOT		0x00000001
+#define CM_CFG_PM_STATUS_PGCB_DLB_IDLE		0x00000002
+#define CM_CFG_PM_STATUS_PGCB_DLB_PG_RDY_ACK_B	0x00000004
+#define CM_CFG_PM_STATUS_PMSM_PGCB_REQ_B	0x00000008
+#define CM_CFG_PM_STATUS_PGBC_PMC_PG_REQ_B	0x00000010
+#define CM_CFG_PM_STATUS_PMC_PGCB_PG_ACK_B	0x00000020
+#define CM_CFG_PM_STATUS_PMC_PGCB_FET_EN_B	0x00000040
+#define CM_CFG_PM_STATUS_PGCB_FET_EN_B		0x00000080
+#define CM_CFG_PM_STATUS_RSVZ0			0x00000100
+#define CM_CFG_PM_STATUS_RSVZ1			0x00000200
+#define CM_CFG_PM_STATUS_FUSE_FORCE_ON		0x00000400
+#define CM_CFG_PM_STATUS_FUSE_PROC_DISABLE	0x00000800
+#define CM_CFG_PM_STATUS_RSVZ2			0x00001000
+#define CM_CFG_PM_STATUS_RSVZ3			0x00002000
+#define CM_CFG_PM_STATUS_PM_FSM_D0TOD3_OK	0x00004000
+#define CM_CFG_PM_STATUS_PM_FSM_D3TOD0_OK	0x00008000
+#define CM_CFG_PM_STATUS_DLB_IN_D3		0x00010000
+#define CM_CFG_PM_STATUS_RSVZ4			0x00FE0000
+#define CM_CFG_PM_STATUS_PMSM			0xFF000000
+#define CM_CFG_PM_STATUS_PROCHOT_LOC			0
+#define CM_CFG_PM_STATUS_PGCB_DLB_IDLE_LOC		1
+#define CM_CFG_PM_STATUS_PGCB_DLB_PG_RDY_ACK_B_LOC	2
+#define CM_CFG_PM_STATUS_PMSM_PGCB_REQ_B_LOC		3
+#define CM_CFG_PM_STATUS_PGBC_PMC_PG_REQ_B_LOC		4
+#define CM_CFG_PM_STATUS_PMC_PGCB_PG_ACK_B_LOC		5
+#define CM_CFG_PM_STATUS_PMC_PGCB_FET_EN_B_LOC		6
+#define CM_CFG_PM_STATUS_PGCB_FET_EN_B_LOC		7
+#define CM_CFG_PM_STATUS_RSVZ0_LOC			8
+#define CM_CFG_PM_STATUS_RSVZ1_LOC			9
+#define CM_CFG_PM_STATUS_FUSE_FORCE_ON_LOC		10
+#define CM_CFG_PM_STATUS_FUSE_PROC_DISABLE_LOC		11
+#define CM_CFG_PM_STATUS_RSVZ2_LOC			12
+#define CM_CFG_PM_STATUS_RSVZ3_LOC			13
+#define CM_CFG_PM_STATUS_PM_FSM_D0TOD3_OK_LOC		14
+#define CM_CFG_PM_STATUS_PM_FSM_D3TOD0_OK_LOC		15
+#define CM_CFG_PM_STATUS_DLB_IN_D3_LOC			16
+#define CM_CFG_PM_STATUS_RSVZ4_LOC			17
+#define CM_CFG_PM_STATUS_PMSM_LOC			24
+
+#define CM_CFG_PM_PMCSR_DISABLE 0xb4000018
+#define CM_CFG_PM_PMCSR_DISABLE_RST 0x1
+
+#define CM_CFG_PM_PMCSR_DISABLE_DISABLE	0x00000001
+#define CM_CFG_PM_PMCSR_DISABLE_RSVZ0	0xFFFFFFFE
+#define CM_CFG_PM_PMCSR_DISABLE_DISABLE_LOC	0
+#define CM_CFG_PM_PMCSR_DISABLE_RSVZ0_LOC	1
+
+#endif /* __DLB_REGS_H */

@@ -0,0 +1,210 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright(C) 2016-2020 Intel Corporation. All rights reserved. */
+
+#include "dlb_regs.h"
+#include "dlb_main.h"
+
+static void dlb_init_fn_rsrc_lists(struct dlb_function_resources *rsrc)
+{
+	int i;
+
+	INIT_LIST_HEAD(&rsrc->avail_domains);
+	INIT_LIST_HEAD(&rsrc->used_domains);
+	INIT_LIST_HEAD(&rsrc->avail_ldb_queues);
+	INIT_LIST_HEAD(&rsrc->avail_dir_pq_pairs);
+
+	for (i = 0; i < DLB_NUM_COS_DOMAINS; i++)
+		INIT_LIST_HEAD(&rsrc->avail_ldb_ports[i]);
+}
+
+static void dlb_init_domain_rsrc_lists(struct dlb_hw_domain *domain)
+{
+	int i;
+
+	INIT_LIST_HEAD(&domain->used_ldb_queues);
+	INIT_LIST_HEAD(&domain->used_dir_pq_pairs);
+	INIT_LIST_HEAD(&domain->avail_ldb_queues);
+	INIT_LIST_HEAD(&domain->avail_dir_pq_pairs);
+
+	for (i = 0; i < DLB_NUM_COS_DOMAINS; i++)
+		INIT_LIST_HEAD(&domain->used_ldb_ports[i]);
+	for (i = 0; i < DLB_NUM_COS_DOMAINS; i++)
+		INIT_LIST_HEAD(&domain->avail_ldb_ports[i]);
+}
+
+/**
+ * dlb_resource_free() - free device state memory
+ * @hw: dlb_hw handle for a particular device.
+ *
+ * This function frees software state pointed to by dlb_hw. This function
+ * should be called when resetting the device or unloading the driver.
+ */
+void dlb_resource_free(struct dlb_hw *hw)
+{
+	int i;
+
+	if (hw->pf.avail_hist_list_entries)
+		dlb_bitmap_free(hw->pf.avail_hist_list_entries);
+
+	for (i = 0; i < DLB_MAX_NUM_VDEVS; i++) {
+		if (hw->vdev[i].avail_hist_list_entries)
+			dlb_bitmap_free(hw->vdev[i].avail_hist_list_entries);
+	}
+}
+
+/**
+ * dlb_resource_init() - initialize the device
+ * @hw: pointer to struct dlb_hw.
+ *
+ * This function initializes the device's software state (pointed to by the hw
+ * argument) and programs global scheduling QoS registers. This function should
+ * be called during driver initialization, and the dlb_hw structure should
+ * be zero-initialized before calling the function.
+ *
+ * The dlb_hw struct must be unique per DLB 2.0 device and persist until the
+ * device is reset.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ */
+int dlb_resource_init(struct dlb_hw *hw)
+{
+	struct dlb_bitmap *map;
+	struct list_head *list;
+	unsigned int i;
+	int ret;
+
+	/*
+	 * For optimal load-balancing, ports that map to one or more QIDs in
+	 * common should not be in numerical sequence. The port->QID mapping is
+	 * application dependent, but the driver interleaves port IDs as much
+	 * as possible to reduce the likelihood of sequential ports mapping to
+	 * the same QID(s). This initial allocation of port IDs maximizes the
+	 * average distance between an ID and its immediate neighbors (i.e.
+	 * the distance from 1 to 0 and to 2, the distance from 2 to 1 and to
+	 * 3, etc.).
+	 */
+	const u8 init_ldb_port_allocation[DLB_MAX_NUM_LDB_PORTS] = {
+		0,  7,  14,  5, 12,  3, 10,  1,  8, 15,  6, 13,  4, 11,  2,  9,
+		16, 23, 30, 21, 28, 19, 26, 17, 24, 31, 22, 29, 20, 27, 18, 25,
+		32, 39, 46, 37, 44, 35, 42, 33, 40, 47, 38, 45, 36, 43, 34, 41,
+		48, 55, 62, 53, 60, 51, 58, 49, 56, 63, 54, 61, 52, 59, 50, 57,
+	};
+
+	dlb_init_fn_rsrc_lists(&hw->pf);
+
+	for (i = 0; i < DLB_MAX_NUM_VDEVS; i++)
+		dlb_init_fn_rsrc_lists(&hw->vdev[i]);
+
+	for (i = 0; i < DLB_MAX_NUM_DOMAINS; i++) {
+		dlb_init_domain_rsrc_lists(&hw->domains[i]);
+		hw->domains[i].parent_func = &hw->pf;
+	}
+
+	/* Give all resources to the PF driver */
+	hw->pf.num_avail_domains = DLB_MAX_NUM_DOMAINS;
+	for (i = 0; i < hw->pf.num_avail_domains; i++) {
+		list = &hw->domains[i].func_list;
+
+		list_add(list, &hw->pf.avail_domains);
+	}
+
+	hw->pf.num_avail_ldb_queues = DLB_MAX_NUM_LDB_QUEUES;
+	for (i = 0; i < hw->pf.num_avail_ldb_queues; i++) {
+		list = &hw->rsrcs.ldb_queues[i].func_list;
+
+		list_add(list, &hw->pf.avail_ldb_queues);
+	}
+
+	for (i = 0; i < DLB_NUM_COS_DOMAINS; i++)
+		hw->pf.num_avail_ldb_ports[i] =
+			DLB_MAX_NUM_LDB_PORTS / DLB_NUM_COS_DOMAINS;
+
+	for (i = 0; i < DLB_MAX_NUM_LDB_PORTS; i++) {
+		int cos_id = i >> DLB_NUM_COS_DOMAINS;
+		struct dlb_ldb_port *port;
+
+		port = &hw->rsrcs.ldb_ports[init_ldb_port_allocation[i]];
+
+		list_add(&port->func_list, &hw->pf.avail_ldb_ports[cos_id]);
+	}
+
+	hw->pf.num_avail_dir_pq_pairs = DLB_MAX_NUM_DIR_PORTS;
+	for (i = 0; i < hw->pf.num_avail_dir_pq_pairs; i++) {
+		list = &hw->rsrcs.dir_pq_pairs[i].func_list;
+
+		list_add(list, &hw->pf.avail_dir_pq_pairs);
+	}
+
+	hw->pf.num_avail_qed_entries = DLB_MAX_NUM_LDB_CREDITS;
+	hw->pf.num_avail_dqed_entries = DLB_MAX_NUM_DIR_CREDITS;
+	hw->pf.num_avail_aqed_entries = DLB_MAX_NUM_AQED_ENTRIES;
+
+	ret = dlb_bitmap_alloc(&hw->pf.avail_hist_list_entries,
+			       DLB_MAX_NUM_HIST_LIST_ENTRIES);
+	if (ret)
+		goto unwind;
+
+	map = hw->pf.avail_hist_list_entries;
+	bitmap_fill(map->map, map->len);
+
+	for (i = 0; i < DLB_MAX_NUM_VDEVS; i++) {
+		ret = dlb_bitmap_alloc(&hw->vdev[i].avail_hist_list_entries,
+				       DLB_MAX_NUM_HIST_LIST_ENTRIES);
+		if (ret)
+			goto unwind;
+
+		map = hw->vdev[i].avail_hist_list_entries;
+		bitmap_zero(map->map, map->len);
+	}
+
+	/* Initialize the hardware resource IDs */
+	for (i = 0; i < DLB_MAX_NUM_DOMAINS; i++)
+		hw->domains[i].id = i;
+
+	for (i = 0; i < DLB_MAX_NUM_LDB_QUEUES; i++)
+		hw->rsrcs.ldb_queues[i].id = i;
+
+	for (i = 0; i < DLB_MAX_NUM_LDB_PORTS; i++)
+		hw->rsrcs.ldb_ports[i].id = i;
+
+	for (i = 0; i < DLB_MAX_NUM_DIR_PORTS; i++)
+		hw->rsrcs.dir_pq_pairs[i].id = i;
+
+	for (i = 0; i < DLB_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
+		hw->rsrcs.sn_groups[i].id = i;
+		/* Default mode (0) is 64 sequence numbers per queue */
+		hw->rsrcs.sn_groups[i].mode = 0;
+		hw->rsrcs.sn_groups[i].sequence_numbers_per_queue = 64;
+		hw->rsrcs.sn_groups[i].slot_use_bitmap = 0;
+	}
+
+	for (i = 0; i < DLB_NUM_COS_DOMAINS; i++)
+		hw->cos_reservation[i] = 100 / DLB_NUM_COS_DOMAINS;
+
+	return 0;
+
+unwind:
+	dlb_resource_free(hw);
+
+	return ret;
+}
+
+/**
+ * dlb_clr_pmcsr_disable() - power on bulk of DLB 2.0 logic
+ * @hw: dlb_hw handle for a particular device.
+ *
+ * Clearing the PMCSR must be done at initialization to make the device fully
+ * operational.
+ */
+void dlb_clr_pmcsr_disable(struct dlb_hw *hw)
+{
+	u32 pmcsr_dis;
+
+	pmcsr_dis = DLB_CSR_RD(hw, CM_CFG_PM_PMCSR_DISABLE);
+
+	/* Clear register bits */
+	pmcsr_dis &= ~CM_CFG_PM_PMCSR_DISABLE_DISABLE;
+
+	DLB_CSR_WR(hw, CM_CFG_PM_PMCSR_DISABLE, pmcsr_dis);
+}