vfio-pci/zdev: Improved zPCI Function Measurement Support

Started by Omar Elghoul on Nov 3, 2026 · 16 messages from 5 authors · Last reply Nov 6, 2026

Series

2 patches, 7 files

+172 −9

Subsystems (5)

Show 2 more

Maintainers (9)

Show 5 more
COOLING9d SUPERSEDED

Status: Superseded by v13 (posted 2026-11-11). [go to v13]

Revision v12 (13 revisions archived).

Revisions (13)

open the first message on its own page

  1. [PATCH v12 1/5] s390/pci: Hold fmb_lock when enabling or disabling PCI devices
    Related patches
  2. [PATCH v12 5/5] vfio-pci/zdev: Add VFIO FMB device features
    Revisions and related patches
    Revisions (4)
    1. v10
    2. v11 [diff against previous]
    3. v12 current [diff against previous]
    4. v13 [diff against previous]
  3. [PATCH v12 3/5] s390/pci: Improve FMB memory management
    Related patches
  4. [PATCH v12 4/5] s390/pci: Fence FMB enable/disable via debugfs for passthrough devices
    Related patches
  5. [PATCH v12 2/5] s390/pci: Improve function measurement state management
    Revisions and related patches
    Revisions (13)
Message details and related content
Hi,

This patch series improves support for function measurement for zPCI
passthrough devices on s390.

Changelog
=========
v11 -> v12:
* Patch 2/5:
  - Replace the statistics_on bool with a bitfield

* Patch 3/5:
  - Add the fmb_enabled bitfield to struct zpci_dev to track the actual
    FMB enablement
  - Replace the kmem_cache allocator with kzalloc/kfree after learning
    it guarantees alignment for power-of-two sizes

* Patch 5/5:
  - Use fmb_enabled as the FMB liveness check

v10 -> v11:
* Patches 2/5 and 3/5:
  - Swap their ordering within the series
  - Set/unset statistics_on after an enable/disable succeeds instead of
    before

* Patch 5/5:
  - On FMB re-enable, use an intermediate request-off step instead of
    directly touching the device state

v9 -> v10:
* Patches 2/5 and 3/5:
  - Split the old 2/4 into two patches; the new patch 2/5 reuses one FMB
    buffer per device, and 3/5 improves FMB state management
  - Rework FMB management functions to decouple requesting enablement or
    disablement from managing the user's intended state
  - Preserve measurement state in pcibios_enable_device()

v8 -> v9:
* Patch 2/4:

139 more lines

[PATCH v12 1/5] s390/pci: Hold fmb_lock when enabling or disabling PCI devices

Message details and related content
Ensure that fmb_lock is held by pcibios_enable_device() and
pcibios_disable_device() when calling zpci_fmb_enable_device() or
zpci_fmb_disable_device(), respectively. Additionally, assert that the
fmb_lock is held within the latter two functions to prevent future race
conditions regarding new callers.

Fixes: af0a8a8453f7 ("s390/pci: implement pcibios_add_device")
Fixes: 944239c59e93 ("s390/pci: implement pcibios_release_device")
Cc: <redacted>
Signed-off-by: Omar Elghoul <redacted>
Reviewed-by: Niklas Schnelle <schnelle@linux.ibm.com>
Reviewed-by: Matthew Rosato <mjrosato@linux.ibm.com>
---
 arch/s390/pci/pci.c | 8 ++++++++
 1 file changed, 8 insertions(+)
diff --git a/arch/s390/pci/pci.c b/arch/s390/pci/pci.c
index d64c544b32b0..c055a9ad0972 100644
--- a/arch/s390/pci/pci.c
+++ b/arch/s390/pci/pci.c
@@ -173,6 +173,8 @@ int zpci_fmb_enable_device(struct zpci_dev *zdev)
 	unsigned long flags;
 	u8 cc, status;
 
+	lockdep_assert_held(&zdev->fmb_lock);
+
 	if (zdev->fmb || sizeof(*zdev->fmb) < zdev->fmb_length)
 		return -EINVAL;
 
@@ -211,6 +213,8 @@ int zpci_fmb_disable_device(struct zpci_dev *zdev)
 	struct zpci_fib fib = {0};
 	u8 cc, status;
 
+	lockdep_assert_held(&zdev->fmb_lock);
+
 	if (!zdev->fmb)
 		return -EINVAL;
 
@@ -639,7 +643,9 @@ int pcibios_enable_device(struct pci_dev *pdev, int mask)
 	struct zpci_dev *zdev = to_zpci(pdev);
 
 	zpci_debug_init_device(zdev, dev_name(&pdev->dev));
+	mutex_lock(&zdev->fmb_lock);
 	zpci_fmb_enable_device(zdev);
+	mutex_unlock(&zdev->fmb_lock);
 
 	return pci_enable_resources(pdev, mask);
 }
@@ -648,7 +654,9 @@ void pcibios_disable_device(struct pci_dev *pdev)
 {
 	struct zpci_dev *zdev = to_zpci(pdev);
 
+	mutex_lock(&zdev->fmb_lock);
 	zpci_fmb_disable_device(zdev);
+	mutex_unlock(&zdev->fmb_lock);
 	zpci_debug_exit_device(zdev);
 }
 
-- 
2.56.0

[PATCH v12 5/5] vfio-pci/zdev: Add VFIO FMB device features

Message details and related content
Introduce new VFIO features for zPCI devices to provide FMB passthrough to
userspace.

Allow the user to enable or disable the FMB using the SET-only feature
VFIO_DEVICE_FEATURE_ZPCI_FMB_ENABLE. Likewise allow the user to read the
latest FMB using the GET-only feature VFIO_DEVICE_FEATURE_ZPCI_FMB_READ
in the case when the FMB is enabled.

Signed-off-by: Omar Elghoul <redacted>
Acked-by: Alex Williamson <alex@shazbot.org>
---
 drivers/vfio/pci/vfio_pci_core.c |  4 ++
 drivers/vfio/pci/vfio_pci_priv.h | 18 +++++++++
 drivers/vfio/pci/vfio_pci_zdev.c | 64 ++++++++++++++++++++++++++++++++
 include/uapi/linux/vfio.h        | 30 +++++++++++++++
 4 files changed, 116 insertions(+)
diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index fcbb2da860e2..f2d5b6857925 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -1614,6 +1614,10 @@ int vfio_pci_core_ioctl_feature(struct vfio_device *device, u32 flags,
 		return vfio_pci_core_feature_dma_buf(vdev, flags, arg, argsz);
 	case VFIO_DEVICE_FEATURE_ZPCI_ERROR:
 		return vfio_pci_zdev_feature_err(device, flags, arg, argsz);
+	case VFIO_DEVICE_FEATURE_ZPCI_FMB_ENABLE:
+		return vfio_pci_zdev_feature_fmb_enable(vdev, flags, arg, argsz);
+	case VFIO_DEVICE_FEATURE_ZPCI_FMB_READ:
+		return vfio_pci_zdev_feature_fmb_read(vdev, flags, arg, argsz);
 	default:
 		return -ENOTTY;
 	}
diff --git a/drivers/vfio/pci/vfio_pci_priv.h b/drivers/vfio/pci/vfio_pci_priv.h
index 4e7162234a2e..e04d7e9d0c30 100644
--- a/drivers/vfio/pci/vfio_pci_priv.h
+++ b/drivers/vfio/pci/vfio_pci_priv.h
@@ -95,6 +95,10 @@ int vfio_pci_zdev_open_device(struct vfio_pci_core_device *vdev);
 void vfio_pci_zdev_close_device(struct vfio_pci_core_device *vdev);
 int vfio_pci_zdev_feature_err(struct vfio_device *device, u32 flags,
 			      void __user *arg, size_t argsz);
+int vfio_pci_zdev_feature_fmb_enable(struct vfio_pci_core_device *vdev, u32 flags,
+				     void __user *arg, size_t argsz);
+int vfio_pci_zdev_feature_fmb_read(struct vfio_pci_core_device *vdev, u32 flags,
+				   void __user *arg, size_t argsz);
 #else
 static inline int vfio_pci_info_zdev_add_caps(struct vfio_pci_core_device *vdev,
 					      struct vfio_info_cap *caps)
@@ -116,6 +120,20 @@ static inline int vfio_pci_zdev_feature_err(struct vfio_device *device,
 {
 	return -ENOTTY;
 }
+
+static inline int vfio_pci_zdev_feature_fmb_enable(struct vfio_pci_core_device *vdev,
+						   u32 flags, void __user *arg,
+						   size_t argsz)
+{
+	return -ENOTTY;
+}
+
+static inline int vfio_pci_zdev_feature_fmb_read(struct vfio_pci_core_device *vdev,
+						 u32 flags, void __user *arg,
+						 size_t argsz)
+{
+	return -ENOTTY;
+}
 #endif
 
 static inline bool vfio_pci_is_vga(struct pci_dev *pdev)
diff --git a/drivers/vfio/pci/vfio_pci_zdev.c b/drivers/vfio/pci/vfio_pci_zdev.c
index f47f36314a1c..0584fbaf8755 100644
--- a/drivers/vfio/pci/vfio_pci_zdev.c
+++ b/drivers/vfio/pci/vfio_pci_zdev.c
@@ -219,3 +219,67 @@ void vfio_pci_zdev_close_device(struct vfio_pci_core_device *vdev)
 	if (zpci_kvm_hook.kvm_unregister)
 		zpci_kvm_hook.kvm_unregister(zdev);
 }
+
+int vfio_pci_zdev_feature_fmb_enable(struct vfio_pci_core_device *vdev, u32 flags,
+				     void __user *arg, size_t argsz)
+{
+	struct zpci_dev *zdev;
+	struct vfio_device_feature_zpci_fmb_enable fmb_enable;
+	int ret;
+
+	ret = vfio_check_feature(flags, argsz, VFIO_DEVICE_FEATURE_SET, sizeof(fmb_enable));
+	if (ret != 1)
+		return ret;
+
+	zdev = to_zpci(vdev->pdev);
+	if (!zdev)
+		return -ENODEV;
+
+	if (copy_from_user(&fmb_enable, arg, sizeof(fmb_enable)))
+		return -EFAULT;
+
+	guard(mutex)(&zdev->fmb_lock);
+
+	if (fmb_enable.enabled) {
+		ret = zpci_fmb_request_off(zdev);
+		if (ret && ret != -EINVAL)
+			return ret;
+
+		return zpci_fmb_request_on(zdev);
+	}
+
+	return zpci_fmb_request_off(zdev);
+}
+
+int vfio_pci_zdev_feature_fmb_read(struct vfio_pci_core_device *vdev, u32 flags,
+				   void __user *arg, size_t argsz)
+{
+	struct zpci_dev *zdev;
+	struct vfio_device_feature_zpci_fmb_read fmb_read;
+	struct zpci_fmb fmb_bounce;
+	int ret;
+
+	ret = vfio_check_feature(flags, argsz, VFIO_DEVICE_FEATURE_GET, sizeof(fmb_read));
+	if (ret != 1)
+		return ret;
+
+	zdev = to_zpci(vdev->pdev);
+	if (!zdev)
+		return -ENODEV;
+
+	if (copy_from_user(&fmb_read, arg, sizeof(fmb_read)))
+		return -EFAULT;
+	if (!fmb_read.data)
+		return -EINVAL;
+
+	scoped_guard(mutex, &zdev->fmb_lock) {
+		if (!zdev->fmb_enabled)
+			return -ENOMSG;
+		memcpy(&fmb_bounce, zdev->fmb, zdev->fmb_length);
+	}
+
+	if (copy_to_user(u64_to_user_ptr(fmb_read.data), &fmb_bounce, zdev->fmb_length))
+		return -EFAULT;
+
+	return 0;
+}
diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
index e41437fa17ad..7e9adb61c5e1 100644
--- a/include/uapi/linux/vfio.h
+++ b/include/uapi/linux/vfio.h
@@ -1555,6 +1555,36 @@ struct vfio_device_feature_zpci_err {
 
 #define VFIO_DEVICE_FEATURE_ZPCI_ERROR 13
 
+/**
+ * Upon VFIO_DEVICE_FEATURE_SET, enable or disable FMB for the VFIO zPCI device.
+ *
+ * enabled is treated as a bool, so any non-zero value evaluates to true. Nested
+ * enabling disables and re-enables the FMB, while nested disabling fails with
+ * EINVAL.
+ *
+ * Returns: 0 on success, -1 and errno set appropriately on error.
+ */
+#define VFIO_DEVICE_FEATURE_ZPCI_FMB_ENABLE 14
+
+struct vfio_device_feature_zpci_fmb_enable {
+	__u8 enabled;
+};
+
+/**
+ * Upon VFIO_DEVICE_FEATURE_GET, provide FMB passthrough for VFIO zPCI devices.
+ *
+ * The user-provided buffer must be at least fmb_length large, where fmb_length
+ * is reported in VFIO_DEVICE_INFO_CAP_ZPCI_BASE. If measurement is not enabled,
+ * fail with ENOMSG.
+ *
+ * Returns: 0 on success, -1 and errno set appropriately on error.
+ */
+#define VFIO_DEVICE_FEATURE_ZPCI_FMB_READ 15
+
+struct vfio_device_feature_zpci_fmb_read {
+	__aligned_u64 data;
+};
+
 /* -------- API for Type1 VFIO IOMMU -------- */
 
 /**
-- 
2.56.0

[PATCH v12 3/5] s390/pci: Improve FMB memory management

Message details and related content
Refactor the FMB memory management code to use kzalloc() instead of a
dedicated kmem_cache. kzalloc() already guarantees adequate alignment
for power-of-two sizes, rendering the kmem_cache approach unnecessary.
This simplifies both the initialization and cleanup procedures.

Allocate the FMB once upon first enable and keep it for the lifetime of
the zdev instead of freeing it on every disable. Defer freeing the
buffer until device teardown in zpci_release_device(). Add the field
fmb_enabled to struct zpci_dev as the new FMB liveness bool, since a
non-null FMB pointer no longer implies FMB enablement.

Signed-off-by: Omar Elghoul <redacted>
---
 arch/s390/include/asm/pci.h |  1 +
 arch/s390/pci/pci.c         | 59 +++++++++++++++++++++----------------
 arch/s390/pci/pci_debug.c   |  2 +-
 3 files changed, 35 insertions(+), 27 deletions(-)
diff --git a/arch/s390/include/asm/pci.h b/arch/s390/include/asm/pci.h
index dd58c48ac2ed..6d3c1b8ac75b 100644
--- a/arch/s390/include/asm/pci.h
+++ b/arch/s390/include/asm/pci.h
@@ -209,6 +209,7 @@ struct zpci_dev {
 	u16		fmb_update;	/* update interval */
 	u16		fmb_length;
 	u8		statistics_on	: 1; /* desired FMB enablement */
+	u8		fmb_enabled	: 1; /* actual FMB enablement */
 
 	u8		version;
 	enum pci_bus_speed max_bus_speed;
diff --git a/arch/s390/pci/pci.c b/arch/s390/pci/pci.c
index f49736386eb7..f066a7ffc35c 100644
--- a/arch/s390/pci/pci.c
+++ b/arch/s390/pci/pci.c
@@ -62,8 +62,6 @@ EXPORT_SYMBOL_GPL(zpci_iomap_start);
 
 DEFINE_STATIC_KEY_FALSE(have_mio);
 
-static struct kmem_cache *zdev_fmb_cache;
-
 /* AEN structures that must be preserved over KVM module re-insertion */
 union zpci_sic_iib *zpci_aipb;
 EXPORT_SYMBOL_GPL(zpci_aipb);
@@ -205,13 +203,17 @@ int zpci_fmb_enable(struct zpci_dev *zdev)
 
 	lockdep_assert_held(&zdev->fmb_lock);
 
-	if (zdev->fmb || sizeof(*zdev->fmb) < zdev->fmb_length)
+	if (zdev->fmb_enabled || sizeof(*zdev->fmb) < zdev->fmb_length)
 		return -EINVAL;
 
-	zdev->fmb = kmem_cache_zalloc(zdev_fmb_cache, GFP_KERNEL);
-	if (!zdev->fmb)
-		return -ENOMEM;
-	WARN_ON((u64) zdev->fmb & 0xf);
+	if (!zdev->fmb) {
+		zdev->fmb = kzalloc_obj(*zdev->fmb, GFP_KERNEL);
+		if (!zdev->fmb)
+			return -ENOMEM;
+	} else {
+		/* reuse the same FMB buffer for as long as the zdev lives */
+		memset(zdev->fmb, 0, sizeof(*zdev->fmb));
+	}
 
 	/* reset software counters */
 	spin_lock_irqsave(&zdev->dom_lock, flags);
@@ -229,11 +231,12 @@ int zpci_fmb_enable(struct zpci_dev *zdev)
 	fib.fmb_addr = virt_to_phys(zdev->fmb);
 	fib.gd = zdev->gisa;
 	cc = zpci_mod_fc(req, &fib, &status);
-	if (cc) {
-		kmem_cache_free(zdev_fmb_cache, zdev->fmb);
-		zdev->fmb = NULL;
-	}
-	return cc ? -EIO : 0;
+
+	if (cc)
+		return -EIO;
+
+	zdev->fmb_enabled = 1;
+	return 0;
 }
 
 /* Modify PCI: Disable PCI function measurement */
@@ -245,7 +248,7 @@ int zpci_fmb_disable(struct zpci_dev *zdev)
 
 	lockdep_assert_held(&zdev->fmb_lock);
 
-	if (!zdev->fmb)
+	if (!zdev->fmb_enabled)
 		return -EINVAL;
 
 	fib.gd = zdev->gisa;
@@ -255,11 +258,11 @@ int zpci_fmb_disable(struct zpci_dev *zdev)
 	if (cc == 3) /* Function already gone. */
 		cc = 0;
 
-	if (!cc) {
-		kmem_cache_free(zdev_fmb_cache, zdev->fmb);
-		zdev->fmb = NULL;
-	}
-	return cc ? -EIO : 0;
+	if (cc)
+		return -EIO;
+
+	zdev->fmb_enabled = 0;
+	return 0;
 }
 
 static int zpci_cfg_load(struct zpci_dev *zdev, int offset, u32 *val, u8 len)
@@ -802,6 +805,12 @@ int zpci_disable_device(struct zpci_dev *zdev)
 	} else {
 		rc = -EIO;
 	}
+
+	if (!rc) {
+		guard(mutex)(&zdev->fmb_lock);
+		zdev->fmb_enabled = 0;
+	}
+
 	return rc;
 }
 EXPORT_SYMBOL_GPL(zpci_disable_device);
@@ -1047,6 +1056,12 @@ void zpci_release_device(struct kref *kref)
 	if (zdev->has_resources)
 		zpci_cleanup_bus_resources(zdev);
 
+	if (zdev->fmb) {
+		zdev->fmb_enabled = 0;
+		kfree(zdev->fmb);
+		zdev->fmb = NULL;
+	}
+
 	zpci_bus_device_unregister(zdev);
 	zpci_destroy_iommu(zdev);
 	zpci_dbg(3, "rem fid:%x\n", zdev->fid);
@@ -1122,11 +1137,6 @@ static int zpci_mem_init(void)
 	BUILD_BUG_ON((CONFIG_ILLEGAL_POINTER_VALUE + 0x10000 > ZPCI_IOMAP_ADDR_BASE) &&
 		     (CONFIG_ILLEGAL_POINTER_VALUE <= ZPCI_IOMAP_ADDR_MAX));
 
-	zdev_fmb_cache = kmem_cache_create("PCI_FMB_cache", sizeof(struct zpci_fmb),
-					   __alignof__(struct zpci_fmb), 0, NULL);
-	if (!zdev_fmb_cache)
-		goto error_fmb;
-
 	zpci_iomap_start = kzalloc_objs(*zpci_iomap_start, ZPCI_IOMAP_ENTRIES);
 	if (!zpci_iomap_start)
 		goto error_iomap;
@@ -1143,8 +1153,6 @@ static int zpci_mem_init(void)
 error_iomap_bitmap:
 	kfree(zpci_iomap_start);
 error_iomap:
-	kmem_cache_destroy(zdev_fmb_cache);
-error_fmb:
 	return -ENOMEM;
 }
 
@@ -1152,7 +1160,6 @@ static void zpci_mem_exit(void)
 {
 	kfree(zpci_iomap_bitmap);
 	kfree(zpci_iomap_start);
-	kmem_cache_destroy(zdev_fmb_cache);
 }
 
 static unsigned int s390_pci_probe __initdata = 1;
diff --git a/arch/s390/pci/pci_debug.c b/arch/s390/pci/pci_debug.c
index 06e2e8bb2f82..454121bc274c 100644
--- a/arch/s390/pci/pci_debug.c
+++ b/arch/s390/pci/pci_debug.c
@@ -97,7 +97,7 @@ static int pci_perf_show(struct seq_file *m, void *v)
 		return 0;
 
 	mutex_lock(&zdev->fmb_lock);
-	if (!zdev->fmb) {
+	if (!zdev->fmb_enabled) {
 		mutex_unlock(&zdev->fmb_lock);
 		seq_puts(m, "FMB statistics disabled\n");
 		return 0;
-- 
2.56.0

[PATCH v12 4/5] s390/pci: Fence FMB enable/disable via debugfs for passthrough devices

Message details and related content
Introduce a fence over enabling or disabling FMB via debugfs when the zPCI
device is associated with a KVM. This will prevent processes on the host
from tampering with the FMB while the guest is still using it, which may
cause partial counter resets and inconsistent reads which have no parallel
in the architecture.

For VFIO devices that are not associated with a KVM (i.e., for userspace
drivers other than QEMU), this fence does not take effect.

Signed-off-by: Omar Elghoul <redacted>
Reviewed-by: Matthew Rosato <mjrosato@linux.ibm.com>
Reviewed-by: Niklas Schnelle <schnelle@linux.ibm.com>
---
 arch/s390/pci/pci_debug.c | 9 +++++++++
 1 file changed, 9 insertions(+)
diff --git a/arch/s390/pci/pci_debug.c b/arch/s390/pci/pci_debug.c
index 454121bc274c..02da077b5170 100644
--- a/arch/s390/pci/pci_debug.c
+++ b/arch/s390/pci/pci_debug.c
@@ -153,6 +153,12 @@ static ssize_t pci_perf_seq_write(struct file *file, const char __user *ubuf,
 	if (rc)
 		return rc;
 
+	mutex_lock(&zdev->kzdev_lock);
+	if (zdev->kzdev) {
+		rc = -EPERM;
+		goto out_unlock_kzdev;
+	}
+
 	mutex_lock(&zdev->fmb_lock);
 	switch (val) {
 	case 0:
@@ -163,6 +169,9 @@ static ssize_t pci_perf_seq_write(struct file *file, const char __user *ubuf,
 		break;
 	}
 	mutex_unlock(&zdev->fmb_lock);
+
+out_unlock_kzdev:
+	mutex_unlock(&zdev->kzdev_lock);
 	return rc ? rc : count;
 }
 
-- 
2.56.0

[PATCH v12 2/5] s390/pci: Improve function measurement state management

Message details and related content
When a zPCI function is disabled and re-enabled, the firmware implicitly
disables function measurement while the kernel still believes it is
enabled, leading to a disconnect between the user-visible state and the
actual device state.

Add the statistics_on field to struct zpci_dev to track the desired FMB
state. Introduce the functions zpci_fmb_request_on()/_off(), which defer
to zpci_fmb_enable()/_disable() to perform the actual firmware change.
This decouples the enablement or disablement step from managing the
user's requested state.

Restore the state in pcibios_enable_device(), pcibios_disable_device(),
and zpci_reenable_device(). New devices start with measurement on by
default, unchanged from before.

Signed-off-by: Omar Elghoul <redacted>
---
 arch/s390/include/asm/pci.h |  7 +++--
 arch/s390/pci/pci.c         | 54 +++++++++++++++++++++++++++++++++----
 arch/s390/pci/pci_debug.c   |  4 +--
 3 files changed, 56 insertions(+), 9 deletions(-)
diff --git a/arch/s390/include/asm/pci.h b/arch/s390/include/asm/pci.h
index 88a125b92bdd..dd58c48ac2ed 100644
--- a/arch/s390/include/asm/pci.h
+++ b/arch/s390/include/asm/pci.h
@@ -208,6 +208,7 @@ struct zpci_dev {
 	struct zpci_fmb *fmb;
 	u16		fmb_update;	/* update interval */
 	u16		fmb_length;
+	u8		statistics_on	: 1; /* desired FMB enablement */
 
 	u8		version;
 	enum pci_bus_speed max_bus_speed;
@@ -349,8 +350,10 @@ int zpci_create_parent_msi_domain(struct zpci_bus *zbus);
 void zpci_remove_parent_msi_domain(struct zpci_bus *zbus);
 
 /* FMB */
-int zpci_fmb_enable_device(struct zpci_dev *);
-int zpci_fmb_disable_device(struct zpci_dev *);
+int zpci_fmb_request_on(struct zpci_dev *zdev);
+int zpci_fmb_request_off(struct zpci_dev *zdev);
+int zpci_fmb_enable(struct zpci_dev *zdev);
+int zpci_fmb_disable(struct zpci_dev *zdev);
 
 /* Debug */
 int zpci_debug_init(void);
diff --git a/arch/s390/pci/pci.c b/arch/s390/pci/pci.c
index c055a9ad0972..f49736386eb7 100644
--- a/arch/s390/pci/pci.c
+++ b/arch/s390/pci/pci.c
@@ -164,8 +164,38 @@ int zpci_unregister_ioat(struct zpci_dev *zdev, u8 dmaas)
 	return cc;
 }
 
+int zpci_fmb_request_on(struct zpci_dev *zdev)
+{
+	int rc;
+
+	lockdep_assert_held(&zdev->fmb_lock);
+
+	rc = zpci_fmb_enable(zdev);
+	if (rc)
+		return rc;
+
+	zdev->statistics_on = 1;
+	return 0;
+}
+EXPORT_SYMBOL_GPL(zpci_fmb_request_on);
+
+int zpci_fmb_request_off(struct zpci_dev *zdev)
+{
+	int rc;
+
+	lockdep_assert_held(&zdev->fmb_lock);
+
+	rc = zpci_fmb_disable(zdev);
+	if (rc)
+		return rc;
+
+	zdev->statistics_on = 0;
+	return 0;
+}
+EXPORT_SYMBOL_GPL(zpci_fmb_request_off);
+
 /* Modify PCI: Set PCI function measurement parameters */
-int zpci_fmb_enable_device(struct zpci_dev *zdev)
+int zpci_fmb_enable(struct zpci_dev *zdev)
 {
 	u64 req = ZPCI_CREATE_REQ(zdev->fh, 0, ZPCI_MOD_FC_SET_MEASURE);
 	struct zpci_iommu_ctrs *ctrs;
@@ -207,7 +237,7 @@ int zpci_fmb_enable_device(struct zpci_dev *zdev)
 }
 
 /* Modify PCI: Disable PCI function measurement */
-int zpci_fmb_disable_device(struct zpci_dev *zdev)
+int zpci_fmb_disable(struct zpci_dev *zdev)
 {
 	u64 req = ZPCI_CREATE_REQ(zdev->fh, 0, ZPCI_MOD_FC_SET_MEASURE);
 	struct zpci_fib fib = {0};
@@ -644,7 +674,8 @@ int pcibios_enable_device(struct pci_dev *pdev, int mask)
 
 	zpci_debug_init_device(zdev, dev_name(&pdev->dev));
 	mutex_lock(&zdev->fmb_lock);
-	zpci_fmb_enable_device(zdev);
+	if (zdev->statistics_on)
+		zpci_fmb_enable(zdev);
 	mutex_unlock(&zdev->fmb_lock);
 
 	return pci_enable_resources(pdev, mask);
@@ -655,7 +686,8 @@ void pcibios_disable_device(struct pci_dev *pdev)
 	struct zpci_dev *zdev = to_zpci(pdev);
 
 	mutex_lock(&zdev->fmb_lock);
-	zpci_fmb_disable_device(zdev);
+	if (zdev->statistics_on)
+		zpci_fmb_disable(zdev);
 	mutex_unlock(&zdev->fmb_lock);
 	zpci_debug_exit_device(zdev);
 }
@@ -737,8 +769,14 @@ int zpci_reenable_device(struct zpci_dev *zdev)
 	}
 
 	rc = zpci_iommu_register_ioat(zdev, &status);
-	if (rc)
+	if (rc) {
 		zpci_disable_device(zdev);
+		return rc;
+	}
+
+	guard(mutex)(&zdev->fmb_lock);
+	if (zdev->statistics_on)
+		zpci_fmb_enable(zdev);
 
 	return rc;
 }
@@ -847,6 +885,12 @@ struct zpci_dev *zpci_create_device(u32 fid, u32 fh, enum zpci_state state)
 		goto error;
 	zdev->state =  state;
 
+	/*
+	 * Measurement is on by default for new functions, this will be honored
+	 * by pcibios_enable_device().
+	 */
+	zdev->statistics_on = 1;
+
 	mutex_init(&zdev->state_lock);
 	mutex_init(&zdev->fmb_lock);
 	mutex_init(&zdev->kzdev_lock);
diff --git a/arch/s390/pci/pci_debug.c b/arch/s390/pci/pci_debug.c
index c7ed7bf254b5..06e2e8bb2f82 100644
--- a/arch/s390/pci/pci_debug.c
+++ b/arch/s390/pci/pci_debug.c
@@ -156,10 +156,10 @@ static ssize_t pci_perf_seq_write(struct file *file, const char __user *ubuf,
 	mutex_lock(&zdev->fmb_lock);
 	switch (val) {
 	case 0:
-		rc = zpci_fmb_disable_device(zdev);
+		rc = zpci_fmb_request_off(zdev);
 		break;
 	case 1:
-		rc = zpci_fmb_enable_device(zdev);
+		rc = zpci_fmb_request_on(zdev);
 		break;
 	}
 	mutex_unlock(&zdev->fmb_lock);
-- 
2.56.0

Re: [PATCH v12 4/5] s390/pci: Fence FMB enable/disable via debugfs for passthrough devices

Message details and related content
Introduce a fence over enabling or disabling FMB via debugfs when the zPCI
device is associated with a KVM. This will prevent processes on the host
from tampering with the FMB while the guest is still using it, which may
cause partial counter resets and inconsistent reads which have no parallel
in the architecture.

For VFIO devices that are not associated with a KVM (i.e., for userspace
drivers other than QEMU), this fence does not take effect.

Signed-off-by: Omar Elghoul <redacted>
Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261103150114.68873-1-oelghoul@linux.ibm.com?part=4

Re: [PATCH v12 5/5] vfio-pci/zdev: Add VFIO FMB device features

Message details and related content
Introduce new VFIO features for zPCI devices to provide FMB passthrough to
userspace.

Allow the user to enable or disable the FMB using the SET-only feature
VFIO_DEVICE_FEATURE_ZPCI_FMB_ENABLE. Likewise allow the user to read the
latest FMB using the GET-only feature VFIO_DEVICE_FEATURE_ZPCI_FMB_READ
in the case when the FMB is enabled.

Signed-off-by: Omar Elghoul <redacted>
Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261103150114.68873-1-oelghoul@linux.ibm.com?part=5

Re: [PATCH v12 1/5] s390/pci: Hold fmb_lock when enabling or disabling PCI devices

Message details and related content
Ensure that fmb_lock is held by pcibios_enable_device() and
pcibios_disable_device() when calling zpci_fmb_enable_device() or
zpci_fmb_disable_device(), respectively. Additionally, assert that the
fmb_lock is held within the latter two functions to prevent future race
conditions regarding new callers.

Fixes: af0a8a8453f7 ("s390/pci: implement pcibios_add_device")
Fixes: 944239c59e93 ("s390/pci: implement pcibios_release_device")
Cc: <redacted>
Signed-off-by: Omar Elghoul <redacted>
Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261103150114.68873-1-oelghoul@linux.ibm.com?part=1

Re: [PATCH v12 3/5] s390/pci: Improve FMB memory management

Message details and related content
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] s390/pci: fmb_enabled not cleared when function is already disabled
--

commit ca3a9f4958dfbbed1e7f4c289498cd7e117c52ca
Author: Omar Elghoul [off-list ref]

s390/pci: Improve FMB memory management

This commit replaces the dedicated kmem_cache for Function Measurement
Blocks (FMB) with kzalloc_obj() and retains the allocated FMB buffer across
disable cycles until device release. Because a non-NULL zdev->fmb pointer no
longer indicates whether measurement is active, it adds a zdev->fmb_enabled
flag to track FMB state.
quoted hunk
diff --git a/arch/s390/pci/pci.c b/arch/s390/pci/pci.c
index f49736386eb7c..f066a7ffc35c1 100644
--- a/arch/s390/pci/pci.c
+++ b/arch/s390/pci/pci.c
[ ... ]
quoted hunk
@@ -205,13 +203,17 @@ int zpci_fmb_enable(struct zpci_dev *zdev)
 
 	lockdep_assert_held(&zdev->fmb_lock);
 
-	if (zdev->fmb || sizeof(*zdev->fmb) < zdev->fmb_length)
+	if (zdev->fmb_enabled || sizeof(*zdev->fmb) < zdev->fmb_length)
 		return -EINVAL;
[ ... ]
quoted hunk
@@ -802,6 +805,12 @@ int zpci_disable_device(struct zpci_dev *zdev)
 	} else {
 		rc = -EIO;
 	}
+
+	if (!rc) {
+		guard(mutex)(&zdev->fmb_lock);
+		zdev->fmb_enabled = 0;
+	}
+
 	return rc;
 }
 EXPORT_SYMBOL_GPL(zpci_disable_device);
[Severity: Medium]
Can zdev->fmb_enabled remain set to 1 when the PCI function is disabled by
firmware during reset or error recovery, preventing zpci_reenable_device()
from re-enabling FMB measurement?

In zpci_disable_device(), when clp_disable_fh() returns
CLP_RC_SETPCIFN_ALRDY because the function is already disabled in firmware,
clp_refresh_fh() updates zdev->fh to the disabled handle and sets rc to
-EINVAL:

arch/s390/pci/pci.c:zpci_disable_device() {
	cc = clp_disable_fh(zdev, &fh);
	if (!cc) {
		zpci_update_fh(zdev, fh);
	} else if (cc == CLP_RC_SETPCIFN_ALRDY) {
		...
		rc = clp_refresh_fh(zdev->fid, &fh);
		if (!rc) {
			zpci_update_fh(zdev, fh);
			rc = -EINVAL;
		}
	} else {
		rc = -EIO;
	}

	if (!rc) {
		guard(mutex)(&zdev->fmb_lock);
		zdev->fmb_enabled = 0;
	}
}

Because zdev->fmb_enabled is only cleared when rc is 0, it stays set to 1 on
the -EINVAL path. Similarly, when a platform error event disables the
function on LPAR, __zpci_event_error() updates zdev->fh to the disabled
handle via zpci_update_fh() without clearing zdev->fmb_enabled.

When zpci_hot_reset_device() (or _do_recover() in pci_sysfs.c) then runs:

arch/s390/pci/pci.c:zpci_hot_reset_device() {

25 more lines

Re: [PATCH v12 2/5] s390/pci: Improve function measurement state management

Message details and related content
When a zPCI function is disabled and re-enabled, the firmware implicitly
disables function measurement while the kernel still believes it is
enabled, leading to a disconnect between the user-visible state and the
actual device state.

Add the statistics_on field to struct zpci_dev to track the desired FMB
state. Introduce the functions zpci_fmb_request_on()/_off(), which defer
to zpci_fmb_enable()/_disable() to perform the actual firmware change.
This decouples the enablement or disablement step from managing the
user's requested state.

Restore the state in pcibios_enable_device(), pcibios_disable_device(),
and zpci_reenable_device(). New devices start with measurement on by
default, unchanged from before.

Signed-off-by: Omar Elghoul <redacted>
Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261103150114.68873-1-oelghoul@linux.ibm.com?part=2

Re: [PATCH v12 5/5] vfio-pci/zdev: Add VFIO FMB device features

Message details and related content
On Tue, 2026-11-03 at 10:01 -0500, Omar Elghoul wrote:
Introduce new VFIO features for zPCI devices to provide FMB passthrough to
userspace.

Allow the user to enable or disable the FMB using the SET-only feature
VFIO_DEVICE_FEATURE_ZPCI_FMB_ENABLE. Likewise allow the user to read the
latest FMB using the GET-only feature VFIO_DEVICE_FEATURE_ZPCI_FMB_READ
in the case when the FMB is enabled.

Signed-off-by: Omar Elghoul <redacted>
Acked-by: Alex Williamson <alex@shazbot.org>
---
 drivers/vfio/pci/vfio_pci_core.c |  4 ++
 drivers/vfio/pci/vfio_pci_priv.h | 18 +++++++++
 drivers/vfio/pci/vfio_pci_zdev.c | 64 ++++++++++++++++++++++++++++++++
 include/uapi/linux/vfio.h        | 30 +++++++++++++++
 4 files changed, 116 insertions(+)
--- snip ---
quoted hunk
+
+int vfio_pci_zdev_feature_fmb_read(struct vfio_pci_core_device *vdev, u32 flags,
+				   void __user *arg, size_t argsz)
+{
+	struct zpci_dev *zdev;
+	struct vfio_device_feature_zpci_fmb_read fmb_read;
+	struct zpci_fmb fmb_bounce;
+	int ret;
+
+	ret = vfio_check_feature(flags, argsz, VFIO_DEVICE_FEATURE_GET, sizeof(fmb_read));
+	if (ret != 1)
+		return ret;
+
+	zdev = to_zpci(vdev->pdev);
+	if (!zdev)
+		return -ENODEV;
+
+	if (copy_from_user(&fmb_read, arg, sizeof(fmb_read)))
+		return -EFAULT;
+	if (!fmb_read.data)
+		return -EINVAL;
+
+	scoped_guard(mutex, &zdev->fmb_lock) {
+		if (!zdev->fmb_enabled)
+			return -ENOMSG;
For those who were also wondering what would happen if the zdev has an
FMB length too large for this kernel's version of struct zpci_fmb. In
that case zpci_fmb_enable() fails the FMB enablement leaving zdev-
fmb_enabled false and then the check above also covers the memcpy()
here even if user-space ignores that the FMB is not enabled.
quoted hunk
+		memcpy(&fmb_bounce, zdev->fmb, zdev->fmb_length);
+	}
+
+	if (copy_to_user(u64_to_user_ptr(fmb_read.data), &fmb_bounce, zdev->fmb_length))
+		return -EFAULT;
+
+	return 0;
+}
Looks good, looking forward to having function measurement in KVM
guests.

Reviewed-by: Niklas Schnelle <schnelle@linux.ibm.com>

Re: [PATCH v12 3/5] s390/pci: Improve FMB memory management

Message details and related content
On Tue, 2026-11-03 at 15:15 +0000, sashiko-bot@kernel.org wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] s390/pci: fmb_enabled not cleared when function is already disabled
--

commit ca3a9f4958dfbbed1e7f4c289498cd7e117c52ca
Author: Omar Elghoul [off-list ref]

s390/pci: Improve FMB memory management

This commit replaces the dedicated kmem_cache for Function Measurement
Blocks (FMB) with kzalloc_obj() and retains the allocated FMB buffer across
disable cycles until device release. Because a non-NULL zdev->fmb pointer no
longer indicates whether measurement is active, it adds a zdev->fmb_enabled
flag to track FMB state.
quoted
diff --git a/arch/s390/pci/pci.c b/arch/s390/pci/pci.c
index f49736386eb7c..f066a7ffc35c1 100644
--- a/arch/s390/pci/pci.c
+++ b/arch/s390/pci/pci.c
[ ... ]
quoted
@@ -205,13 +203,17 @@ int zpci_fmb_enable(struct zpci_dev *zdev)
 
 	lockdep_assert_held(&zdev->fmb_lock);
 
-	if (zdev->fmb || sizeof(*zdev->fmb) < zdev->fmb_length)
+	if (zdev->fmb_enabled || sizeof(*zdev->fmb) < zdev->fmb_length)
 		return -EINVAL;
[ ... ]
quoted
@@ -802,6 +805,12 @@ int zpci_disable_device(struct zpci_dev *zdev)
 	} else {
 		rc = -EIO;
 	}
+
+	if (!rc) {
+		guard(mutex)(&zdev->fmb_lock);
+		zdev->fmb_enabled = 0;
+	}
+
 	return rc;
 }

33 more quoted lines

Before anyone thinks we didn't test error recovery it turns out that
actually works despite this issue being real. During error recovery
drivers end up calling pcibios_disable_device() before the
zpci_hot_reset_device() that gets us out of the error state so the
zdev->fmb_enabled gets cleared. Note that the current code also relies
on this mechanism already so this actually isn't changed by this patch
that just replaces zdev->fmb NULL checks with an explicit flag.

Looking at it from a fresh angle I think the real solution isn't to
just clear zdev->fmb_enabled in both branches above. Instead we should
clear zdev->fmb_enabled in zpci_update_fh() if and only if that
transitions the function handle from enabled to disabled. That covers
both the !cc and CLP_RC_SETPCIFN_ALRDY branches above but also any
other place where we might learn that the platform has disabled the
device and also have zdev_enabled(zdev) reporting go from true to
false. 

Due to the nature of learning of changes to the actual state of the
device via events this of course happens with a delay so we can still
run into cases where we try to disable an FMB and it's already disabled
under us. There's no way around that. It's sadly just quite finicky to
have synchronous transitions on one side and the hard reality of
"someone pulled the plug" on the other.

Thanks,
Niklas

Re: [PATCH v12 5/5] vfio-pci/zdev: Add VFIO FMB device features

Message details and related content
On 11/3/26 4:08 PM, Niklas Schnelle wrote:
On Tue, 2026-11-03 at 10:01 -0500, Omar Elghoul wrote:
quoted
Introduce new VFIO features for zPCI devices to provide FMB passthrough to
userspace.

Allow the user to enable or disable the FMB using the SET-only feature
VFIO_DEVICE_FEATURE_ZPCI_FMB_ENABLE. Likewise allow the user to read the
latest FMB using the GET-only feature VFIO_DEVICE_FEATURE_ZPCI_FMB_READ
in the case when the FMB is enabled.

Signed-off-by: Omar Elghoul <redacted>
Acked-by: Alex Williamson <alex@shazbot.org>
Reviewed-by: Matthew Rosato <mjrosato@linux.ibm.com>

Re: [PATCH v12 2/5] s390/pci: Improve function measurement state management

Message details and related content
On Tue, 2026-11-03 at 10:01 -0500, Omar Elghoul wrote:
quoted hunk
When a zPCI function is disabled and re-enabled, the firmware implicitly
disables function measurement while the kernel still believes it is
enabled, leading to a disconnect between the user-visible state and the
actual device state.

Add the statistics_on field to struct zpci_dev to track the desired FMB
state. Introduce the functions zpci_fmb_request_on()/_off(), which defer
to zpci_fmb_enable()/_disable() to perform the actual firmware change.
This decouples the enablement or disablement step from managing the
user's requested state.

Restore the state in pcibios_enable_device(), pcibios_disable_device(),
and zpci_reenable_device(). New devices start with measurement on by
default, unchanged from before.

Signed-off-by: Omar Elghoul <redacted>
---
 arch/s390/include/asm/pci.h |  7 +++--
 arch/s390/pci/pci.c         | 54 +++++++++++++++++++++++++++++++++----
 arch/s390/pci/pci_debug.c   |  4 +--
 3 files changed, 56 insertions(+), 9 deletions(-)
diff --git a/arch/s390/include/asm/pci.h b/arch/s390/include/asm/pci.h
index 88a125b92bdd..dd58c48ac2ed 100644
--- a/arch/s390/include/asm/pci.h
+++ b/arch/s390/include/asm/pci.h
@@ -208,6 +208,7 @@ struct zpci_dev {
 	struct zpci_fmb *fmb;
 	u16		fmb_update;	/* update interval */
 	u16		fmb_length;
+	u8		statistics_on	: 1; /* desired FMB enablement */
 
 	u8		version;
 	enum pci_bus_speed max_bus_speed;
@@ -349,8 +350,10 @@ int zpci_create_parent_msi_domain(struct zpci_bus *zbus);
 void zpci_remove_parent_msi_domain(struct zpci_bus *zbus);
 
 /* FMB */
-int zpci_fmb_enable_device(struct zpci_dev *);
-int zpci_fmb_disable_device(struct zpci_dev *);

126 more quoted lines

Yeah - this looks good to me!
Reviewed-by: Gerd Bayer <gbayer@linux.ibm.com>

Thank you for your work!

Re: [PATCH v12 3/5] s390/pci: Improve FMB memory management

Message details and related content
On Tue, 2026-11-03 at 10:01 -0500, Omar Elghoul wrote:
Refactor the FMB memory management code to use kzalloc() instead of a
dedicated kmem_cache. kzalloc() already guarantees adequate alignment
for power-of-two sizes, rendering the kmem_cache approach unnecessary.
This simplifies both the initialization and cleanup procedures.
TIL about the power-of-two alignment guarantee of kmalloc(). Nice!
Allocate the FMB once upon first enable and keep it for the lifetime of
the zdev instead of freeing it on every disable. Defer freeing the
buffer until device teardown in zpci_release_device(). Add the field
fmb_enabled to struct zpci_dev as the new FMB liveness bool, since a
non-null FMB pointer no longer implies FMB enablement.
Nit: maybe add "in hardware" to the last sentence?
quoted hunk
Signed-off-by: Omar Elghoul <redacted>
---
 arch/s390/include/asm/pci.h |  1 +
 arch/s390/pci/pci.c         | 59 +++++++++++++++++++++----------------
 arch/s390/pci/pci_debug.c   |  2 +-
 3 files changed, 35 insertions(+), 27 deletions(-)
diff --git a/arch/s390/include/asm/pci.h b/arch/s390/include/asm/pci.h
index dd58c48ac2ed..6d3c1b8ac75b 100644
--- a/arch/s390/include/asm/pci.h
+++ b/arch/s390/include/asm/pci.h
@@ -209,6 +209,7 @@ struct zpci_dev {
 	u16		fmb_update;	/* update interval */
 	u16		fmb_length;
 	u8		statistics_on	: 1; /* desired FMB enablement */
+	u8		fmb_enabled	: 1; /* actual FMB enablement */
Nit: Maybe reword the comment for fmb_enabled to "enabled in hardware"
- to better distinguish the layers where enablement happens?
quoted hunk
 
 	u8		version;
 	enum pci_bus_speed max_bus_speed;
diff --git a/arch/s390/pci/pci.c b/arch/s390/pci/pci.c
index f49736386eb7..f066a7ffc35c 100644
--- a/arch/s390/pci/pci.c
+++ b/arch/s390/pci/pci.c
@@ -62,8 +62,6 @@ EXPORT_SYMBOL_GPL(zpci_iomap_start);
 
 DEFINE_STATIC_KEY_FALSE(have_mio);
 
-static struct kmem_cache *zdev_fmb_cache;
-
 /* AEN structures that must be preserved over KVM module re-insertion */
 union zpci_sic_iib *zpci_aipb;
 EXPORT_SYMBOL_GPL(zpci_aipb);
@@ -205,13 +203,17 @@ int zpci_fmb_enable(struct zpci_dev *zdev)
 
 	lockdep_assert_held(&zdev->fmb_lock);
 
-	if (zdev->fmb || sizeof(*zdev->fmb) < zdev->fmb_length)
+	if (zdev->fmb_enabled || sizeof(*zdev->fmb) < zdev->fmb_length)
 		return -EINVAL;
 
-	zdev->fmb = kmem_cache_zalloc(zdev_fmb_cache, GFP_KERNEL);
-	if (!zdev->fmb)
-		return -ENOMEM;
-	WARN_ON((u64) zdev->fmb & 0xf);
+	if (!zdev->fmb) {
+		zdev->fmb = kzalloc_obj(*zdev->fmb, GFP_KERNEL);
+		if (!zdev->fmb)
+			return -ENOMEM;
+	} else {
+		/* reuse the same FMB buffer for as long as the zdev lives */
+		memset(zdev->fmb, 0, sizeof(*zdev->fmb));
+	}
 
 	/* reset software counters */
 	spin_lock_irqsave(&zdev->dom_lock, flags);
@@ -229,11 +231,12 @@ int zpci_fmb_enable(struct zpci_dev *zdev)

111 more quoted lines

Otherwise, this looks good - and I think Niklas did cover nicely the
Sashiko comments regarding implicit disablement of PCI functions and
the implication of that to fmb_enable.

Thanks,
Gerd
Keyboard shortcuts
hback out one level
jnext message in thread / next series-diff position
kprevious message in thread / previous series-diff position
ldrill in
Escclose help / fold thread tree
?toggle this help