[PATCH v4 00/16] File system wide monitoring

STALE1861d

Revision v4 of 7 in this series.

28 messages, 4 authors, 2021-08-03 · open the first message on its own page

[PATCH v4 00/16] File system wide monitoring

From: Gabriel Krisman Bertazi <hidden>
Date: 2021-07-20 16:08:50

Hi,

This is the fourth version of the FAN_FS_ERROR patches.  This applies
the feedback from last version (thanks Amir, Jan), in particular how we
report file handlers for errors not related to a file.  Now it is
reported as an invalid, empty fh.

This was tested with LTP for regressions, and also using the sample on
the last patch, with a corrupted image.  I can publish the bad image
upon request.

In addition, I wrote a man-page that can be pulled from:

  https://gitlab.collabora.com/krisman/man-pages.git -b fan-fs-error

And I'm sending it shortly for review.

I'm automating my tests in LTP next and will be sharing shortly.

I also pushed the full series to:

  https://gitlab.collabora.com/krisman/linux -b fanotify-notifications-single-slot

Thank you

Original cover letter
---------------------
Hi,

This series follow up on my previous proposal [1] to support file system
wide monitoring.  As suggested by Amir, this proposal drops the ring
buffer in favor of a single slot associated with each mark.  This
simplifies a bit the implementation, as you can see in the code.

As a reminder, This proposal is limited to an interface for
administrators to monitor the health of a file system, instead of a
generic inteface for file errors.  Therefore, this doesn't solve the
problem of writeback errors or the need to watch a specific subtree.

In comparison to the previous RFC, this implementation also drops the
per-fs data and location, and leave those as future extensions.

* Implementation

The feature is implemented on top of fanotify, as a new type of fanotify
mark, FAN_ERROR, which a file system monitoring tool can register to
receive error notifications.  When an error occurs a new notification is
generated, in addition followed by this info field:

 - FS generic data: A file system agnostic structure that has a generic
 error code and identifies the filesystem.  Basically, it let's
 userspace know something happened on a monitored filesystem.  Since
 only the first error is recorded since the last read, this also
 includes a counter of errors that happened since the last read.

* Testing

This was tested by watching notifications flowing from an intentionally
corrupted filesystem in different places.  In addition, other events
were watched in an attempt to detect regressions.

Is there a specific testsuite for fanotify I should be running?

* Patches

This patchset is divided as follows: Patch 1 through 5 are refactoring
to fsnotify/fanotify in preparation for FS_ERROR/FAN_ERROR; patch 6 and
7 implement the FS_ERROR API for filesystems to report error; patch 8
add support for FAN_ERROR in fanotify; Patch 9 is an example
implementation for ext4; patch 10 and 11 provide a sample userspace code
and documentation.

I also pushed the full series to:

  https://gitlab.collabora.com/krisman/linux -b fanotify-notifications-single-slot

[1] https://lwn.net/Articles/854545/
[2] https://lwn.net/Articles/856916/

Cc: Darrick J. Wong <djwong@kernel.org>
Cc: Theodore Ts'o <tytso@mit.edu>
Cc: Dave Chinner <david@fromorbit.com>
Cc: jack@suse.com
To: amir73il@gmail.com
Cc: dhowells@redhat.com
Cc: khazhy@google.com
Cc: linux-fsdevel@vger.kernel.org
Cc: linux-ext4@vger.kernel.org

Amir Goldstein (1):
  fsnotify: pass arguments of fsnotify() in struct fsnotify_event_info

Gabriel Krisman Bertazi (15):
  fsnotify: Don't insert unmergeable events in hashtable
  fanotify: Fold event size calculation to its own function
  fanotify: Split fsid check from other fid mode checks
  fsnotify: Reserve mark bits for backends
  fanotify: Split superblock marks out to a new cache
  inotify: Don't force FS_IN_IGNORED
  fsnotify: Add helper to detect overflow_event
  fsnotify: Add wrapper around fsnotify_add_event
  fsnotify: Support passing argument to insert callback on add_event
  fsnotify: Support FS_ERROR event type
  fsnotify: Introduce helpers to send error_events
  fanotify: Introduce FAN_FS_ERROR event
  ext4: Send notifications on error
  samples: Add fs error monitoring example
  docs: Document the FAN_FS_ERROR event

 .../admin-guide/filesystem-monitoring.rst     |  70 +++++
 Documentation/admin-guide/index.rst           |   1 +
 fs/ext4/super.c                               |   8 +
 fs/notify/fanotify/fanotify.c                 | 217 ++++++++++----
 fs/notify/fanotify/fanotify.h                 |  79 ++++-
 fs/notify/fanotify/fanotify_user.c            | 278 +++++++++++++++---
 fs/notify/fsnotify.c                          |  75 ++---
 fs/notify/inotify/inotify_fsnotify.c          |   2 +-
 fs/notify/inotify/inotify_user.c              |   6 +-
 fs/notify/notification.c                      |  16 +-
 include/linux/fanotify.h                      |   8 +-
 include/linux/fsnotify.h                      |  13 +
 include/linux/fsnotify_backend.h              | 140 +++++++--
 include/uapi/linux/fanotify.h                 |   8 +
 samples/Kconfig                               |   9 +
 samples/Makefile                              |   1 +
 samples/fanotify/Makefile                     |   5 +
 samples/fanotify/fs-monitor.c                 | 134 +++++++++
 18 files changed, 897 insertions(+), 173 deletions(-)
 create mode 100644 Documentation/admin-guide/filesystem-monitoring.rst
 create mode 100644 samples/fanotify/Makefile
 create mode 100644 samples/fanotify/fs-monitor.c

-- 
2.32.0

[PATCH v4 02/16] fanotify: Fold event size calculation to its own function

From: Gabriel Krisman Bertazi <hidden>
Date: 2021-07-20 16:08:22

Every time this function is invoked, it is immediately added to
FAN_EVENT_METADATA_LEN, since there is no need to just calculate the
length of info records. This minor clean up folds the rest of the
calculation into the function, which now operates in terms of events,
returning the size of the entire event, including metadata.

Reviewed-by: Amir Goldstein <amir73il@gmail.com>
Reviewed-by: Jan Kara <jack@suse.cz>
Signed-off-by: Gabriel Krisman Bertazi <redacted>

---
Changes since v1:
  - rebased on top of hashing patches
---
 fs/notify/fanotify/fanotify_user.c | 33 +++++++++++++++++-------------
 1 file changed, 19 insertions(+), 14 deletions(-)
diff --git a/fs/notify/fanotify/fanotify_user.c b/fs/notify/fanotify/fanotify_user.c
index 64864fb40b40..68a53d3534f8 100644
--- a/fs/notify/fanotify/fanotify_user.c
+++ b/fs/notify/fanotify/fanotify_user.c
@@ -117,17 +117,24 @@ static int fanotify_fid_info_len(int fh_len, int name_len)
 	return roundup(FANOTIFY_INFO_HDR_LEN + info_len, FANOTIFY_EVENT_ALIGN);
 }
 
-static int fanotify_event_info_len(unsigned int fid_mode,
-				   struct fanotify_event *event)
+static size_t fanotify_event_len(struct fanotify_event *event,
+				 unsigned int fid_mode)
 {
-	struct fanotify_info *info = fanotify_event_info(event);
-	int dir_fh_len = fanotify_event_dir_fh_len(event);
-	int fh_len = fanotify_event_object_fh_len(event);
-	int info_len = 0;
+	size_t event_len = FAN_EVENT_METADATA_LEN;
+	struct fanotify_info *info;
+	int dir_fh_len;
+	int fh_len;
 	int dot_len = 0;
 
+	if (!fid_mode)
+		return event_len;
+
+	info = fanotify_event_info(event);
+	dir_fh_len = fanotify_event_dir_fh_len(event);
+	fh_len = fanotify_event_object_fh_len(event);
+
 	if (dir_fh_len) {
-		info_len += fanotify_fid_info_len(dir_fh_len, info->name_len);
+		event_len += fanotify_fid_info_len(dir_fh_len, info->name_len);
 	} else if ((fid_mode & FAN_REPORT_NAME) && (event->mask & FAN_ONDIR)) {
 		/*
 		 * With group flag FAN_REPORT_NAME, if name was not recorded in
@@ -137,9 +144,9 @@ static int fanotify_event_info_len(unsigned int fid_mode,
 	}
 
 	if (fh_len)
-		info_len += fanotify_fid_info_len(fh_len, dot_len);
+		event_len += fanotify_fid_info_len(fh_len, dot_len);
 
-	return info_len;
+	return event_len;
 }
 
 /*
@@ -168,7 +175,7 @@ static void fanotify_unhash_event(struct fsnotify_group *group,
 static struct fanotify_event *get_one_event(struct fsnotify_group *group,
 					    size_t count)
 {
-	size_t event_size = FAN_EVENT_METADATA_LEN;
+	size_t event_size;
 	struct fanotify_event *event = NULL;
 	struct fsnotify_event *fsn_event;
 	unsigned int fid_mode = FAN_GROUP_FLAG(group, FANOTIFY_FID_BITS);
@@ -181,8 +188,7 @@ static struct fanotify_event *get_one_event(struct fsnotify_group *group,
 		goto out;
 
 	event = FANOTIFY_E(fsn_event);
-	if (fid_mode)
-		event_size += fanotify_event_info_len(fid_mode, event);
+	event_size = fanotify_event_len(event, fid_mode);
 
 	if (event_size > count) {
 		event = ERR_PTR(-EINVAL);
@@ -412,8 +418,7 @@ static ssize_t copy_event_to_user(struct fsnotify_group *group,
 
 	pr_debug("%s: group=%p event=%p\n", __func__, group, event);
 
-	metadata.event_len = FAN_EVENT_METADATA_LEN +
-				fanotify_event_info_len(fid_mode, event);
+	metadata.event_len = fanotify_event_len(event, fid_mode);
 	metadata.metadata_len = FAN_EVENT_METADATA_LEN;
 	metadata.vers = FANOTIFY_METADATA_VERSION;
 	metadata.reserved = 0;
-- 
2.32.0

[PATCH v4 01/16] fsnotify: Don't insert unmergeable events in hashtable

From: Gabriel Krisman Bertazi <hidden>
Date: 2021-07-20 16:08:22

Some events, like the overflow event, are not mergeable, so they are not
hashed.  But, when failing inside fsnotify_add_event for lack of space,
fsnotify_add_event() still calls the insert hook, which adds the
overflow event to the merge list.  Add a check to prevent any kind of
unmergeable event to be inserted in the hashtable.

Fixes: 94e00d28a680 ("fsnotify: use hash table for faster events merge")
Reviewed-by: Amir Goldstein <amir73il@gmail.com>
Reviewed-by: Jan Kara <jack@suse.cz>
Signed-off-by: Gabriel Krisman Bertazi <redacted>

---
Changes since v2:
  - Do check for hashed events inside the insert hook (Amir)
---
 fs/notify/fanotify/fanotify.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/fs/notify/fanotify/fanotify.c b/fs/notify/fanotify/fanotify.c
index 057abd2cf887..310246f8d3f1 100644
--- a/fs/notify/fanotify/fanotify.c
+++ b/fs/notify/fanotify/fanotify.c
@@ -702,6 +702,9 @@ static void fanotify_insert_event(struct fsnotify_group *group,
 
 	assert_spin_locked(&group->notification_lock);
 
+	if (!fanotify_is_hashed_event(event->mask))
+		return;
+
 	pr_debug("%s: group=%p event=%p bucket=%u\n", __func__,
 		 group, event, bucket);
 
@@ -779,8 +782,7 @@ static int fanotify_handle_event(struct fsnotify_group *group, u32 mask,
 
 	fsn_event = &event->fse;
 	ret = fsnotify_add_event(group, fsn_event, fanotify_merge,
-				 fanotify_is_hashed_event(mask) ?
-				 fanotify_insert_event : NULL);
+				 fanotify_insert_event);
 	if (ret) {
 		/* Permission events shouldn't be merged */
 		BUG_ON(ret == 1 && mask & FANOTIFY_PERM_EVENTS);
-- 
2.32.0

[PATCH v4 03/16] fanotify: Split fsid check from other fid mode checks

From: Gabriel Krisman Bertazi <hidden>
Date: 2021-07-20 16:08:25

FAN_FS_ERROR will require fsid, but not necessarily require the
filesystem to expose a file handle.  Split those checks into different
functions, so they can be used separately when setting up an event.

While there, update a comment about tmpfs having 0 fsid, which is no
longer true.

Reviewed-by: Amir Goldstein <amir73il@gmail.com>
Reviewed-by: Jan Kara <jack@suse.cz>
Signed-off-by: Gabriel Krisman Bertazi <redacted>

---
Changes since v2:
  - FAN_ERROR -> FAN_FS_ERROR (Amir)
  - Update comment (Amir)

Changes since v1:
  (Amir)
  - Sort hunks to simplify diff.
Changes since RFC:
  (Amir)
  - Rename fanotify_check_path_fsid -> fanotify_test_fsid.
  - Use dentry directly instead of path.
---
 fs/notify/fanotify/fanotify_user.c | 27 ++++++++++++++++++---------
 1 file changed, 18 insertions(+), 9 deletions(-)
diff --git a/fs/notify/fanotify/fanotify_user.c b/fs/notify/fanotify/fanotify_user.c
index 68a53d3534f8..67b18dfe0025 100644
--- a/fs/notify/fanotify/fanotify_user.c
+++ b/fs/notify/fanotify/fanotify_user.c
@@ -1192,16 +1192,15 @@ SYSCALL_DEFINE2(fanotify_init, unsigned int, flags, unsigned int, event_f_flags)
 	return fd;
 }
 
-/* Check if filesystem can encode a unique fid */
-static int fanotify_test_fid(struct path *path, __kernel_fsid_t *fsid)
+static int fanotify_test_fsid(struct dentry *dentry, __kernel_fsid_t *fsid)
 {
 	__kernel_fsid_t root_fsid;
 	int err;
 
 	/*
-	 * Make sure path is not in filesystem with zero fsid (e.g. tmpfs).
+	 * Make sure dentry is not of a filesystem with zero fsid (e.g. fuse).
 	 */
-	err = vfs_get_fsid(path->dentry, fsid);
+	err = vfs_get_fsid(dentry, fsid);
 	if (err)
 		return err;
 
@@ -1209,10 +1208,10 @@ static int fanotify_test_fid(struct path *path, __kernel_fsid_t *fsid)
 		return -ENODEV;
 
 	/*
-	 * Make sure path is not inside a filesystem subvolume (e.g. btrfs)
+	 * Make sure dentry is not of a filesystem subvolume (e.g. btrfs)
 	 * which uses a different fsid than sb root.
 	 */
-	err = vfs_get_fsid(path->dentry->d_sb->s_root, &root_fsid);
+	err = vfs_get_fsid(dentry->d_sb->s_root, &root_fsid);
 	if (err)
 		return err;
 
@@ -1220,6 +1219,12 @@ static int fanotify_test_fid(struct path *path, __kernel_fsid_t *fsid)
 	    root_fsid.val[1] != fsid->val[1])
 		return -EXDEV;
 
+	return 0;
+}
+
+/* Check if filesystem can encode a unique fid */
+static int fanotify_test_fid(struct dentry *dentry)
+{
 	/*
 	 * We need to make sure that the file system supports at least
 	 * encoding a file handle so user can use name_to_handle_at() to
@@ -1227,8 +1232,8 @@ static int fanotify_test_fid(struct path *path, __kernel_fsid_t *fsid)
 	 * objects. However, name_to_handle_at() requires that the
 	 * filesystem also supports decoding file handles.
 	 */
-	if (!path->dentry->d_sb->s_export_op ||
-	    !path->dentry->d_sb->s_export_op->fh_to_dentry)
+	if (!dentry->d_sb->s_export_op ||
+	    !dentry->d_sb->s_export_op->fh_to_dentry)
 		return -EOPNOTSUPP;
 
 	return 0;
@@ -1379,7 +1384,11 @@ static int do_fanotify_mark(int fanotify_fd, unsigned int flags, __u64 mask,
 	}
 
 	if (fid_mode) {
-		ret = fanotify_test_fid(&path, &__fsid);
+		ret = fanotify_test_fsid(path.dentry, &__fsid);
+		if (ret)
+			goto path_put_and_out;
+
+		ret = fanotify_test_fid(path.dentry);
 		if (ret)
 			goto path_put_and_out;
 
-- 
2.32.0

[PATCH v4 08/16] fsnotify: pass arguments of fsnotify() in struct fsnotify_event_info

From: Gabriel Krisman Bertazi <hidden>
Date: 2021-07-20 16:08:30

From: Amir Goldstein <amir73il@gmail.com>

There are a lot of arguments to fsnotify() and the handle_event() method.
Pass them in a const struct instead of on the argument list.

Apart from being more tidy, this helps with passing error reports to the
backend.  __fsnotify_parent() argument list was intentionally left
untouched, because its argument list is still short enough and because
most of the event info arguments are initialized inside
__fsnotify_parent().

Signed-off-by: Amir Goldstein <amir73il@gmail.com>
[ Fix kernelbot warnings]
[avoid __fsnotify version when not needed]
Signed-off-by: Gabriel Krisman Bertazi <redacted>
---
 fs/notify/fanotify/fanotify.c    | 59 ++++++++++++--------------
 fs/notify/fsnotify.c             | 73 +++++++++++++++++---------------
 include/linux/fsnotify_backend.h | 73 +++++++++++++++++++++++---------
 3 files changed, 120 insertions(+), 85 deletions(-)
diff --git a/fs/notify/fanotify/fanotify.c b/fs/notify/fanotify/fanotify.c
index c3eefe3f6494..6875d4d34c0c 100644
--- a/fs/notify/fanotify/fanotify.c
+++ b/fs/notify/fanotify/fanotify.c
@@ -253,21 +253,22 @@ static int fanotify_get_response(struct fsnotify_group *group,
  * been included within the event mask, but have not been explicitly
  * requested by the user, will not be present in the returned mask.
  */
-static u32 fanotify_group_event_mask(struct fsnotify_group *group,
-				     struct fsnotify_iter_info *iter_info,
-				     u32 event_mask, const void *data,
-				     int data_type, struct inode *dir)
+static u32 fanotify_group_event_mask(
+				struct fsnotify_group *group, u32 event_mask,
+				const struct fsnotify_event_info *event_info,
+				struct fsnotify_iter_info *iter_info)
 {
 	__u32 marks_mask = 0, marks_ignored_mask = 0;
 	__u32 test_mask, user_mask = FANOTIFY_OUTGOING_EVENTS |
 				     FANOTIFY_EVENT_FLAGS;
-	const struct path *path = fsnotify_data_path(data, data_type);
+	const struct path *path = fsnotify_event_info_path(event_info);
 	unsigned int fid_mode = FAN_GROUP_FLAG(group, FANOTIFY_FID_BITS);
 	struct fsnotify_mark *mark;
 	int type;
 
 	pr_debug("%s: report_mask=%x mask=%x data=%p data_type=%d\n",
-		 __func__, iter_info->report_mask, event_mask, data, data_type);
+		 __func__, iter_info->report_mask, event_mask,
+		 event_info->data, event_info->data_type);
 
 	if (!fid_mode) {
 		/* Do we have path to open a file descriptor? */
@@ -278,7 +279,7 @@ static u32 fanotify_group_event_mask(struct fsnotify_group *group,
 			return 0;
 	} else if (!(fid_mode & FAN_REPORT_FID)) {
 		/* Do we have a directory inode to report? */
-		if (!dir && !(event_mask & FS_ISDIR))
+		if (!event_info->dir && !(event_mask & FS_ISDIR))
 			return 0;
 	}
 
@@ -427,13 +428,13 @@ static int fanotify_encode_fh(struct fanotify_fh *fh, struct inode *inode,
  * FS_ATTRIB reports the child inode even if reported on a watched parent.
  * FS_CREATE reports the modified dir inode and not the created inode.
  */
-static struct inode *fanotify_fid_inode(u32 event_mask, const void *data,
-					int data_type, struct inode *dir)
+static struct inode *fanotify_fid_inode(u32 event_mask,
+				const struct fsnotify_event_info *event_info)
 {
 	if (event_mask & ALL_FSNOTIFY_DIRENT_EVENTS)
-		return dir;
+		return event_info->dir;
 
-	return fsnotify_data_inode(data, data_type);
+	return fsnotify_event_info_inode(event_info);
 }
 
 /*
@@ -444,18 +445,18 @@ static struct inode *fanotify_fid_inode(u32 event_mask, const void *data,
  * reported to parent.
  * Otherwise, do not report dir fid.
  */
-static struct inode *fanotify_dfid_inode(u32 event_mask, const void *data,
-					 int data_type, struct inode *dir)
+static struct inode *fanotify_dfid_inode(u32 event_mask,
+				const struct fsnotify_event_info *event_info)
 {
-	struct inode *inode = fsnotify_data_inode(data, data_type);
+	struct inode *inode = fsnotify_event_info_inode(event_info);
 
 	if (event_mask & ALL_FSNOTIFY_DIRENT_EVENTS)
-		return dir;
+		return event_info->dir;
 
 	if (S_ISDIR(inode->i_mode))
 		return inode;
 
-	return dir;
+	return event_info->dir;
 }
 
 static struct fanotify_event *fanotify_alloc_path_event(const struct path *path,
@@ -563,17 +564,17 @@ static struct fanotify_event *fanotify_alloc_name_event(struct inode *id,
 	return &fne->fae;
 }
 
-static struct fanotify_event *fanotify_alloc_event(struct fsnotify_group *group,
-						   u32 mask, const void *data,
-						   int data_type, struct inode *dir,
-						   const struct qstr *file_name,
-						   __kernel_fsid_t *fsid)
+static struct fanotify_event *fanotify_alloc_event(
+				struct fsnotify_group *group, u32 mask,
+				const struct fsnotify_event_info *event_info,
+				__kernel_fsid_t *fsid)
 {
 	struct fanotify_event *event = NULL;
 	gfp_t gfp = GFP_KERNEL_ACCOUNT;
-	struct inode *id = fanotify_fid_inode(mask, data, data_type, dir);
-	struct inode *dirid = fanotify_dfid_inode(mask, data, data_type, dir);
-	const struct path *path = fsnotify_data_path(data, data_type);
+	struct inode *id = fanotify_fid_inode(mask, event_info);
+	struct inode *dirid = fanotify_dfid_inode(mask, event_info);
+	const struct path *path = fsnotify_event_info_path(event_info);
+	const struct qstr *file_name = event_info->name;
 	unsigned int fid_mode = FAN_GROUP_FLAG(group, FANOTIFY_FID_BITS);
 	struct mem_cgroup *old_memcg;
 	struct inode *child = NULL;
@@ -712,9 +713,7 @@ static void fanotify_insert_event(struct fsnotify_group *group,
 }
 
 static int fanotify_handle_event(struct fsnotify_group *group, u32 mask,
-				 const void *data, int data_type,
-				 struct inode *dir,
-				 const struct qstr *file_name, u32 cookie,
+				 const struct fsnotify_event_info *event_info,
 				 struct fsnotify_iter_info *iter_info)
 {
 	int ret = 0;
@@ -744,8 +743,7 @@ static int fanotify_handle_event(struct fsnotify_group *group, u32 mask,
 
 	BUILD_BUG_ON(HWEIGHT32(ALL_FANOTIFY_EVENT_BITS) != 19);
 
-	mask = fanotify_group_event_mask(group, iter_info, mask, data,
-					 data_type, dir);
+	mask = fanotify_group_event_mask(group, mask, event_info, iter_info);
 	if (!mask)
 		return 0;
 
@@ -767,8 +765,7 @@ static int fanotify_handle_event(struct fsnotify_group *group, u32 mask,
 			return 0;
 	}
 
-	event = fanotify_alloc_event(group, mask, data, data_type, dir,
-				     file_name, &fsid);
+	event = fanotify_alloc_event(group, mask, event_info, &fsid);
 	ret = -ENOMEM;
 	if (unlikely(!event)) {
 		/*
diff --git a/fs/notify/fsnotify.c b/fs/notify/fsnotify.c
index 30d422b8c0fc..7c783c9df1dd 100644
--- a/fs/notify/fsnotify.c
+++ b/fs/notify/fsnotify.c
@@ -240,13 +240,11 @@ int __fsnotify_parent(struct dentry *dentry, __u32 mask, const void *data,
 EXPORT_SYMBOL_GPL(__fsnotify_parent);
 
 static int fsnotify_handle_inode_event(struct fsnotify_group *group,
-				       struct fsnotify_mark *inode_mark,
-				       u32 mask, const void *data, int data_type,
-				       struct inode *dir, const struct qstr *name,
-				       u32 cookie)
+			       struct fsnotify_mark *inode_mark, u32 mask,
+			       const struct fsnotify_event_info *event_info)
 {
-	const struct path *path = fsnotify_data_path(data, data_type);
-	struct inode *inode = fsnotify_data_inode(data, data_type);
+	const struct path *path = fsnotify_event_info_path(event_info);
+	struct inode *inode = fsnotify_event_info_inode(event_info);
 	const struct fsnotify_ops *ops = group->ops;
 
 	if (WARN_ON_ONCE(!ops->handle_inode_event))
@@ -260,16 +258,17 @@ static int fsnotify_handle_inode_event(struct fsnotify_group *group,
 	if (!(mask & inode_mark->mask & ALL_FSNOTIFY_EVENTS))
 		return 0;
 
-	return ops->handle_inode_event(inode_mark, mask, inode, dir, name, cookie);
+	return ops->handle_inode_event(inode_mark, mask, inode, event_info->dir,
+				       event_info->name, event_info->cookie);
 }
 
 static int fsnotify_handle_event(struct fsnotify_group *group, __u32 mask,
-				 const void *data, int data_type,
-				 struct inode *dir, const struct qstr *name,
-				 u32 cookie, struct fsnotify_iter_info *iter_info)
+				 const struct fsnotify_event_info *event_info,
+				 struct fsnotify_iter_info *iter_info)
 {
 	struct fsnotify_mark *inode_mark = fsnotify_iter_inode_mark(iter_info);
 	struct fsnotify_mark *parent_mark = fsnotify_iter_parent_mark(iter_info);
+	struct fsnotify_event_info child_event_info;
 	int ret;
 
 	if (WARN_ON_ONCE(fsnotify_iter_sb_mark(iter_info)) ||
@@ -284,8 +283,8 @@ static int fsnotify_handle_event(struct fsnotify_group *group, __u32 mask,
 		 * interested in this event?
 		 */
 		if (parent_mark->mask & FS_EVENT_ON_CHILD) {
-			ret = fsnotify_handle_inode_event(group, parent_mark, mask,
-							  data, data_type, dir, name, 0);
+			ret = fsnotify_handle_inode_event(group, parent_mark,
+							  mask, event_info);
 			if (ret)
 				return ret;
 		}
@@ -302,18 +301,22 @@ static int fsnotify_handle_event(struct fsnotify_group *group, __u32 mask,
 		 * The child watcher is expecting an event without a file name
 		 * and without the FS_EVENT_ON_CHILD flag.
 		 */
+		if (WARN_ON_ONCE(!event_info->inode))
+			return 0;
+
 		mask &= ~FS_EVENT_ON_CHILD;
-		dir = NULL;
-		name = NULL;
+		child_event_info = *event_info;
+		child_event_info.dir = NULL;
+		child_event_info.name = NULL;
+		event_info = &child_event_info;
 	}
 
-	return fsnotify_handle_inode_event(group, inode_mark, mask, data, data_type,
-					   dir, name, cookie);
+	return fsnotify_handle_inode_event(group, inode_mark, mask, event_info);
 }
 
-static int send_to_group(__u32 mask, const void *data, int data_type,
-			 struct inode *dir, const struct qstr *file_name,
-			 u32 cookie, struct fsnotify_iter_info *iter_info)
+static int send_to_group(__u32 mask,
+			 const struct fsnotify_event_info *event_info,
+			 struct fsnotify_iter_info *iter_info)
 {
 	struct fsnotify_group *group = NULL;
 	__u32 test_mask = (mask & ALL_FSNOTIFY_EVENTS);
@@ -351,18 +354,18 @@ static int send_to_group(__u32 mask, const void *data, int data_type,
 
 	pr_debug("%s: group=%p mask=%x marks_mask=%x marks_ignored_mask=%x data=%p data_type=%d dir=%p cookie=%d\n",
 		 __func__, group, mask, marks_mask, marks_ignored_mask,
-		 data, data_type, dir, cookie);
+		 event_info->data, event_info->data_type, event_info->dir,
+		 event_info->cookie);
 
 	if (!(test_mask & marks_mask & ~marks_ignored_mask))
 		return 0;
 
 	if (group->ops->handle_event) {
-		return group->ops->handle_event(group, mask, data, data_type, dir,
-						file_name, cookie, iter_info);
+		return group->ops->handle_event(group, mask, event_info,
+						iter_info);
 	}
 
-	return fsnotify_handle_event(group, mask, data, data_type, dir,
-				     file_name, cookie, iter_info);
+	return fsnotify_handle_event(group, mask, event_info, iter_info);
 }
 
 static struct fsnotify_mark *fsnotify_first_mark(struct fsnotify_mark_connector **connp)
@@ -448,21 +451,22 @@ static void fsnotify_iter_next(struct fsnotify_iter_info *iter_info)
  * in whatever means they feel necessary.
  *
  * @mask:	event type and flags
+ * Input args in struct fsnotify_event_info:
  * @data:	object that event happened on
  * @data_type:	type of object for fanotify_data_XXX() accessors
  * @dir:	optional directory associated with event -
- *		if @file_name is not NULL, this is the directory that
- *		@file_name is relative to
- * @file_name:	optional file name associated with event
+ *		if @name is not NULL, this is the directory that
+ *		@name is relative to
+ * @name:	optional file name associated with event
  * @inode:	optional inode associated with event -
  *		either @dir or @inode must be non-NULL.
  *		if both are non-NULL event may be reported to both.
  * @cookie:	inotify rename cookie
  */
-int fsnotify(__u32 mask, const void *data, int data_type, struct inode *dir,
-	     const struct qstr *file_name, struct inode *inode, u32 cookie)
+int __fsnotify(__u32 mask, const struct fsnotify_event_info *event_info)
 {
-	const struct path *path = fsnotify_data_path(data, data_type);
+	const struct path *path = fsnotify_event_info_path(event_info);
+	struct inode *inode = event_info->inode;
 	struct fsnotify_iter_info iter_info = {};
 	struct super_block *sb;
 	struct mount *mnt = NULL;
@@ -475,13 +479,13 @@ int fsnotify(__u32 mask, const void *data, int data_type, struct inode *dir,
 
 	if (!inode) {
 		/* Dirent event - report on TYPE_INODE to dir */
-		inode = dir;
+		inode = event_info->dir;
 	} else if (mask & FS_EVENT_ON_CHILD) {
 		/*
 		 * Event on child - report on TYPE_PARENT to dir if it is
 		 * watching children and on TYPE_INODE to child.
 		 */
-		parent = dir;
+		parent = event_info->dir;
 	}
 	sb = inode->i_sb;
 
@@ -538,8 +542,7 @@ int fsnotify(__u32 mask, const void *data, int data_type, struct inode *dir,
 	 * That's why this traversal is so complicated...
 	 */
 	while (fsnotify_iter_select_report_types(&iter_info)) {
-		ret = send_to_group(mask, data, data_type, dir, file_name,
-				    cookie, &iter_info);
+		ret = send_to_group(mask, event_info, &iter_info);
 
 		if (ret && (mask & ALL_FSNOTIFY_PERM_EVENTS))
 			goto out;
@@ -552,7 +555,7 @@ int fsnotify(__u32 mask, const void *data, int data_type, struct inode *dir,
 
 	return ret;
 }
-EXPORT_SYMBOL_GPL(fsnotify);
+EXPORT_SYMBOL_GPL(__fsnotify);
 
 static __init int fsnotify_init(void)
 {
diff --git a/include/linux/fsnotify_backend.h b/include/linux/fsnotify_backend.h
index 2b5fb9327a77..3c6fb43276ba 100644
--- a/include/linux/fsnotify_backend.h
+++ b/include/linux/fsnotify_backend.h
@@ -112,6 +112,28 @@ struct fsnotify_iter_info;
 
 struct mem_cgroup;
 
+/*
+ * Event info args passed to fsnotify() and to backends on handle_event():
+ * @data:	object that event happened on
+ * @data_type:	type of object for fanotify_data_XXX() accessors
+ * @dir:	optional directory associated with event -
+ *		if @name is not NULL, this is the directory that
+ *		@name is relative to
+ * @name:	optional file name associated with event
+ * @inode:	optional inode associated with event -
+ *		either @dir or @inode must be non-NULL.
+ *		if both are non-NULL event may be reported to both.
+ * @cookie:	inotify rename cookie
+ */
+struct fsnotify_event_info {
+	const void *data;
+	int data_type;
+	struct inode *dir;
+	const struct qstr *name;
+	struct inode *inode;
+	u32 cookie;
+};
+
 /*
  * Each group much define these ops.  The fsnotify infrastructure will call
  * these operations for each relevant group.
@@ -119,13 +141,7 @@ struct mem_cgroup;
  * handle_event - main call for a group to handle an fs event
  * @group:	group to notify
  * @mask:	event type and flags
- * @data:	object that event happened on
- * @data_type:	type of object for fanotify_data_XXX() accessors
- * @dir:	optional directory associated with event -
- *		if @file_name is not NULL, this is the directory that
- *		@file_name is relative to
- * @file_name:	optional file name associated with event
- * @cookie:	inotify rename cookie
+ * @event_info: information attached to the event
  * @iter_info:	array of marks from this group that are interested in the event
  *
  * handle_inode_event - simple variant of handle_event() for groups that only
@@ -147,8 +163,7 @@ struct mem_cgroup;
  */
 struct fsnotify_ops {
 	int (*handle_event)(struct fsnotify_group *group, u32 mask,
-			    const void *data, int data_type, struct inode *dir,
-			    const struct qstr *file_name, u32 cookie,
+			    const struct fsnotify_event_info *event_info,
 			    struct fsnotify_iter_info *iter_info);
 	int (*handle_inode_event)(struct fsnotify_mark *mark, u32 mask,
 			    struct inode *inode, struct inode *dir,
@@ -262,6 +277,12 @@ static inline struct inode *fsnotify_data_inode(const void *data, int data_type)
 	}
 }
 
+static inline struct inode *fsnotify_event_info_inode(
+				const struct fsnotify_event_info *event_info)
+{
+	return fsnotify_data_inode(event_info->data, event_info->data_type);
+}
+
 static inline const struct path *fsnotify_data_path(const void *data,
 						    int data_type)
 {
@@ -273,6 +294,12 @@ static inline const struct path *fsnotify_data_path(const void *data,
 	}
 }
 
+static inline const struct path *fsnotify_event_info_path(
+				const struct fsnotify_event_info *event_info)
+{
+	return fsnotify_data_path(event_info->data, event_info->data_type);
+}
+
 enum fsnotify_obj_type {
 	FSNOTIFY_OBJ_TYPE_INODE,
 	FSNOTIFY_OBJ_TYPE_PARENT,
@@ -422,11 +449,11 @@ struct fsnotify_mark {
 /* called from the vfs helpers */
 
 /* main fsnotify call to send events */
-extern int fsnotify(__u32 mask, const void *data, int data_type,
-		    struct inode *dir, const struct qstr *name,
-		    struct inode *inode, u32 cookie);
-extern int __fsnotify_parent(struct dentry *dentry, __u32 mask, const void *data,
-			   int data_type);
+extern int __fsnotify(__u32 mask,
+		      const struct fsnotify_event_info *event_info);
+extern int __fsnotify_parent(struct dentry *dentry, __u32 mask,
+			     const void *data, int data_type);
+
 extern void __fsnotify_inode_delete(struct inode *inode);
 extern void __fsnotify_vfsmount_delete(struct vfsmount *mnt);
 extern void fsnotify_sb_delete(struct super_block *sb);
@@ -605,16 +632,14 @@ static inline void fsnotify_init_event(struct fsnotify_event *event)
 }
 
 #else
-
-static inline int fsnotify(__u32 mask, const void *data, int data_type,
-			   struct inode *dir, const struct qstr *name,
-			   struct inode *inode, u32 cookie)
+static inline int __fsnotify(__u32 mask,
+			     const struct fsnotify_event_info *event_info)
 {
 	return 0;
 }
 
 static inline int __fsnotify_parent(struct dentry *dentry, __u32 mask,
-				  const void *data, int data_type)
+				    const void *data, int data_type)
 {
 	return 0;
 }
@@ -641,6 +666,16 @@ static inline void fsnotify_unmount_inodes(struct super_block *sb)
 
 #endif	/* CONFIG_FSNOTIFY */
 
+static inline int fsnotify(__u32 mask, const void *data, int data_type,
+			   struct inode *dir, const struct qstr *name,
+			   struct inode *inode, u32 cookie)
+{
+	return __fsnotify(mask, &(struct fsnotify_event_info) {
+			.data = (data), .data_type = (data_type), .dir = (dir),
+			.name = (name), .inode = (inode), .cookie = (cookie)
+		});
+}
+
 #endif	/* __KERNEL __ */
 
 #endif	/* __LINUX_FSNOTIFY_BACKEND_H */
-- 
2.32.0

[PATCH v4 09/16] fsnotify: Add wrapper around fsnotify_add_event

From: Gabriel Krisman Bertazi <hidden>
Date: 2021-07-20 16:08:39

fsnotify_add_event is growing in number of parameters, which is most
case are just passed a NULL pointer.  So, split out a new
fsnotify_insert_event function to clean things up for users who don't
need an insert hook.

Suggested-by: Amir Goldstein <amir73il@gmail.com>
Signed-off-by: Gabriel Krisman Bertazi <redacted>
---
 fs/notify/fanotify/fanotify.c        |  4 ++--
 fs/notify/inotify/inotify_fsnotify.c |  2 +-
 fs/notify/notification.c             | 12 ++++++------
 include/linux/fsnotify_backend.h     | 23 ++++++++++++++++-------
 4 files changed, 25 insertions(+), 16 deletions(-)
diff --git a/fs/notify/fanotify/fanotify.c b/fs/notify/fanotify/fanotify.c
index 6875d4d34c0c..93f96589ad1e 100644
--- a/fs/notify/fanotify/fanotify.c
+++ b/fs/notify/fanotify/fanotify.c
@@ -778,8 +778,8 @@ static int fanotify_handle_event(struct fsnotify_group *group, u32 mask,
 	}
 
 	fsn_event = &event->fse;
-	ret = fsnotify_add_event(group, fsn_event, fanotify_merge,
-				 fanotify_insert_event);
+	ret = fsnotify_insert_event(group, fsn_event, fanotify_merge,
+				    fanotify_insert_event);
 	if (ret) {
 		/* Permission events shouldn't be merged */
 		BUG_ON(ret == 1 && mask & FANOTIFY_PERM_EVENTS);
diff --git a/fs/notify/inotify/inotify_fsnotify.c b/fs/notify/inotify/inotify_fsnotify.c
index d1a64daa0171..a96582cbfad1 100644
--- a/fs/notify/inotify/inotify_fsnotify.c
+++ b/fs/notify/inotify/inotify_fsnotify.c
@@ -116,7 +116,7 @@ int inotify_handle_inode_event(struct fsnotify_mark *inode_mark, u32 mask,
 	if (len)
 		strcpy(event->name, name->name);
 
-	ret = fsnotify_add_event(group, fsn_event, inotify_merge, NULL);
+	ret = fsnotify_add_event(group, fsn_event, inotify_merge);
 	if (ret) {
 		/* Our event wasn't used in the end. Free it. */
 		fsnotify_destroy_event(group, fsn_event);
diff --git a/fs/notify/notification.c b/fs/notify/notification.c
index 32f45543b9c6..44bb10f50715 100644
--- a/fs/notify/notification.c
+++ b/fs/notify/notification.c
@@ -78,12 +78,12 @@ void fsnotify_destroy_event(struct fsnotify_group *group,
  * 2 if the event was not queued - either the queue of events has overflown
  *   or the group is shutting down.
  */
-int fsnotify_add_event(struct fsnotify_group *group,
-		       struct fsnotify_event *event,
-		       int (*merge)(struct fsnotify_group *,
-				    struct fsnotify_event *),
-		       void (*insert)(struct fsnotify_group *,
-				      struct fsnotify_event *))
+int fsnotify_insert_event(struct fsnotify_group *group,
+			  struct fsnotify_event *event,
+			  int (*merge)(struct fsnotify_group *,
+				       struct fsnotify_event *),
+			  void (*insert)(struct fsnotify_group *,
+					 struct fsnotify_event *))
 {
 	int ret = 0;
 	struct list_head *list = &group->notification_list;
diff --git a/include/linux/fsnotify_backend.h b/include/linux/fsnotify_backend.h
index 3c6fb43276ba..435982f88687 100644
--- a/include/linux/fsnotify_backend.h
+++ b/include/linux/fsnotify_backend.h
@@ -522,16 +522,25 @@ extern int fsnotify_fasync(int fd, struct file *file, int on);
 extern void fsnotify_destroy_event(struct fsnotify_group *group,
 				   struct fsnotify_event *event);
 /* attach the event to the group notification queue */
-extern int fsnotify_add_event(struct fsnotify_group *group,
-			      struct fsnotify_event *event,
-			      int (*merge)(struct fsnotify_group *,
-					   struct fsnotify_event *),
-			      void (*insert)(struct fsnotify_group *,
-					     struct fsnotify_event *));
+extern int fsnotify_insert_event(struct fsnotify_group *group,
+				 struct fsnotify_event *event,
+				 int (*merge)(struct fsnotify_group *,
+					      struct fsnotify_event *),
+				 void (*insert)(struct fsnotify_group *,
+						struct fsnotify_event *));
+
+static inline int fsnotify_add_event(struct fsnotify_group *group,
+				     struct fsnotify_event *event,
+				     int (*merge)(struct fsnotify_group *,
+						  struct fsnotify_event *))
+{
+	return fsnotify_insert_event(group, event, merge, NULL);
+}
+
 /* Queue overflow event to a notification group */
 static inline void fsnotify_queue_overflow(struct fsnotify_group *group)
 {
-	fsnotify_add_event(group, group->overflow_event, NULL, NULL);
+	fsnotify_add_event(group, group->overflow_event, NULL);
 }
 
 static inline bool fsnotify_is_overflow_event(u32 mask)
-- 
2.32.0

[PATCH v4 11/16] fsnotify: Support FS_ERROR event type

From: Gabriel Krisman Bertazi <hidden>
Date: 2021-07-20 16:08:52

Expose a new type of fsnotify event for filesystems to report errors for
userspace monitoring tools.  fanotify will send this type of
notification for FAN_FS_ERROR events.

Reviewed-by: Amir Goldstein <amir73il@gmail.com>
Reviewed-by: Jan Kara <jack@suse.cz>
Signed-off-by: Gabriel Krisman Bertazi <redacted>

---
Changes since v2:
  - FAN_ERROR->FAN_FS_ERROR (Amir)

Changes since v1:
  - Overload FS_ERROR with FS_IN_IGNORED
  - Implement support for this type on fsnotify_data_inode (Amir)
---
 include/linux/fsnotify_backend.h | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
diff --git a/include/linux/fsnotify_backend.h b/include/linux/fsnotify_backend.h
index ff2cf04c7496..33988fdf391a 100644
--- a/include/linux/fsnotify_backend.h
+++ b/include/linux/fsnotify_backend.h
@@ -42,6 +42,12 @@
 
 #define FS_UNMOUNT		0x00002000	/* inode on umount fs */
 #define FS_Q_OVERFLOW		0x00004000	/* Event queued overflowed */
+#define FS_ERROR		0x00008000	/* Filesystem Error (fanotify) */
+
+/*
+ * FS_IN_IGNORED overloads FS_ERROR.  It is only used internally by inotify
+ * which does not support FS_ERROR.
+ */
 #define FS_IN_IGNORED		0x00008000	/* last inotify event here */
 
 #define FS_OPEN_PERM		0x00010000	/* open event in an permission hook */
@@ -95,7 +101,8 @@
 #define ALL_FSNOTIFY_EVENTS (ALL_FSNOTIFY_DIRENT_EVENTS | \
 			     FS_EVENTS_POSS_ON_CHILD | \
 			     FS_DELETE_SELF | FS_MOVE_SELF | FS_DN_RENAME | \
-			     FS_UNMOUNT | FS_Q_OVERFLOW | FS_IN_IGNORED)
+			     FS_UNMOUNT | FS_Q_OVERFLOW | FS_IN_IGNORED | \
+			     FS_ERROR)
 
 /* Extra flags that may be reported with event or control handling of events */
 #define ALL_FSNOTIFY_FLAGS  (FS_EXCL_UNLINK | FS_ISDIR | FS_IN_ONESHOT | \
@@ -263,6 +270,12 @@ enum fsnotify_data_type {
 	FSNOTIFY_EVENT_NONE,
 	FSNOTIFY_EVENT_PATH,
 	FSNOTIFY_EVENT_INODE,
+	FSNOTIFY_EVENT_ERROR,
+};
+
+struct fs_error_report {
+	int error;
+	struct inode *inode;
 };
 
 static inline struct inode *fsnotify_data_inode(const void *data, int data_type)
@@ -272,6 +285,8 @@ static inline struct inode *fsnotify_data_inode(const void *data, int data_type)
 		return (struct inode *)data;
 	case FSNOTIFY_EVENT_PATH:
 		return d_inode(((const struct path *)data)->dentry);
+	case FSNOTIFY_EVENT_ERROR:
+		return ((struct fs_error_report *)data)->inode;
 	default:
 		return NULL;
 	}
-- 
2.32.0

[PATCH v4 16/16] docs: Document the FAN_FS_ERROR event

From: Gabriel Krisman Bertazi <hidden>
Date: 2021-07-20 16:08:59

Document the FAN_FS_ERROR event for user administrators and user space
developers.

Signed-off-by: Gabriel Krisman Bertazi <redacted>

---
Changes Since v3:
  - Move FAN_FS_ERROR notification into a subsection of the file.
Changes Since v2:
  - NTR
Changes since v1:
  - Drop references to location record
  - Explain that the inode field is optional
  - Explain we are reporting only the first error
---
 .../admin-guide/filesystem-monitoring.rst     | 70 +++++++++++++++++++
 Documentation/admin-guide/index.rst           |  1 +
 2 files changed, 71 insertions(+)
 create mode 100644 Documentation/admin-guide/filesystem-monitoring.rst
diff --git a/Documentation/admin-guide/filesystem-monitoring.rst b/Documentation/admin-guide/filesystem-monitoring.rst
new file mode 100644
index 000000000000..6c8982985a27
--- /dev/null
+++ b/Documentation/admin-guide/filesystem-monitoring.rst
@@ -0,0 +1,70 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+====================================
+File system Monitoring with fanotify
+====================================
+
+File system Error Reporting
+===========================
+
+fanotify supports the FAN_FS_ERROR mark for file system-wide error
+reporting.  It is meant to be used by file system health monitoring
+daemons who listen on that interface and take actions (notify sysadmin,
+start recovery) when a file system problem is detected by the kernel.
+
+By design, A FAN_FS_ERROR notification exposes sufficient information for a
+monitoring tool to know a problem in the file system has happened.  It
+doesn't necessarily provide a user space application with semantics to
+verify an IO operation was successfully executed.  That is outside of
+scope of this feature. Instead, it is only meant as a framework for
+early file system problem detection and reporting recovery tools.
+
+When a file system operation fails, it is common for dozens of kernel
+errors to cascade after the initial failure, hiding the original failure
+log, which is usually the most useful debug data to troubleshoot the
+problem.  For this reason, FAN_FS_ERROR only reports the first error that
+occurred since the last notification, and it simply counts addition
+errors.  This ensures that the most important piece of error information
+is never lost.
+
+FAN_FS_ERROR requires the fanotify group to be setup with the
+FAN_REPORT_FID flag.
+
+At the time of this writing, the only file system that emits FAN_FS_ERROR
+notifications is Ext4.
+
+A user space example code is provided at ``samples/fanotify/fs-monitor.c``.
+
+A FAN_FS_ERROR Notification has the following format::
+
+  [ Notification Metadata (Mandatory) ]
+  [ Generic Error Record  (Mandatory) ]
+  [ FID record            (Mandatory) ]
+
+Generic error record
+--------------------
+
+The generic error record provides enough information for a file system
+agnostic tool to learn about a problem in the file system, without
+providing any additional details about the problem.  This record is
+identified by ``struct fanotify_event_info_header.info_type`` being set
+to FAN_EVENT_INFO_TYPE_ERROR.
+
+  struct fanotify_event_info_error {
+	struct fanotify_event_info_header hdr;
+	__s32 error;
+	__u32 error_count;
+  };
+
+The `error` field identifies the type of error. `error_count` count
+tracks the number of errors that occurred and were suppressed to
+preserve the original error, since the last notification.
+
+FID record
+----------
+
+The FID record can be used to uniquely identify the inode that triggered
+the error through the combination of fsid and file handler.  A
+filesystem specific handler can use that information to attempt a
+recovery procedure.  Errors that are not related to an inode are
+reported against the root inode.
diff --git a/Documentation/admin-guide/index.rst b/Documentation/admin-guide/index.rst
index dc00afcabb95..1bedab498104 100644
--- a/Documentation/admin-guide/index.rst
+++ b/Documentation/admin-guide/index.rst
@@ -82,6 +82,7 @@ configure specific aspects of kernel behavior to your liking.
    edid
    efi-stub
    ext4
+   filesystem-monitoring
    nfs/index
    gpio/index
    highuid
-- 
2.32.0

[PATCH v4 10/16] fsnotify: Support passing argument to insert callback on add_event

From: Gabriel Krisman Bertazi <hidden>
Date: 2021-07-20 16:09:05

FAN_FS_ERROR requires some initialization to happen from inside the
insert hook.  This allows a user of fanotify_add_event to pass an
argument to be sent to the insert callback.

Reviewed-by: Amir Goldstein <amir73il@gmail.com>
Reviewed-by: Jan Kara <jack@suse.cz>
Signed-off-by: Gabriel Krisman Bertazi <redacted>
---
 fs/notify/fanotify/fanotify.c    | 5 +++--
 fs/notify/notification.c         | 6 ++++--
 include/linux/fsnotify_backend.h | 6 ++++--
 3 files changed, 11 insertions(+), 6 deletions(-)
diff --git a/fs/notify/fanotify/fanotify.c b/fs/notify/fanotify/fanotify.c
index 93f96589ad1e..912d120b9e48 100644
--- a/fs/notify/fanotify/fanotify.c
+++ b/fs/notify/fanotify/fanotify.c
@@ -695,7 +695,8 @@ static __kernel_fsid_t fanotify_get_fsid(struct fsnotify_iter_info *iter_info)
  * Add an event to hash table for faster merge.
  */
 static void fanotify_insert_event(struct fsnotify_group *group,
-				  struct fsnotify_event *fsn_event)
+				  struct fsnotify_event *fsn_event,
+				  const void *data)
 {
 	struct fanotify_event *event = FANOTIFY_E(fsn_event);
 	unsigned int bucket = fanotify_event_hash_bucket(group, event);
@@ -779,7 +780,7 @@ static int fanotify_handle_event(struct fsnotify_group *group, u32 mask,
 
 	fsn_event = &event->fse;
 	ret = fsnotify_insert_event(group, fsn_event, fanotify_merge,
-				    fanotify_insert_event);
+				    fanotify_insert_event, NULL);
 	if (ret) {
 		/* Permission events shouldn't be merged */
 		BUG_ON(ret == 1 && mask & FANOTIFY_PERM_EVENTS);
diff --git a/fs/notify/notification.c b/fs/notify/notification.c
index 44bb10f50715..206a17346ca6 100644
--- a/fs/notify/notification.c
+++ b/fs/notify/notification.c
@@ -83,7 +83,9 @@ int fsnotify_insert_event(struct fsnotify_group *group,
 			  int (*merge)(struct fsnotify_group *,
 				       struct fsnotify_event *),
 			  void (*insert)(struct fsnotify_group *,
-					 struct fsnotify_event *))
+					 struct fsnotify_event *,
+					 const void *),
+			  const void *insert_data)
 {
 	int ret = 0;
 	struct list_head *list = &group->notification_list;
@@ -121,7 +123,7 @@ int fsnotify_insert_event(struct fsnotify_group *group,
 	group->q_len++;
 	list_add_tail(&event->list, list);
 	if (insert)
-		insert(group, event);
+		insert(group, event, insert_data);
 	spin_unlock(&group->notification_lock);
 
 	wake_up(&group->notification_waitq);
diff --git a/include/linux/fsnotify_backend.h b/include/linux/fsnotify_backend.h
index 435982f88687..ff2cf04c7496 100644
--- a/include/linux/fsnotify_backend.h
+++ b/include/linux/fsnotify_backend.h
@@ -527,14 +527,16 @@ extern int fsnotify_insert_event(struct fsnotify_group *group,
 				 int (*merge)(struct fsnotify_group *,
 					      struct fsnotify_event *),
 				 void (*insert)(struct fsnotify_group *,
-						struct fsnotify_event *));
+						struct fsnotify_event *,
+						const void *),
+				 const void *insert_data);
 
 static inline int fsnotify_add_event(struct fsnotify_group *group,
 				     struct fsnotify_event *event,
 				     int (*merge)(struct fsnotify_group *,
 						  struct fsnotify_event *))
 {
-	return fsnotify_insert_event(group, event, merge, NULL);
+	return fsnotify_insert_event(group, event, merge, NULL, NULL);
 }
 
 /* Queue overflow event to a notification group */
-- 
2.32.0

[PATCH v4 12/16] fsnotify: Introduce helpers to send error_events

From: Gabriel Krisman Bertazi <hidden>
Date: 2021-07-20 16:09:09

Introduce helpers for filesystems interested in reporting FS_ERROR
events.  When notifying errors, the file system might not have an inode
to report on the error.  To support this, allow the caller to specify
the superblock to which the error applies.

Reviewed-by: Amir Goldstein <amir73il@gmail.com>
Reviewed-by: Jan Kara <jack@suse.cz>
Signed-off-by: Gabriel Krisman Bertazi <redacted>

---
Changes since v2:
  - Drop reference to s_fnotify_marks and guards (Amir)

Changes since v1:
  - Use the inode argument (Amir)
  - Protect s_fsnotify_marks with ifdef guard
---
 fs/notify/fsnotify.c             |  2 +-
 include/linux/fsnotify.h         | 13 +++++++++++++
 include/linux/fsnotify_backend.h |  1 +
 3 files changed, 15 insertions(+), 1 deletion(-)
diff --git a/fs/notify/fsnotify.c b/fs/notify/fsnotify.c
index 7c783c9df1dd..f66dff6e2d4e 100644
--- a/fs/notify/fsnotify.c
+++ b/fs/notify/fsnotify.c
@@ -487,7 +487,7 @@ int __fsnotify(__u32 mask, const struct fsnotify_event_info *event_info)
 		 */
 		parent = event_info->dir;
 	}
-	sb = inode->i_sb;
+	sb = event_info->sb ?: inode->i_sb;
 
 	/*
 	 * Optimization: srcu_read_lock() has a memory barrier which can
diff --git a/include/linux/fsnotify.h b/include/linux/fsnotify.h
index f8acddcf54fb..f118e20a9926 100644
--- a/include/linux/fsnotify.h
+++ b/include/linux/fsnotify.h
@@ -317,4 +317,17 @@ static inline void fsnotify_change(struct dentry *dentry, unsigned int ia_valid)
 		fsnotify_dentry(dentry, mask);
 }
 
+static inline int fsnotify_sb_error(struct super_block *sb, struct inode *inode,
+				    int error)
+{
+	struct fs_error_report report = {
+		.error = error,
+		.inode = inode,
+	};
+
+	return __fsnotify(FS_ERROR, &(struct fsnotify_event_info) {
+			.data = &report, .data_type = FSNOTIFY_EVENT_ERROR,
+			.sb = sb});
+}
+
 #endif	/* _LINUX_FS_NOTIFY_H */
diff --git a/include/linux/fsnotify_backend.h b/include/linux/fsnotify_backend.h
index 33988fdf391a..fa1303b8c599 100644
--- a/include/linux/fsnotify_backend.h
+++ b/include/linux/fsnotify_backend.h
@@ -138,6 +138,7 @@ struct fsnotify_event_info {
 	struct inode *dir;
 	const struct qstr *name;
 	struct inode *inode;
+	struct super_block *sb;
 	u32 cookie;
 };
 
-- 
2.32.0

[PATCH v4 06/16] inotify: Don't force FS_IN_IGNORED

From: Gabriel Krisman Bertazi <hidden>
Date: 2021-07-20 16:09:43

According to Amir:

"FS_IN_IGNORED is completely internal to inotify and there is no need
to set it in i_fsnotify_mask at all, so if we remove the bit from the
output of inotify_arg_to_mask() no functionality will change and we will
be able to overload the event bit for FS_ERROR."

This is done in preparation to overload FS_ERROR with the notification
mechanism in fanotify.

Suggested-by: Amir Goldstein <amir73il@gmail.com>
Reviewed-by: Amir Goldstein <amir73il@gmail.com>
Reviewed-by: Jan Kara <jack@suse.cz>
Signed-off-by: Gabriel Krisman Bertazi <redacted>
---
 fs/notify/inotify/inotify_user.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/fs/notify/inotify/inotify_user.c b/fs/notify/inotify/inotify_user.c
index 98f61b31745a..4d17be6dd58d 100644
--- a/fs/notify/inotify/inotify_user.c
+++ b/fs/notify/inotify/inotify_user.c
@@ -89,10 +89,10 @@ static inline __u32 inotify_arg_to_mask(struct inode *inode, u32 arg)
 	__u32 mask;
 
 	/*
-	 * Everything should accept their own ignored and should receive events
-	 * when the inode is unmounted.  All directories care about children.
+	 * Everything should receive events when the inode is unmounted.
+	 * All directories care about children.
 	 */
-	mask = (FS_IN_IGNORED | FS_UNMOUNT);
+	mask = (FS_UNMOUNT);
 	if (S_ISDIR(inode->i_mode))
 		mask |= FS_EVENT_ON_CHILD;
 
-- 
2.32.0

[PATCH v4 04/16] fsnotify: Reserve mark bits for backends

From: Gabriel Krisman Bertazi <hidden>
Date: 2021-07-20 16:12:05

Split out the final bits of struct fsnotify_mark->flags for use by a
backend.

Signed-off-by: Gabriel Krisman Bertazi <redacted>
---
 include/linux/fsnotify_backend.h | 19 ++++++++++++++++---
 1 file changed, 16 insertions(+), 3 deletions(-)
diff --git a/include/linux/fsnotify_backend.h b/include/linux/fsnotify_backend.h
index 1ce66748a2d2..9d5586445c65 100644
--- a/include/linux/fsnotify_backend.h
+++ b/include/linux/fsnotify_backend.h
@@ -363,6 +363,21 @@ struct fsnotify_mark_connector {
 	struct hlist_head list;
 };
 
+#define FSNOTIFY_MARK_FLAG(flag)	\
+static const unsigned int FSNOTIFY_MARK_FLAG_##flag = \
+	(1 << FSN_MARK_FL_BIT_##flag)
+
+enum fsnotify_mark_bits {
+	FSN_MARK_FL_BIT_IGNORED_SURV_MODIFY,
+	FSN_MARK_FL_BIT_ALIVE,
+	FSN_MARK_FL_BIT_ATTACHED,
+	FSN_MARK_PRIVATE_FLAGS,
+};
+
+FSNOTIFY_MARK_FLAG(IGNORED_SURV_MODIFY);
+FSNOTIFY_MARK_FLAG(ALIVE);
+FSNOTIFY_MARK_FLAG(ATTACHED);
+
 /*
  * A mark is simply an object attached to an in core inode which allows an
  * fsnotify listener to indicate they are either no longer interested in events
@@ -398,9 +413,7 @@ struct fsnotify_mark {
 	struct fsnotify_mark_connector *connector;
 	/* Events types to ignore [mark->lock, group->mark_mutex] */
 	__u32 ignored_mask;
-#define FSNOTIFY_MARK_FLAG_IGNORED_SURV_MODIFY	0x01
-#define FSNOTIFY_MARK_FLAG_ALIVE		0x02
-#define FSNOTIFY_MARK_FLAG_ATTACHED		0x04
+	/* Upper bits [31:PRIVATE_FLAGS] are reserved for backend usage */
 	unsigned int flags;		/* flags [mark->lock] */
 };
 
-- 
2.32.0

[PATCH v4 07/16] fsnotify: Add helper to detect overflow_event

From: Gabriel Krisman Bertazi <hidden>
Date: 2021-07-20 16:12:10

Similarly to fanotify_is_perm_event and friends, provide a helper
predicate to say whether a mask is of an overflow event.

Suggested-by: Amir Goldstein <amir73il@gmail.com>
Reviewed-by: Amir Goldstein <amir73il@gmail.com>
Reviewed-by: Jan Kara <jack@suse.cz>
Signed-off-by: Gabriel Krisman Bertazi <redacted>
---
 fs/notify/fanotify/fanotify.h    | 3 ++-
 include/linux/fsnotify_backend.h | 5 +++++
 2 files changed, 7 insertions(+), 1 deletion(-)
diff --git a/fs/notify/fanotify/fanotify.h b/fs/notify/fanotify/fanotify.h
index c548b96472dd..d4a562c2619f 100644
--- a/fs/notify/fanotify/fanotify.h
+++ b/fs/notify/fanotify/fanotify.h
@@ -338,7 +338,8 @@ static inline struct path *fanotify_event_path(struct fanotify_event *event)
  */
 static inline bool fanotify_is_hashed_event(u32 mask)
 {
-	return !fanotify_is_perm_event(mask) && !(mask & FS_Q_OVERFLOW);
+	return !(fanotify_is_perm_event(mask) ||
+		 fsnotify_is_overflow_event(mask));
 }
 
 static inline unsigned int fanotify_event_hash_bucket(
diff --git a/include/linux/fsnotify_backend.h b/include/linux/fsnotify_backend.h
index 9d5586445c65..2b5fb9327a77 100644
--- a/include/linux/fsnotify_backend.h
+++ b/include/linux/fsnotify_backend.h
@@ -507,6 +507,11 @@ static inline void fsnotify_queue_overflow(struct fsnotify_group *group)
 	fsnotify_add_event(group, group->overflow_event, NULL, NULL);
 }
 
+static inline bool fsnotify_is_overflow_event(u32 mask)
+{
+	return mask & FS_Q_OVERFLOW;
+}
+
 static inline bool fsnotify_notify_queue_is_empty(struct fsnotify_group *group)
 {
 	assert_spin_locked(&group->notification_lock);
-- 
2.32.0

[PATCH v4 05/16] fanotify: Split superblock marks out to a new cache

From: Gabriel Krisman Bertazi <hidden>
Date: 2021-07-20 16:12:10

FAN_FS_ERROR will require an error structure to be stored per mark.
But, since FAN_FS_ERROR doesn't apply to inode/mount marks, it should
suffice to only expose this information for superblock marks. Therefore,
wrap this kind of marks into a container and plumb it for the future.

Reviewed-by: Amir Goldstein <amir73il@gmail.com>
Signed-off-by: Gabriel Krisman Bertazi <redacted>

---
Changes since v2:
  - Move mark initialization to fanotify_alloc_mark (Amir)

Changes since v1:
  - Only extend superblock marks (Amir)
---
 fs/notify/fanotify/fanotify.c      | 10 ++++++--
 fs/notify/fanotify/fanotify.h      | 23 ++++++++++++++++++
 fs/notify/fanotify/fanotify_user.c | 38 ++++++++++++++++++++++++++++--
 3 files changed, 67 insertions(+), 4 deletions(-)
diff --git a/fs/notify/fanotify/fanotify.c b/fs/notify/fanotify/fanotify.c
index 310246f8d3f1..c3eefe3f6494 100644
--- a/fs/notify/fanotify/fanotify.c
+++ b/fs/notify/fanotify/fanotify.c
@@ -869,9 +869,15 @@ static void fanotify_freeing_mark(struct fsnotify_mark *mark,
 		dec_ucount(group->fanotify_data.ucounts, UCOUNT_FANOTIFY_MARKS);
 }
 
-static void fanotify_free_mark(struct fsnotify_mark *fsn_mark)
+static void fanotify_free_mark(struct fsnotify_mark *mark)
 {
-	kmem_cache_free(fanotify_mark_cache, fsn_mark);
+	if (mark->flags & FANOTIFY_MARK_FLAG_SB_MARK) {
+		struct fanotify_sb_mark *fa_mark = FANOTIFY_SB_MARK(mark);
+
+		kmem_cache_free(fanotify_sb_mark_cache, fa_mark);
+	} else {
+		kmem_cache_free(fanotify_mark_cache, mark);
+	}
 }
 
 const struct fsnotify_ops fanotify_fsnotify_ops = {
diff --git a/fs/notify/fanotify/fanotify.h b/fs/notify/fanotify/fanotify.h
index 4a5e555dc3d2..c548b96472dd 100644
--- a/fs/notify/fanotify/fanotify.h
+++ b/fs/notify/fanotify/fanotify.h
@@ -6,6 +6,7 @@
 #include <linux/hashtable.h>
 
 extern struct kmem_cache *fanotify_mark_cache;
+extern struct kmem_cache *fanotify_sb_mark_cache;
 extern struct kmem_cache *fanotify_fid_event_cachep;
 extern struct kmem_cache *fanotify_path_event_cachep;
 extern struct kmem_cache *fanotify_perm_event_cachep;
@@ -129,6 +130,28 @@ static inline void fanotify_info_copy_name(struct fanotify_info *info,
 	       name->name);
 }
 
+#define FANOTIFY_MARK_FLAG(flag) \
+static const unsigned int FANOTIFY_MARK_FLAG_##flag = \
+	(1 << FANOTIFY_MARK_FLAG_BIT_##flag)
+
+enum fanotify_mark_bits {
+	FANOTIFY_MARK_FLAG_BIT_SB_MARK = FSN_MARK_PRIVATE_FLAGS,
+};
+
+FANOTIFY_MARK_FLAG(SB_MARK);
+
+struct fanotify_sb_mark {
+	struct fsnotify_mark fsn_mark;
+};
+
+static inline
+struct fanotify_sb_mark *FANOTIFY_SB_MARK(struct fsnotify_mark *mark)
+{
+	WARN_ON(!(mark->flags & FANOTIFY_MARK_FLAG_SB_MARK));
+
+	return container_of(mark, struct fanotify_sb_mark, fsn_mark);
+}
+
 /*
  * Common structure for fanotify events. Concrete structs are allocated in
  * fanotify_handle_event() and freed when the information is retrieved by
diff --git a/fs/notify/fanotify/fanotify_user.c b/fs/notify/fanotify/fanotify_user.c
index 67b18dfe0025..0696f2121781 100644
--- a/fs/notify/fanotify/fanotify_user.c
+++ b/fs/notify/fanotify/fanotify_user.c
@@ -99,6 +99,7 @@ struct ctl_table fanotify_table[] = {
 extern const struct fsnotify_ops fanotify_fsnotify_ops;
 
 struct kmem_cache *fanotify_mark_cache __read_mostly;
+struct kmem_cache *fanotify_sb_mark_cache __read_mostly;
 struct kmem_cache *fanotify_fid_event_cachep __read_mostly;
 struct kmem_cache *fanotify_path_event_cachep __read_mostly;
 struct kmem_cache *fanotify_perm_event_cachep __read_mostly;
@@ -915,6 +916,38 @@ static __u32 fanotify_mark_add_to_mask(struct fsnotify_mark *fsn_mark,
 	return mask & ~oldmask;
 }
 
+static struct fsnotify_mark *fanotify_alloc_mark(struct fsnotify_group *group,
+						 unsigned int type)
+{
+	struct fanotify_sb_mark *sb_mark;
+	struct fsnotify_mark *mark;
+
+	switch (type) {
+	case FSNOTIFY_OBJ_TYPE_SB:
+		sb_mark = kmem_cache_zalloc(fanotify_sb_mark_cache, GFP_KERNEL);
+		if (!sb_mark)
+			return NULL;
+		mark = &sb_mark->fsn_mark;
+		break;
+
+	case FSNOTIFY_OBJ_TYPE_INODE:
+	case FSNOTIFY_OBJ_TYPE_PARENT:
+	case FSNOTIFY_OBJ_TYPE_VFSMOUNT:
+		mark = kmem_cache_alloc(fanotify_mark_cache, GFP_KERNEL);
+		break;
+	default:
+		WARN_ON(1);
+		return NULL;
+	}
+
+	fsnotify_init_mark(mark, group);
+
+	if (type == FSNOTIFY_OBJ_TYPE_SB)
+		mark->flags |= FANOTIFY_MARK_FLAG_SB_MARK;
+
+	return mark;
+}
+
 static struct fsnotify_mark *fanotify_add_new_mark(struct fsnotify_group *group,
 						   fsnotify_connp_t *connp,
 						   unsigned int type,
@@ -933,13 +966,12 @@ static struct fsnotify_mark *fanotify_add_new_mark(struct fsnotify_group *group,
 	    !inc_ucount(ucounts->ns, ucounts->uid, UCOUNT_FANOTIFY_MARKS))
 		return ERR_PTR(-ENOSPC);
 
-	mark = kmem_cache_alloc(fanotify_mark_cache, GFP_KERNEL);
+	mark = fanotify_alloc_mark(group, type);
 	if (!mark) {
 		ret = -ENOMEM;
 		goto out_dec_ucounts;
 	}
 
-	fsnotify_init_mark(mark, group);
 	ret = fsnotify_add_mark_locked(mark, connp, type, 0, fsid);
 	if (ret) {
 		fsnotify_put_mark(mark);
@@ -1497,6 +1529,8 @@ static int __init fanotify_user_setup(void)
 
 	fanotify_mark_cache = KMEM_CACHE(fsnotify_mark,
 					 SLAB_PANIC|SLAB_ACCOUNT);
+	fanotify_sb_mark_cache = KMEM_CACHE(fanotify_sb_mark,
+					    SLAB_PANIC|SLAB_ACCOUNT);
 	fanotify_fid_event_cachep = KMEM_CACHE(fanotify_fid_event,
 					       SLAB_PANIC);
 	fanotify_path_event_cachep = KMEM_CACHE(fanotify_path_event,
-- 
2.32.0

[PATCH v4 14/16] ext4: Send notifications on error

From: Gabriel Krisman Bertazi <hidden>
Date: 2021-07-20 16:12:10

Send a FS_ERROR message via fsnotify to a userspace monitoring tool
whenever a ext4 error condition is triggered.  This follows the existing
error conditions in ext4, so it is hooked to the ext4_error* functions.

It also follows the current dmesg reporting in the format.  The
filesystem message is composed mostly by the string that would be
otherwise printed in dmesg.

A new ext4 specific record format is exposed in the uapi, such that a
monitoring tool knows what to expect when listening errors of an ext4
filesystem.

Signed-off-by: Gabriel Krisman Bertazi <redacted>
Reviewed-by: Amir Goldstein <amir73il@gmail.com>
---
 fs/ext4/super.c | 8 ++++++++
 1 file changed, 8 insertions(+)
diff --git a/fs/ext4/super.c b/fs/ext4/super.c
index dfa09a277b56..b9ecd43678d7 100644
--- a/fs/ext4/super.c
+++ b/fs/ext4/super.c
@@ -46,6 +46,7 @@
 #include <linux/part_stat.h>
 #include <linux/kthread.h>
 #include <linux/freezer.h>
+#include <linux/fsnotify.h>
 
 #include "ext4.h"
 #include "ext4_extents.h"	/* Needed for trace points definition */
@@ -762,6 +763,8 @@ void __ext4_error(struct super_block *sb, const char *function,
 		       sb->s_id, function, line, current->comm, &vaf);
 		va_end(args);
 	}
+	fsnotify_sb_error(sb, NULL, error);
+
 	ext4_handle_error(sb, force_ro, error, 0, block, function, line);
 }
 
@@ -792,6 +795,8 @@ void __ext4_error_inode(struct inode *inode, const char *function,
 			       current->comm, &vaf);
 		va_end(args);
 	}
+	fsnotify_sb_error(inode->i_sb, inode, error);
+
 	ext4_handle_error(inode->i_sb, false, error, inode->i_ino, block,
 			  function, line);
 }
@@ -830,6 +835,8 @@ void __ext4_error_file(struct file *file, const char *function,
 			       current->comm, path, &vaf);
 		va_end(args);
 	}
+	fsnotify_sb_error(inode->i_sb, inode, EFSCORRUPTED);
+
 	ext4_handle_error(inode->i_sb, false, EFSCORRUPTED, inode->i_ino, block,
 			  function, line);
 }
@@ -897,6 +904,7 @@ void __ext4_std_error(struct super_block *sb, const char *function,
 		printk(KERN_CRIT "EXT4-fs error (device %s) in %s:%d: %s\n",
 		       sb->s_id, function, line, errstr);
 	}
+	fsnotify_sb_error(sb, sb->s_root->d_inode, errno);
 
 	ext4_handle_error(sb, false, -errno, 0, 0, function, line);
 }
-- 
2.32.0

[PATCH v4 13/16] fanotify: Introduce FAN_FS_ERROR event

From: Gabriel Krisman Bertazi <hidden>
Date: 2021-07-20 16:12:10

The FAN_FS_ERROR event is a new inode event used by filesystem wide
monitoring tools to receive notifications of type FS_ERROR_EVENT,
emitted directly by filesystems when a problem is detected.  The error
notification includes a generic error descriptor and a FID identifying
the file affected.

FID is sent for every FAN_FS_ERROR. Errors not linked to a regular inode
are reported against the root inode.

An error reporting structure is attached per-mark, and only a single
error can be stored at a time.  This is ok, since once an error occurs,
it is common for a stream of related errors to be reported.  We only log
accumulate the total of errors occurred since the last notification.

Signed-off-by: Gabriel Krisman Bertazi <redacted>

---
Changes since v3:
  - Convert WARN_ON to pr_warn (amir)
  - Remove unecessary READ/WRITE_ONCE (amir)
  - Alloc with GFP_KERNEL_ACCOUNT(amir)
  - Simplify flags on mark allocation (amir)
  - Avoid atomic set of error_count (amir)
  - Simplify rules when merging error_event (amir)
  - Allocate new error_event on get_one_event (amir)
  - Report superblock error with invalid FH (amir,jan)

Changes since v2:
  - Support and equire FID mode (amir)
  - Goto error path instead of early return (amir)
  - Simplify get_one_event (me)
  - Base merging on error_count
  - drop fanotify_queue_error_event

Changes since v1:
  - Pass dentry to fanotify_check_fsid (Amir)
  - FANOTIFY_EVENT_TYPE_ERROR -> FANOTIFY_EVENT_TYPE_FS_ERROR
  - Merge previous patch into it
  - Use a single slot
  - Move fanotify_mark.error_event definition to this commit
  - Rename FAN_ERROR -> FAN_FS_ERROR
  - Restrict FAN_FS_ERROR to FAN_MARK_FILESYSTEM
---
 fs/notify/fanotify/fanotify.c      | 137 ++++++++++++++++++----
 fs/notify/fanotify/fanotify.h      |  53 +++++++++
 fs/notify/fanotify/fanotify_user.c | 180 +++++++++++++++++++++++++++--
 include/linux/fanotify.h           |   8 +-
 include/uapi/linux/fanotify.h      |   8 ++
 5 files changed, 353 insertions(+), 33 deletions(-)
diff --git a/fs/notify/fanotify/fanotify.c b/fs/notify/fanotify/fanotify.c
index 912d120b9e48..477596b92bc5 100644
--- a/fs/notify/fanotify/fanotify.c
+++ b/fs/notify/fanotify/fanotify.c
@@ -335,24 +335,6 @@ static u32 fanotify_group_event_mask(
 	return test_mask & user_mask;
 }
 
-/*
- * Check size needed to encode fanotify_fh.
- *
- * Return size of encoded fh without fanotify_fh header.
- * Return 0 on failure to encode.
- */
-static int fanotify_encode_fh_len(struct inode *inode)
-{
-	int dwords = 0;
-
-	if (!inode)
-		return 0;
-
-	exportfs_encode_inode_fh(inode, NULL, &dwords, NULL);
-
-	return dwords << 2;
-}
-
 /*
  * Encode fanotify_fh.
  *
@@ -404,8 +386,12 @@ static int fanotify_encode_fh(struct fanotify_fh *fh, struct inode *inode,
 	fh->type = type;
 	fh->len = fh_len;
 
-	/* Mix fh into event merge key */
-	*hash ^= fanotify_hash_fh(fh);
+	/*
+	 * Mix fh into event merge key.  Hash might be NULL in case of
+	 * unhashed FID events (i.e. FAN_FS_ERROR).
+	 */
+	if (hash)
+		*hash ^= fanotify_hash_fh(fh);
 
 	return FANOTIFY_FH_HDR_LEN + fh_len;
 
@@ -420,6 +406,27 @@ static int fanotify_encode_fh(struct fanotify_fh *fh, struct inode *inode,
 	return 0;
 }
 
+#define FANOTIFY_EMPTY_FH_LEN	8
+/*
+ * Encode an empty fanotify_fh
+ *
+ * Empty FHs are used on FAN_FS_ERROR for errors not linked to any
+ * inode. fh needs to guarantee at least 8 bytes of inline space.
+ */
+static int fanotify_encode_empty_fh(struct fanotify_fh *fh, int max_len)
+{
+	if (max_len < FANOTIFY_EMPTY_FH_LEN || fh->flags)
+		return -EINVAL;
+
+	fh->type = FILEID_INVALID;
+	fh->len = FANOTIFY_EMPTY_FH_LEN;
+	fh->flags = 0;
+
+	memset(fh->buf, 0, FANOTIFY_EMPTY_FH_LEN);
+
+	return 0;
+}
+
 /*
  * The inode to use as identifier when reporting fid depends on the event.
  * Report the modified directory inode on dirent modification events.
@@ -691,6 +698,63 @@ static __kernel_fsid_t fanotify_get_fsid(struct fsnotify_iter_info *iter_info)
 	return fsid;
 }
 
+static int fanotify_merge_error_event(struct fsnotify_group *group,
+				      struct fsnotify_event *event)
+{
+	struct fanotify_event *fae = FANOTIFY_E(event);
+	struct fanotify_error_event *fee = FANOTIFY_EE(fae);
+
+	/*
+	 * When err_count > 0, the reporting slot is full.  Just account
+	 * the additional error and abort the insertion.
+	 */
+	if (fee->err_count) {
+		fee->err_count++;
+		return 1;
+	}
+
+	return 0;
+}
+
+static void fanotify_insert_error_event(struct fsnotify_group *group,
+					struct fsnotify_event *event,
+					const void *data)
+{
+	struct fanotify_event *fae = FANOTIFY_E(event);
+	const struct fsnotify_event_info *ei =
+		(struct fsnotify_event_info *) data;
+	const struct fs_error_report *report =
+		(struct fs_error_report *) ei->data;
+	struct inode *inode = report->inode;
+	struct fanotify_error_event *fee;
+	int fh_len;
+
+	/* This might be an unexpected type of event (i.e. overflow). */
+	if (!fanotify_is_error_event(fae->mask))
+		return;
+
+	fee = FANOTIFY_EE(fae);
+	fee->fae.type = FANOTIFY_EVENT_TYPE_FS_ERROR;
+	fee->error = report->error;
+	fee->fsid = fee->sb_mark->fsn_mark.connector->fsid;
+	fee->err_count = 1;
+
+	/*
+	 * Error reporting needs to happen in atomic context.  If this
+	 * inode's file handler is more than we initially predicted,
+	 * there is nothing better we can do than report the error with
+	 * a bad FH.
+	 */
+	fh_len = inode ? fanotify_encode_fh_len(inode) : FANOTIFY_EMPTY_FH_LEN;
+	if (fh_len > fee->max_fh_len)
+		return;
+
+	if (inode)
+		fanotify_encode_fh(&fee->object_fh, inode, fh_len, NULL, 0);
+	else
+		fanotify_encode_empty_fh(&fee->object_fh, fee->max_fh_len);
+}
+
 /*
  * Add an event to hash table for faster merge.
  */
@@ -741,8 +805,9 @@ static int fanotify_handle_event(struct fsnotify_group *group, u32 mask,
 	BUILD_BUG_ON(FAN_ONDIR != FS_ISDIR);
 	BUILD_BUG_ON(FAN_OPEN_EXEC != FS_OPEN_EXEC);
 	BUILD_BUG_ON(FAN_OPEN_EXEC_PERM != FS_OPEN_EXEC_PERM);
+	BUILD_BUG_ON(FAN_FS_ERROR != FS_ERROR);
 
-	BUILD_BUG_ON(HWEIGHT32(ALL_FANOTIFY_EVENT_BITS) != 19);
+	BUILD_BUG_ON(HWEIGHT32(ALL_FANOTIFY_EVENT_BITS) != 20);
 
 	mask = fanotify_group_event_mask(group, mask, event_info, iter_info);
 	if (!mask)
@@ -766,6 +831,18 @@ static int fanotify_handle_event(struct fsnotify_group *group, u32 mask,
 			return 0;
 	}
 
+	if (fanotify_is_error_event(mask)) {
+		struct fanotify_sb_mark *sb_mark =
+			FANOTIFY_SB_MARK(fsnotify_iter_sb_mark(iter_info));
+
+		ret = fsnotify_insert_event(group,
+					    &sb_mark->error_event->fae.fse,
+					    fanotify_merge_error_event,
+					    fanotify_insert_error_event,
+					    event_info);
+		goto finish;
+	}
+
 	event = fanotify_alloc_event(group, mask, event_info, &fsid);
 	ret = -ENOMEM;
 	if (unlikely(!event)) {
@@ -833,6 +910,20 @@ static void fanotify_free_name_event(struct fanotify_event *event)
 	kfree(FANOTIFY_NE(event));
 }
 
+static void fanotify_free_error_event(struct fanotify_event *event)
+{
+	struct fanotify_error_event *fee = FANOTIFY_EE(event);
+
+	/*
+	 * The event currently associated with the mark is freed by
+	 * fanotify_free_mark.
+	 */
+	if (fee->sb_mark->error_event == fee)
+		return;
+
+	kfree(fee);
+}
+
 static void fanotify_free_event(struct fsnotify_event *fsn_event)
 {
 	struct fanotify_event *event;
@@ -855,6 +946,9 @@ static void fanotify_free_event(struct fsnotify_event *fsn_event)
 	case FANOTIFY_EVENT_TYPE_OVERFLOW:
 		kfree(event);
 		break;
+	case FANOTIFY_EVENT_TYPE_FS_ERROR:
+		fanotify_free_error_event(event);
+		break;
 	default:
 		WARN_ON_ONCE(1);
 	}
@@ -872,6 +966,7 @@ static void fanotify_free_mark(struct fsnotify_mark *mark)
 	if (mark->flags & FANOTIFY_MARK_FLAG_SB_MARK) {
 		struct fanotify_sb_mark *fa_mark = FANOTIFY_SB_MARK(mark);
 
+		kfree(fa_mark->error_event);
 		kmem_cache_free(fanotify_sb_mark_cache, fa_mark);
 	} else {
 		kmem_cache_free(fanotify_mark_cache, mark);
diff --git a/fs/notify/fanotify/fanotify.h b/fs/notify/fanotify/fanotify.h
index d4a562c2619f..00dfec5f9a80 100644
--- a/fs/notify/fanotify/fanotify.h
+++ b/fs/notify/fanotify/fanotify.h
@@ -142,6 +142,7 @@ FANOTIFY_MARK_FLAG(SB_MARK);
 
 struct fanotify_sb_mark {
 	struct fsnotify_mark fsn_mark;
+	struct fanotify_error_event *error_event;
 };
 
 static inline
@@ -164,6 +165,7 @@ enum fanotify_event_type {
 	FANOTIFY_EVENT_TYPE_PATH,
 	FANOTIFY_EVENT_TYPE_PATH_PERM,
 	FANOTIFY_EVENT_TYPE_OVERFLOW, /* struct fanotify_event */
+	FANOTIFY_EVENT_TYPE_FS_ERROR, /* struct fanotify_error_event */
 	__FANOTIFY_EVENT_TYPE_NUM
 };
 
@@ -219,12 +221,37 @@ FANOTIFY_NE(struct fanotify_event *event)
 	return container_of(event, struct fanotify_name_event, fae);
 }
 
+struct fanotify_error_event {
+	struct fanotify_event fae;
+	s32 error;				/* Error reported by the Filesystem. */
+	u32 err_count;				/* Suppressed errors count */
+	__kernel_fsid_t fsid;			/* FSID this error refers to. */
+
+	struct fanotify_sb_mark *sb_mark;	/* Back reference to the mark. */
+	int max_fh_len;				/* Maximum object_fh buffer size. */
+
+	/*
+	 * object_fh is followed by a variable sized buffer, so it must
+	 * be the last element of this structure.
+	 */
+	struct fanotify_fh object_fh;
+};
+
+
+static inline struct fanotify_error_event *
+FANOTIFY_EE(struct fanotify_event *event)
+{
+	return container_of(event, struct fanotify_error_event, fae);
+}
+
 static inline __kernel_fsid_t *fanotify_event_fsid(struct fanotify_event *event)
 {
 	if (event->type == FANOTIFY_EVENT_TYPE_FID)
 		return &FANOTIFY_FE(event)->fsid;
 	else if (event->type == FANOTIFY_EVENT_TYPE_FID_NAME)
 		return &FANOTIFY_NE(event)->fsid;
+	else if (event->type == FANOTIFY_EVENT_TYPE_FS_ERROR)
+		return &FANOTIFY_EE(event)->fsid;
 	else
 		return NULL;
 }
@@ -236,6 +263,8 @@ static inline struct fanotify_fh *fanotify_event_object_fh(
 		return &FANOTIFY_FE(event)->object_fh;
 	else if (event->type == FANOTIFY_EVENT_TYPE_FID_NAME)
 		return fanotify_info_file_fh(&FANOTIFY_NE(event)->info);
+	else if (event->type == FANOTIFY_EVENT_TYPE_FS_ERROR)
+		return &FANOTIFY_EE(event)->object_fh;
 	else
 		return NULL;
 }
@@ -310,6 +339,11 @@ static inline struct fanotify_event *FANOTIFY_E(struct fsnotify_event *fse)
 	return container_of(fse, struct fanotify_event, fse);
 }
 
+static inline bool fanotify_is_error_event(u32 mask)
+{
+	return mask & FAN_FS_ERROR;
+}
+
 static inline bool fanotify_event_has_path(struct fanotify_event *event)
 {
 	return event->type == FANOTIFY_EVENT_TYPE_PATH ||
@@ -339,6 +373,7 @@ static inline struct path *fanotify_event_path(struct fanotify_event *event)
 static inline bool fanotify_is_hashed_event(u32 mask)
 {
 	return !(fanotify_is_perm_event(mask) ||
+		 fanotify_is_error_event(mask) ||
 		 fsnotify_is_overflow_event(mask));
 }
 
@@ -348,3 +383,21 @@ static inline unsigned int fanotify_event_hash_bucket(
 {
 	return event->hash & FANOTIFY_HTABLE_MASK;
 }
+
+/*
+ * Check size needed to encode fanotify_fh.
+ *
+ * Return size of encoded fh without fanotify_fh header.
+ * Return 0 on failure to encode.
+ */
+static inline int fanotify_encode_fh_len(struct inode *inode)
+{
+	int dwords = 0;
+
+	if (!inode)
+		return 0;
+
+	exportfs_encode_inode_fh(inode, NULL, &dwords, NULL);
+
+	return dwords << 2;
+}
diff --git a/fs/notify/fanotify/fanotify_user.c b/fs/notify/fanotify/fanotify_user.c
index 0696f2121781..bfc6bf6be197 100644
--- a/fs/notify/fanotify/fanotify_user.c
+++ b/fs/notify/fanotify/fanotify_user.c
@@ -107,6 +107,8 @@ struct kmem_cache *fanotify_perm_event_cachep __read_mostly;
 #define FANOTIFY_EVENT_ALIGN 4
 #define FANOTIFY_INFO_HDR_LEN \
 	(sizeof(struct fanotify_event_info_fid) + sizeof(struct file_handle))
+#define FANOTIFY_INFO_ERROR_LEN \
+	(sizeof(struct fanotify_event_info_error))
 
 static int fanotify_fid_info_len(int fh_len, int name_len)
 {
@@ -130,6 +132,9 @@ static size_t fanotify_event_len(struct fanotify_event *event,
 	if (!fid_mode)
 		return event_len;
 
+	if (fanotify_is_error_event(event->mask))
+		event_len += FANOTIFY_INFO_ERROR_LEN;
+
 	info = fanotify_event_info(event);
 	dir_fh_len = fanotify_event_dir_fh_len(event);
 	fh_len = fanotify_event_object_fh_len(event);
@@ -167,6 +172,90 @@ static void fanotify_unhash_event(struct fsnotify_group *group,
 	hlist_del_init(&event->merge_list);
 }
 
+static struct fanotify_error_event *fanotify_alloc_error_event(
+					struct fanotify_sb_mark *sb_mark,
+					int fh_len)
+{
+	struct fanotify_error_event *fee;
+	struct super_block *sb;
+
+	if (!fh_len) {
+		/*
+		 * The FH buffer size is predicted to be the same size
+		 * as the root inode file handler.  This should work for
+		 * file systems without variable sized FH.
+		 */
+		sb = container_of(sb_mark->fsn_mark.connector->obj,
+				  struct super_block, s_fsnotify_marks);
+		fh_len = fanotify_encode_fh_len(sb->s_root->d_inode);
+	}
+
+	fee = kzalloc(sizeof(*fee) + fh_len, GFP_KERNEL_ACCOUNT);
+	if (!fee)
+		return NULL;
+
+	fanotify_init_event(&fee->fae, 0, FS_ERROR);
+	fee->sb_mark = sb_mark;
+	fee->max_fh_len = fh_len;
+
+	return fee;
+}
+
+/*
+ * Replace a mark's error event with a new structure in preparation for
+ * it to be dequeued.  This is a bit annoying since we need to drop the
+ * lock, so another thread might just steal the event from us.
+ */
+static struct fanotify_event *fanotify_replace_fs_error_event(
+					struct fsnotify_group *group,
+					struct fanotify_event *fae)
+{
+	struct fanotify_error_event *new, *fee = FANOTIFY_EE(fae);
+	struct fanotify_sb_mark *sb_mark = fee->sb_mark;
+	struct fsnotify_event *fse;
+	int max_fh_len = fee->max_fh_len;
+	int fh_len = fanotify_event_object_fh_len(fae);
+
+	pr_debug("%s: event=%p\n", __func__, fae);
+
+	assert_spin_locked(&group->notification_lock);
+
+	spin_unlock(&group->notification_lock);
+	new = fanotify_alloc_error_event(sb_mark, fee->max_fh_len);
+	spin_lock(&group->notification_lock);
+
+	if (!new)
+		return ERR_PTR(-ENOMEM);
+
+	/*
+	 * Since we temporarily dropped the notification_lock, the event
+	 * might have been taken from under us and reported by another
+	 * reader.  Peek again prior to removal.
+	 *
+	 * Maybe this is not the same event we started handling.  But as
+	 * long as it is also a same size error event for the same
+	 * filesystem, it is safe to reuse the allocated memory.
+	 */
+	fse = fsnotify_peek_first_event(group);
+	if (!fse || !fanotify_is_error_event(FANOTIFY_E(fse)->mask))
+		goto fail;
+
+	fae = FANOTIFY_E(fse);
+	fee = FANOTIFY_EE(fae);
+	if (fee->sb_mark != sb_mark || max_fh_len != fee->max_fh_len  ||
+	    fh_len < fanotify_event_object_fh_len(fae))
+		goto fail;
+
+	sb_mark->error_event = new;
+
+	return fae;
+
+fail:
+	kfree(new);
+
+	return ERR_PTR(-EAGAIN);
+}
+
 /*
  * Get an fanotify notification event if one exists and is small
  * enough to fit in "count". Return an error pointer if the count
@@ -196,9 +285,20 @@ static struct fanotify_event *get_one_event(struct fsnotify_group *group,
 		goto out;
 	}
 
+	if (fanotify_is_error_event(event->mask)) {
+		/*
+		 * Recreate the error event ahead of dequeueing so we
+		 * don't need to handle a incorrectly dequeued event.
+		 */
+		event = fanotify_replace_fs_error_event(group, event);
+		if (IS_ERR(event))
+			goto out;
+	}
+
 	/*
-	 * Held the notification_lock the whole time, so this is the
-	 * same event we peeked above.
+	 * This might not be the same event peeked above, if
+	 * fanotify_recreate_fs_error raced with another reader. It is
+	 * guaranteed to succeed, though.
 	 */
 	fsnotify_remove_first_event(group);
 	if (fanotify_is_perm_event(event->mask))
@@ -310,6 +410,28 @@ static int process_access_response(struct fsnotify_group *group,
 	return -ENOENT;
 }
 
+static size_t copy_error_info_to_user(struct fanotify_event *event,
+				      char __user *buf, int count)
+{
+	struct fanotify_event_info_error info;
+	struct fanotify_error_event *fee = FANOTIFY_EE(event);
+
+	info.hdr.info_type = FAN_EVENT_INFO_TYPE_ERROR;
+	info.hdr.pad = 0;
+	info.hdr.len = sizeof(struct fanotify_event_info_error);
+
+	if (WARN_ON(count < info.hdr.len))
+		return -EFAULT;
+
+	info.error = fee->error;
+	info.error_count = fee->err_count;
+
+	if (copy_to_user(buf, &info, sizeof(info)))
+		return -EFAULT;
+
+	return info.hdr.len;
+}
+
 static int copy_info_to_user(__kernel_fsid_t *fsid, struct fanotify_fh *fh,
 			     int info_type, const char *name, size_t name_len,
 			     char __user *buf, size_t count)
@@ -468,6 +590,14 @@ static ssize_t copy_event_to_user(struct fsnotify_group *group,
 	if (f)
 		fd_install(fd, f);
 
+	if (fanotify_is_error_event(event->mask)) {
+		ret = copy_error_info_to_user(event, buf, count);
+		if (ret < 0)
+			goto out_close_fd;
+		buf += ret;
+		count -= ret;
+	}
+
 	/* Event info records order is: dir fid + name, child fid */
 	if (fanotify_event_dir_fh_len(event)) {
 		info_type = info->name_len ? FAN_EVENT_INFO_TYPE_DFID_NAME :
@@ -580,6 +710,8 @@ static ssize_t fanotify_read(struct file *file, char __user *buf,
 		event = get_one_event(group, count);
 		if (IS_ERR(event)) {
 			ret = PTR_ERR(event);
+			if (ret == -EAGAIN)
+				continue;
 			break;
 		}
 
@@ -993,7 +1125,9 @@ static int fanotify_add_mark(struct fsnotify_group *group,
 			     __kernel_fsid_t *fsid)
 {
 	struct fsnotify_mark *fsn_mark;
+	struct fanotify_sb_mark *sb_mark;
 	__u32 added;
+	int ret = 0;
 
 	mutex_lock(&group->mark_mutex);
 	fsn_mark = fsnotify_find_mark(connp, group);
@@ -1004,13 +1138,34 @@ static int fanotify_add_mark(struct fsnotify_group *group,
 			return PTR_ERR(fsn_mark);
 		}
 	}
+
+	/*
+	 * Error events are allocated per super-block mark, but only if
+	 * strictly needed (i.e. FAN_FS_ERROR was requested).
+	 */
+	if (type == FSNOTIFY_OBJ_TYPE_SB && !(flags & FAN_MARK_IGNORED_MASK) &&
+	    (mask & FAN_FS_ERROR)) {
+		sb_mark = FANOTIFY_SB_MARK(fsn_mark);
+
+		if (!sb_mark->error_event) {
+			sb_mark->error_event =
+				fanotify_alloc_error_event(sb_mark, 0);
+			if (!sb_mark->error_event) {
+				ret = -ENOMEM;
+				goto out;
+			}
+		}
+	}
+
 	added = fanotify_mark_add_to_mask(fsn_mark, mask, flags);
 	if (added & ~fsnotify_conn_mask(fsn_mark->connector))
 		fsnotify_recalc_mask(fsn_mark->connector);
+
+out:
 	mutex_unlock(&group->mark_mutex);
 
 	fsnotify_put_mark(fsn_mark);
-	return 0;
+	return ret;
 }
 
 static int fanotify_add_vfsmount_mark(struct fsnotify_group *group,
@@ -1382,14 +1537,14 @@ static int do_fanotify_mark(int fanotify_fd, unsigned int flags, __u64 mask,
 		goto fput_and_out;
 
 	/*
-	 * Events with data type inode do not carry enough information to report
-	 * event->fd, so we do not allow setting a mask for inode events unless
-	 * group supports reporting fid.
-	 * inode events are not supported on a mount mark, because they do not
-	 * carry enough information (i.e. path) to be filtered by mount point.
-	 */
+	* Events that do not carry enough information to report
+	* event->fd require a group that supports reporting fid.  Those
+	* events are not supported on a mount mark, because they do not
+	* carry enough information (i.e. path) to be filtered by mount
+	* point.
+	*/
 	fid_mode = FAN_GROUP_FLAG(group, FANOTIFY_FID_BITS);
-	if (mask & FANOTIFY_INODE_EVENTS &&
+	if (!(mask & FANOTIFY_FD_EVENTS) &&
 	    (!fid_mode || mark_type == FAN_MARK_MOUNT))
 		goto fput_and_out;
 
@@ -1427,6 +1582,11 @@ static int do_fanotify_mark(int fanotify_fd, unsigned int flags, __u64 mask,
 		fsid = &__fsid;
 	}
 
+	if (mask & FAN_FS_ERROR && mark_type != FAN_MARK_FILESYSTEM) {
+		ret = -EINVAL;
+		goto path_put_and_out;
+	}
+
 	/* inode held in place by reference to path; group by fget on fd */
 	if (mark_type == FAN_MARK_INODE)
 		inode = path.dentry->d_inode;
diff --git a/include/linux/fanotify.h b/include/linux/fanotify.h
index a16dbeced152..407f3f14bac4 100644
--- a/include/linux/fanotify.h
+++ b/include/linux/fanotify.h
@@ -81,9 +81,13 @@ extern struct ctl_table fanotify_table[]; /* for sysctl */
  */
 #define FANOTIFY_DIRENT_EVENTS	(FAN_MOVE | FAN_CREATE | FAN_DELETE)
 
-/* Events that can only be reported with data type FSNOTIFY_EVENT_INODE */
+/* Events that can be reported with event->fd */
+#define FANOTIFY_FD_EVENTS (FANOTIFY_PATH_EVENTS | FANOTIFY_PERM_EVENTS)
+
+/* Events that can only be reported to groups that support FID mode */
 #define FANOTIFY_INODE_EVENTS	(FANOTIFY_DIRENT_EVENTS | \
-				 FAN_ATTRIB | FAN_MOVE_SELF | FAN_DELETE_SELF)
+				 FAN_ATTRIB | FAN_MOVE_SELF | \
+				 FAN_DELETE_SELF | FAN_FS_ERROR)
 
 /* Events that user can request to be notified on */
 #define FANOTIFY_EVENTS		(FANOTIFY_PATH_EVENTS | \
diff --git a/include/uapi/linux/fanotify.h b/include/uapi/linux/fanotify.h
index fbf9c5c7dd59..80040a92e9d9 100644
--- a/include/uapi/linux/fanotify.h
+++ b/include/uapi/linux/fanotify.h
@@ -20,6 +20,7 @@
 #define FAN_OPEN_EXEC		0x00001000	/* File was opened for exec */
 
 #define FAN_Q_OVERFLOW		0x00004000	/* Event queued overflowed */
+#define FAN_FS_ERROR		0x00008000	/* Filesystem error */
 
 #define FAN_OPEN_PERM		0x00010000	/* File open in perm check */
 #define FAN_ACCESS_PERM		0x00020000	/* File accessed in perm check */
@@ -123,6 +124,7 @@ struct fanotify_event_metadata {
 #define FAN_EVENT_INFO_TYPE_FID		1
 #define FAN_EVENT_INFO_TYPE_DFID_NAME	2
 #define FAN_EVENT_INFO_TYPE_DFID	3
+#define FAN_EVENT_INFO_TYPE_ERROR	4
 
 /* Variable length info record following event metadata */
 struct fanotify_event_info_header {
@@ -148,6 +150,12 @@ struct fanotify_event_info_fid {
 	unsigned char handle[0];
 };
 
+struct fanotify_event_info_error {
+	struct fanotify_event_info_header hdr;
+	__s32 error;
+	__u32 error_count;
+};
+
 struct fanotify_response {
 	__s32 fd;
 	__u32 response;
-- 
2.32.0

[PATCH v4 15/16] samples: Add fs error monitoring example

From: Gabriel Krisman Bertazi <hidden>
Date: 2021-07-20 16:12:10

Introduce an example of a FAN_FS_ERROR fanotify user to track filesystem
errors.

Signed-off-by: Gabriel Krisman Bertazi <redacted>

---
Changes since v1:
  - minor fixes
---
 samples/Kconfig               |   9 +++
 samples/Makefile              |   1 +
 samples/fanotify/Makefile     |   5 ++
 samples/fanotify/fs-monitor.c | 134 ++++++++++++++++++++++++++++++++++
 4 files changed, 149 insertions(+)
 create mode 100644 samples/fanotify/Makefile
 create mode 100644 samples/fanotify/fs-monitor.c
diff --git a/samples/Kconfig b/samples/Kconfig
index b0503ef058d3..88353b8eac0b 100644
--- a/samples/Kconfig
+++ b/samples/Kconfig
@@ -120,6 +120,15 @@ config SAMPLE_CONNECTOR
 	  with it.
 	  See also Documentation/driver-api/connector.rst
 
+config SAMPLE_FANOTIFY_ERROR
+	bool "Build fanotify error monitoring sample"
+	depends on FANOTIFY
+	help
+	  When enabled, this builds an example code that uses the
+	  FAN_FS_ERROR fanotify mechanism to monitor filesystem
+	  errors.
+	  See also Documentation/admin-guide/filesystem-monitoring.rst.
+
 config SAMPLE_HIDRAW
 	bool "hidraw sample"
 	depends on CC_CAN_LINK && HEADERS_INSTALL
diff --git a/samples/Makefile b/samples/Makefile
index 087e0988ccc5..931a81847c48 100644
--- a/samples/Makefile
+++ b/samples/Makefile
@@ -5,6 +5,7 @@ subdir-$(CONFIG_SAMPLE_AUXDISPLAY)	+= auxdisplay
 subdir-$(CONFIG_SAMPLE_ANDROID_BINDERFS) += binderfs
 obj-$(CONFIG_SAMPLE_CONFIGFS)		+= configfs/
 obj-$(CONFIG_SAMPLE_CONNECTOR)		+= connector/
+obj-$(CONFIG_SAMPLE_FANOTIFY_ERROR)	+= fanotify/
 subdir-$(CONFIG_SAMPLE_HIDRAW)		+= hidraw
 obj-$(CONFIG_SAMPLE_HW_BREAKPOINT)	+= hw_breakpoint/
 obj-$(CONFIG_SAMPLE_KDB)		+= kdb/
diff --git a/samples/fanotify/Makefile b/samples/fanotify/Makefile
new file mode 100644
index 000000000000..e20db1bdde3b
--- /dev/null
+++ b/samples/fanotify/Makefile
@@ -0,0 +1,5 @@
+# SPDX-License-Identifier: GPL-2.0-only
+userprogs-always-y += fs-monitor
+
+userccflags += -I usr/include -Wall
+
diff --git a/samples/fanotify/fs-monitor.c b/samples/fanotify/fs-monitor.c
new file mode 100644
index 000000000000..ff74ba077f34
--- /dev/null
+++ b/samples/fanotify/fs-monitor.c
@@ -0,0 +1,134 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright 2021, Collabora Ltd.
+ */
+
+#define _GNU_SOURCE
+#include <errno.h>
+#include <err.h>
+#include <stdlib.h>
+#include <stdio.h>
+#include <fcntl.h>
+#include <sys/fanotify.h>
+#include <sys/types.h>
+#include <unistd.h>
+#include <sys/stat.h>
+#include <sys/types.h>
+
+#ifndef FAN_FS_ERROR
+#define FAN_FS_ERROR		0x00008000
+#define FAN_EVENT_INFO_TYPE_ERROR	4
+
+#define FILEID_INO32_GEN	1
+#define	FILEID_INVALID		0xff
+
+struct fanotify_event_info_error {
+	struct fanotify_event_info_header hdr;
+	__s32 error;
+	__u32 error_count;
+};
+#endif
+
+static void print_fh(struct file_handle *fh)
+{
+	int i;
+	uint32_t *h = (uint32_t *) fh->f_handle;
+
+	printf("\tfh: ");
+	for (i = 0; i < fh->handle_bytes; i++)
+		printf("%hhx", fh->f_handle[i]);
+	printf("\n");
+
+	printf("\tdecoded fh: ");
+	if (fh->handle_type == FILEID_INO32_GEN)
+		printf("inode=%u gen=%u\n", h[0], h[1]);
+	else if (fh->handle_type == FILEID_INVALID && !h[0] && !h[1])
+		printf("Type %d (Superblock error)\n", fh->handle_type);
+	else
+		printf("Type %d (Unknown)\n", fh->handle_type);
+
+}
+
+static void handle_notifications(char *buffer, int len)
+{
+	struct fanotify_event_metadata *metadata;
+	struct fanotify_event_info_error *error;
+	struct fanotify_event_info_fid *fid;
+	char *next;
+
+	for (metadata = (struct fanotify_event_metadata *) buffer;
+	     FAN_EVENT_OK(metadata, len);
+	     metadata = FAN_EVENT_NEXT(metadata, len)) {
+		next = (char *)metadata + metadata->event_len;
+		if (metadata->mask != FAN_FS_ERROR) {
+			printf("unexpected FAN MARK: %llx\n", metadata->mask);
+			goto next_event;
+		} else if (metadata->fd != FAN_NOFD) {
+			printf("Unexpected fd (!= FAN_NOFD)\n");
+			goto next_event;
+		}
+
+		printf("FAN_FS_ERROR found len=%d\n", metadata->event_len);
+
+		error = (struct fanotify_event_info_error *) (metadata+1);
+		if (error->hdr.info_type != FAN_EVENT_INFO_TYPE_ERROR) {
+			printf("unknown record: %d (Expecting TYPE_ERROR)\n",
+			       error->hdr.info_type);
+			goto next_event;
+		}
+
+		printf("\tGeneric Error Record: len=%d\n", error->hdr.len);
+		printf("\terror: %d\n", error->error);
+		printf("\terror_count: %d\n", error->error_count);
+
+		fid = (struct fanotify_event_info_fid *) (error + 1);
+		if ((char *) fid >= next) {
+			printf("Event doesn't have FID\n");
+			goto next_event;
+		}
+		printf("FID record found\n");
+
+		if (fid->hdr.info_type != FAN_EVENT_INFO_TYPE_FID) {
+			printf("unknown record: %d (Expecting TYPE_FID)\n",
+			       fid->hdr.info_type);
+			goto next_event;
+		}
+		printf("\tfsid: %x%x\n", fid->fsid.val[0], fid->fsid.val[1]);
+		print_fh((struct file_handle *) &fid->handle);
+
+next_event:
+		printf("---\n\n");
+	}
+}
+
+int main(int argc, char **argv)
+{
+	int fd;
+
+	char buffer[BUFSIZ];
+
+	if (argc < 2) {
+		printf("Missing path argument\n");
+		return 1;
+	}
+
+	fd = fanotify_init(FAN_CLASS_NOTIF|FAN_REPORT_FID, O_RDONLY);
+	if (fd < 0)
+		errx(1, "fanotify_init");
+
+	if (fanotify_mark(fd, FAN_MARK_ADD|FAN_MARK_FILESYSTEM,
+			  FAN_FS_ERROR, AT_FDCWD, argv[1])) {
+		errx(1, "fanotify_mark");
+	}
+
+	while (1) {
+		int n = read(fd, buffer, BUFSIZ);
+
+		if (n < 0)
+			errx(1, "read");
+
+		handle_notifications(buffer, n);
+	}
+
+	return 0;
+}
-- 
2.32.0

Re: [PATCH v4 09/16] fsnotify: Add wrapper around fsnotify_add_event

From: Amir Goldstein <amir73il@gmail.com>
Date: 2021-07-20 16:56:27

On Tue, Jul 20, 2021 at 7:00 PM Gabriel Krisman Bertazi
[off-list ref] wrote:
fsnotify_add_event is growing in number of parameters, which is most
case are just passed a NULL pointer.  So, split out a new
fsnotify_insert_event function to clean things up for users who don't
need an insert hook.

Suggested-by: Amir Goldstein <amir73il@gmail.com>
Signed-off-by: Gabriel Krisman Bertazi <redacted>
---
Reviewed-by: Amir Goldstein <amir73il@gmail.com>
quoted hunk
 fs/notify/fanotify/fanotify.c        |  4 ++--
 fs/notify/inotify/inotify_fsnotify.c |  2 +-
 fs/notify/notification.c             | 12 ++++++------
 include/linux/fsnotify_backend.h     | 23 ++++++++++++++++-------
 4 files changed, 25 insertions(+), 16 deletions(-)
diff --git a/fs/notify/fanotify/fanotify.c b/fs/notify/fanotify/fanotify.c
index 6875d4d34c0c..93f96589ad1e 100644
--- a/fs/notify/fanotify/fanotify.c
+++ b/fs/notify/fanotify/fanotify.c
@@ -778,8 +778,8 @@ static int fanotify_handle_event(struct fsnotify_group *group, u32 mask,
        }

        fsn_event = &event->fse;
-       ret = fsnotify_add_event(group, fsn_event, fanotify_merge,
-                                fanotify_insert_event);
+       ret = fsnotify_insert_event(group, fsn_event, fanotify_merge,
+                                   fanotify_insert_event);
        if (ret) {
                /* Permission events shouldn't be merged */
                BUG_ON(ret == 1 && mask & FANOTIFY_PERM_EVENTS);
diff --git a/fs/notify/inotify/inotify_fsnotify.c b/fs/notify/inotify/inotify_fsnotify.c
index d1a64daa0171..a96582cbfad1 100644
--- a/fs/notify/inotify/inotify_fsnotify.c
+++ b/fs/notify/inotify/inotify_fsnotify.c
@@ -116,7 +116,7 @@ int inotify_handle_inode_event(struct fsnotify_mark *inode_mark, u32 mask,
        if (len)
                strcpy(event->name, name->name);

-       ret = fsnotify_add_event(group, fsn_event, inotify_merge, NULL);
+       ret = fsnotify_add_event(group, fsn_event, inotify_merge);
        if (ret) {
                /* Our event wasn't used in the end. Free it. */
                fsnotify_destroy_event(group, fsn_event);
diff --git a/fs/notify/notification.c b/fs/notify/notification.c
index 32f45543b9c6..44bb10f50715 100644
--- a/fs/notify/notification.c
+++ b/fs/notify/notification.c
@@ -78,12 +78,12 @@ void fsnotify_destroy_event(struct fsnotify_group *group,
  * 2 if the event was not queued - either the queue of events has overflown
  *   or the group is shutting down.
  */
-int fsnotify_add_event(struct fsnotify_group *group,
-                      struct fsnotify_event *event,
-                      int (*merge)(struct fsnotify_group *,
-                                   struct fsnotify_event *),
-                      void (*insert)(struct fsnotify_group *,
-                                     struct fsnotify_event *))
+int fsnotify_insert_event(struct fsnotify_group *group,
+                         struct fsnotify_event *event,
+                         int (*merge)(struct fsnotify_group *,
+                                      struct fsnotify_event *),
+                         void (*insert)(struct fsnotify_group *,
+                                        struct fsnotify_event *))
 {
        int ret = 0;
        struct list_head *list = &group->notification_list;
diff --git a/include/linux/fsnotify_backend.h b/include/linux/fsnotify_backend.h
index 3c6fb43276ba..435982f88687 100644
--- a/include/linux/fsnotify_backend.h
+++ b/include/linux/fsnotify_backend.h
@@ -522,16 +522,25 @@ extern int fsnotify_fasync(int fd, struct file *file, int on);
 extern void fsnotify_destroy_event(struct fsnotify_group *group,
                                   struct fsnotify_event *event);
 /* attach the event to the group notification queue */
-extern int fsnotify_add_event(struct fsnotify_group *group,
-                             struct fsnotify_event *event,
-                             int (*merge)(struct fsnotify_group *,
-                                          struct fsnotify_event *),
-                             void (*insert)(struct fsnotify_group *,
-                                            struct fsnotify_event *));
+extern int fsnotify_insert_event(struct fsnotify_group *group,
+                                struct fsnotify_event *event,
+                                int (*merge)(struct fsnotify_group *,
+                                             struct fsnotify_event *),
+                                void (*insert)(struct fsnotify_group *,
+                                               struct fsnotify_event *));
+
+static inline int fsnotify_add_event(struct fsnotify_group *group,
+                                    struct fsnotify_event *event,
+                                    int (*merge)(struct fsnotify_group *,
+                                                 struct fsnotify_event *))
+{
+       return fsnotify_insert_event(group, event, merge, NULL);
+}
+
 /* Queue overflow event to a notification group */
 static inline void fsnotify_queue_overflow(struct fsnotify_group *group)
 {
-       fsnotify_add_event(group, group->overflow_event, NULL, NULL);
+       fsnotify_add_event(group, group->overflow_event, NULL);
 }

 static inline bool fsnotify_is_overflow_event(u32 mask)
--
2.32.0

Re: [PATCH v4 15/16] samples: Add fs error monitoring example

From: Amir Goldstein <amir73il@gmail.com>
Date: 2021-07-20 17:05:32

On Tue, Jul 20, 2021 at 7:00 PM Gabriel Krisman Bertazi
[off-list ref] wrote:
Introduce an example of a FAN_FS_ERROR fanotify user to track filesystem
errors.

Signed-off-by: Gabriel Krisman Bertazi <redacted>
Reviewed-by: Amir Goldstein <amir73il@gmail.com>

except for one small nit
quoted hunk
---
Changes since v1:
  - minor fixes
---
 samples/Kconfig               |   9 +++
 samples/Makefile              |   1 +
 samples/fanotify/Makefile     |   5 ++
 samples/fanotify/fs-monitor.c | 134 ++++++++++++++++++++++++++++++++++
 4 files changed, 149 insertions(+)
 create mode 100644 samples/fanotify/Makefile
 create mode 100644 samples/fanotify/fs-monitor.c
diff --git a/samples/Kconfig b/samples/Kconfig
index b0503ef058d3..88353b8eac0b 100644
--- a/samples/Kconfig
+++ b/samples/Kconfig
@@ -120,6 +120,15 @@ config SAMPLE_CONNECTOR
          with it.
          See also Documentation/driver-api/connector.rst

+config SAMPLE_FANOTIFY_ERROR
+       bool "Build fanotify error monitoring sample"
+       depends on FANOTIFY
+       help
+         When enabled, this builds an example code that uses the
+         FAN_FS_ERROR fanotify mechanism to monitor filesystem
+         errors.
+         See also Documentation/admin-guide/filesystem-monitoring.rst.
+
 config SAMPLE_HIDRAW
        bool "hidraw sample"
        depends on CC_CAN_LINK && HEADERS_INSTALL
diff --git a/samples/Makefile b/samples/Makefile
index 087e0988ccc5..931a81847c48 100644
--- a/samples/Makefile
+++ b/samples/Makefile
@@ -5,6 +5,7 @@ subdir-$(CONFIG_SAMPLE_AUXDISPLAY)      += auxdisplay
 subdir-$(CONFIG_SAMPLE_ANDROID_BINDERFS) += binderfs
 obj-$(CONFIG_SAMPLE_CONFIGFS)          += configfs/
 obj-$(CONFIG_SAMPLE_CONNECTOR)         += connector/
+obj-$(CONFIG_SAMPLE_FANOTIFY_ERROR)    += fanotify/
 subdir-$(CONFIG_SAMPLE_HIDRAW)         += hidraw
 obj-$(CONFIG_SAMPLE_HW_BREAKPOINT)     += hw_breakpoint/
 obj-$(CONFIG_SAMPLE_KDB)               += kdb/
diff --git a/samples/fanotify/Makefile b/samples/fanotify/Makefile
new file mode 100644
index 000000000000..e20db1bdde3b
--- /dev/null
+++ b/samples/fanotify/Makefile
@@ -0,0 +1,5 @@
+# SPDX-License-Identifier: GPL-2.0-only
+userprogs-always-y += fs-monitor
+
+userccflags += -I usr/include -Wall
+
diff --git a/samples/fanotify/fs-monitor.c b/samples/fanotify/fs-monitor.c
new file mode 100644
index 000000000000..ff74ba077f34
--- /dev/null
+++ b/samples/fanotify/fs-monitor.c
@@ -0,0 +1,134 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright 2021, Collabora Ltd.
+ */
+
+#define _GNU_SOURCE
+#include <errno.h>
+#include <err.h>
+#include <stdlib.h>
+#include <stdio.h>
+#include <fcntl.h>
+#include <sys/fanotify.h>
+#include <sys/types.h>
+#include <unistd.h>
+#include <sys/stat.h>
+#include <sys/types.h>
+
+#ifndef FAN_FS_ERROR
+#define FAN_FS_ERROR           0x00008000
+#define FAN_EVENT_INFO_TYPE_ERROR      4
+
+#define FILEID_INO32_GEN       1
+#define        FILEID_INVALID          0xff
If you install the kernel uapi header and try to build this
sample it will fail because FAN_FS_ERROR will be
defined but FILEID_INO32_GEN and FILEID_INVALID
will not be defined.

Thanks,
Amir.
+
+struct fanotify_event_info_error {
+       struct fanotify_event_info_header hdr;
+       __s32 error;
+       __u32 error_count;
+};
+#endif
+
+static void print_fh(struct file_handle *fh)
+{
+       int i;
+       uint32_t *h = (uint32_t *) fh->f_handle;
+
+       printf("\tfh: ");
+       for (i = 0; i < fh->handle_bytes; i++)
+               printf("%hhx", fh->f_handle[i]);
+       printf("\n");
+
+       printf("\tdecoded fh: ");
+       if (fh->handle_type == FILEID_INO32_GEN)
+               printf("inode=%u gen=%u\n", h[0], h[1]);
+       else if (fh->handle_type == FILEID_INVALID && !h[0] && !h[1])
+               printf("Type %d (Superblock error)\n", fh->handle_type);
+       else
+               printf("Type %d (Unknown)\n", fh->handle_type);
+
+}
+
+static void handle_notifications(char *buffer, int len)
+{
+       struct fanotify_event_metadata *metadata;
+       struct fanotify_event_info_error *error;
+       struct fanotify_event_info_fid *fid;
+       char *next;
+
+       for (metadata = (struct fanotify_event_metadata *) buffer;
+            FAN_EVENT_OK(metadata, len);
+            metadata = FAN_EVENT_NEXT(metadata, len)) {
+               next = (char *)metadata + metadata->event_len;
+               if (metadata->mask != FAN_FS_ERROR) {
+                       printf("unexpected FAN MARK: %llx\n", metadata->mask);
+                       goto next_event;
+               } else if (metadata->fd != FAN_NOFD) {
+                       printf("Unexpected fd (!= FAN_NOFD)\n");
+                       goto next_event;
+               }
+
+               printf("FAN_FS_ERROR found len=%d\n", metadata->event_len);
+
+               error = (struct fanotify_event_info_error *) (metadata+1);
+               if (error->hdr.info_type != FAN_EVENT_INFO_TYPE_ERROR) {
+                       printf("unknown record: %d (Expecting TYPE_ERROR)\n",
+                              error->hdr.info_type);
+                       goto next_event;
+               }
+
+               printf("\tGeneric Error Record: len=%d\n", error->hdr.len);
+               printf("\terror: %d\n", error->error);
+               printf("\terror_count: %d\n", error->error_count);
+
+               fid = (struct fanotify_event_info_fid *) (error + 1);
+               if ((char *) fid >= next) {
+                       printf("Event doesn't have FID\n");
+                       goto next_event;
+               }
+               printf("FID record found\n");
+
+               if (fid->hdr.info_type != FAN_EVENT_INFO_TYPE_FID) {
+                       printf("unknown record: %d (Expecting TYPE_FID)\n",
+                              fid->hdr.info_type);
+                       goto next_event;
+               }
+               printf("\tfsid: %x%x\n", fid->fsid.val[0], fid->fsid.val[1]);
+               print_fh((struct file_handle *) &fid->handle);
+
+next_event:
+               printf("---\n\n");
+       }
+}
+
+int main(int argc, char **argv)
+{
+       int fd;
+
+       char buffer[BUFSIZ];
+
+       if (argc < 2) {
+               printf("Missing path argument\n");
+               return 1;
+       }
+
+       fd = fanotify_init(FAN_CLASS_NOTIF|FAN_REPORT_FID, O_RDONLY);
+       if (fd < 0)
+               errx(1, "fanotify_init");
+
+       if (fanotify_mark(fd, FAN_MARK_ADD|FAN_MARK_FILESYSTEM,
+                         FAN_FS_ERROR, AT_FDCWD, argv[1])) {
+               errx(1, "fanotify_mark");
+       }
+
+       while (1) {
+               int n = read(fd, buffer, BUFSIZ);
+
+               if (n < 0)
+                       errx(1, "read");
+
+               handle_notifications(buffer, n);
+       }
+
+       return 0;
+}
--
2.32.0

Re: [PATCH v4 16/16] docs: Document the FAN_FS_ERROR event

From: Amir Goldstein <amir73il@gmail.com>
Date: 2021-07-20 17:13:05

On Tue, Jul 20, 2021 at 7:00 PM Gabriel Krisman Bertazi
[off-list ref] wrote:
Document the FAN_FS_ERROR event for user administrators and user space
developers.

Signed-off-by: Gabriel Krisman Bertazi <redacted>
Reviewed-by: Amir Goldstein <amir73il@gmail.com>

Expect one outdated detail...
quoted hunk
---
Changes Since v3:
  - Move FAN_FS_ERROR notification into a subsection of the file.
Changes Since v2:
  - NTR
Changes since v1:
  - Drop references to location record
  - Explain that the inode field is optional
  - Explain we are reporting only the first error
---
 .../admin-guide/filesystem-monitoring.rst     | 70 +++++++++++++++++++
 Documentation/admin-guide/index.rst           |  1 +
 2 files changed, 71 insertions(+)
 create mode 100644 Documentation/admin-guide/filesystem-monitoring.rst
diff --git a/Documentation/admin-guide/filesystem-monitoring.rst b/Documentation/admin-guide/filesystem-monitoring.rst
new file mode 100644
index 000000000000..6c8982985a27
--- /dev/null
+++ b/Documentation/admin-guide/filesystem-monitoring.rst
@@ -0,0 +1,70 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+====================================
+File system Monitoring with fanotify
+====================================
+
+File system Error Reporting
+===========================
+
+fanotify supports the FAN_FS_ERROR mark for file system-wide error
+reporting.  It is meant to be used by file system health monitoring
+daemons who listen on that interface and take actions (notify sysadmin,
+start recovery) when a file system problem is detected by the kernel.
+
+By design, A FAN_FS_ERROR notification exposes sufficient information for a
+monitoring tool to know a problem in the file system has happened.  It
+doesn't necessarily provide a user space application with semantics to
+verify an IO operation was successfully executed.  That is outside of
+scope of this feature. Instead, it is only meant as a framework for
+early file system problem detection and reporting recovery tools.
+
+When a file system operation fails, it is common for dozens of kernel
+errors to cascade after the initial failure, hiding the original failure
+log, which is usually the most useful debug data to troubleshoot the
+problem.  For this reason, FAN_FS_ERROR only reports the first error that
+occurred since the last notification, and it simply counts addition
+errors.  This ensures that the most important piece of error information
+is never lost.
+
+FAN_FS_ERROR requires the fanotify group to be setup with the
+FAN_REPORT_FID flag.
+
+At the time of this writing, the only file system that emits FAN_FS_ERROR
+notifications is Ext4.
+
+A user space example code is provided at ``samples/fanotify/fs-monitor.c``.
+
+A FAN_FS_ERROR Notification has the following format::
+
+  [ Notification Metadata (Mandatory) ]
+  [ Generic Error Record  (Mandatory) ]
+  [ FID record            (Mandatory) ]
+
+Generic error record
+--------------------
+
+The generic error record provides enough information for a file system
+agnostic tool to learn about a problem in the file system, without
+providing any additional details about the problem.  This record is
+identified by ``struct fanotify_event_info_header.info_type`` being set
+to FAN_EVENT_INFO_TYPE_ERROR.
+
+  struct fanotify_event_info_error {
+       struct fanotify_event_info_header hdr;
+       __s32 error;
+       __u32 error_count;
+  };
+
+The `error` field identifies the type of error. `error_count` count
+tracks the number of errors that occurred and were suppressed to
+preserve the original error, since the last notification.
+
+FID record
+----------
+
+The FID record can be used to uniquely identify the inode that triggered
+the error through the combination of fsid and file handler.  A
+filesystem specific handler can use that information to attempt a
+recovery procedure.  Errors that are not related to an inode are
+reported against the root inode.
Not uptodate...

Thanks,
Amir.

Re: [PATCH v4 13/16] fanotify: Introduce FAN_FS_ERROR event

From: Amir Goldstein <amir73il@gmail.com>
Date: 2021-07-21 08:06:53

On Tue, Jul 20, 2021 at 7:00 PM Gabriel Krisman Bertazi
[off-list ref] wrote:
The FAN_FS_ERROR event is a new inode event used by filesystem wide
monitoring tools to receive notifications of type FS_ERROR_EVENT,
emitted directly by filesystems when a problem is detected.  The error
notification includes a generic error descriptor and a FID identifying
the file affected.

FID is sent for every FAN_FS_ERROR. Errors not linked to a regular inode
are reported against the root inode.

An error reporting structure is attached per-mark, and only a single
error can be stored at a time.  This is ok, since once an error occurs,
it is common for a stream of related errors to be reported.  We only log
accumulate the total of errors occurred since the last notification.

Signed-off-by: Gabriel Krisman Bertazi <redacted>

---
Changes since v3:
  - Convert WARN_ON to pr_warn (amir)
  - Remove unecessary READ/WRITE_ONCE (amir)
  - Alloc with GFP_KERNEL_ACCOUNT(amir)
  - Simplify flags on mark allocation (amir)
  - Avoid atomic set of error_count (amir)
  - Simplify rules when merging error_event (amir)
  - Allocate new error_event on get_one_event (amir)
  - Report superblock error with invalid FH (amir,jan)

Changes since v2:
  - Support and equire FID mode (amir)
  - Goto error path instead of early return (amir)
  - Simplify get_one_event (me)
  - Base merging on error_count
  - drop fanotify_queue_error_event

Changes since v1:
  - Pass dentry to fanotify_check_fsid (Amir)
  - FANOTIFY_EVENT_TYPE_ERROR -> FANOTIFY_EVENT_TYPE_FS_ERROR
  - Merge previous patch into it
  - Use a single slot
  - Move fanotify_mark.error_event definition to this commit
  - Rename FAN_ERROR -> FAN_FS_ERROR
  - Restrict FAN_FS_ERROR to FAN_MARK_FILESYSTEM
---
 fs/notify/fanotify/fanotify.c      | 137 ++++++++++++++++++----
 fs/notify/fanotify/fanotify.h      |  53 +++++++++
 fs/notify/fanotify/fanotify_user.c | 180 +++++++++++++++++++++++++++--
 include/linux/fanotify.h           |   8 +-
 include/uapi/linux/fanotify.h      |   8 ++
 5 files changed, 353 insertions(+), 33 deletions(-)
General comment: this patch is pretty big and has been hard for me
to review in every revision of the patch set.
I think this revision is easier for review, but still there are some unrelated
cleanups that could be split out of this patch for next revisions.
I'll leave it to you to decide where the cost/effective line crosses
(i.e. not too much work for you - less work for me)
quoted hunk
diff --git a/fs/notify/fanotify/fanotify.c b/fs/notify/fanotify/fanotify.c
index 912d120b9e48..477596b92bc5 100644
--- a/fs/notify/fanotify/fanotify.c
+++ b/fs/notify/fanotify/fanotify.c
@@ -335,24 +335,6 @@ static u32 fanotify_group_event_mask(
        return test_mask & user_mask;
 }

-/*
- * Check size needed to encode fanotify_fh.
- *
- * Return size of encoded fh without fanotify_fh header.
- * Return 0 on failure to encode.
- */
-static int fanotify_encode_fh_len(struct inode *inode)
-{
-       int dwords = 0;
-
-       if (!inode)
-               return 0;
-
-       exportfs_encode_inode_fh(inode, NULL, &dwords, NULL);
-
-       return dwords << 2;
-}
-
 /*
  * Encode fanotify_fh.
  *
@@ -404,8 +386,12 @@ static int fanotify_encode_fh(struct fanotify_fh *fh, struct inode *inode,
        fh->type = type;
        fh->len = fh_len;

-       /* Mix fh into event merge key */
-       *hash ^= fanotify_hash_fh(fh);
+       /*
+        * Mix fh into event merge key.  Hash might be NULL in case of
+        * unhashed FID events (i.e. FAN_FS_ERROR).
+        */
+       if (hash)
+               *hash ^= fanotify_hash_fh(fh);

        return FANOTIFY_FH_HDR_LEN + fh_len;
@@ -420,6 +406,27 @@ static int fanotify_encode_fh(struct fanotify_fh *fh, struct inode *inode,
        return 0;
 }

+#define FANOTIFY_EMPTY_FH_LEN  8
+/*
+ * Encode an empty fanotify_fh
+ *
+ * Empty FHs are used on FAN_FS_ERROR for errors not linked to any
+ * inode. fh needs to guarantee at least 8 bytes of inline space.
+ */
+static int fanotify_encode_empty_fh(struct fanotify_fh *fh, int max_len)
IMO FANOTIFY_NULL_FH_LEN and fanotify_encode_null_fh()
are better choices, but I do not insist.
+{
+       if (max_len < FANOTIFY_EMPTY_FH_LEN || fh->flags)
+               return -EINVAL;
First of all, this condition must never happen so if we want to
be defensive it should be WARN_ON_ONCE().
I don't think that max_len should even be an input to this helper.
All allocated fh buffers should be at least of size FANOTIFY_NULL_FH_LEN.

Second, even if we return with WARN_ON, we must initialize
fh->len = 0 like fanotify_encode_fh() does otherwise the code repoorting
this event will trip over undefined values.

FWIW, fanotify_encode_fh() is never really called in the code with NULL
inode value - the fh->type = FILEID_ROOT code is a left over from some
old code that is not used and no code is looking at fh->type values
expecting them to be FILEID_ROOT or FILEID_INVALID.
The event reporting code just looks at fh->len to determine if fid info record
should be reported.

So if you like, you can also call fanotify_encode_null_fh() from
fanotify_encode_fh() if inode is NULL.
quoted hunk
+
+       fh->type = FILEID_INVALID;
+       fh->len = FANOTIFY_EMPTY_FH_LEN;
+       fh->flags = 0;
+
+       memset(fh->buf, 0, FANOTIFY_EMPTY_FH_LEN);
+
+       return 0;
+}
+
 /*
  * The inode to use as identifier when reporting fid depends on the event.
  * Report the modified directory inode on dirent modification events.
@@ -691,6 +698,63 @@ static __kernel_fsid_t fanotify_get_fsid(struct fsnotify_iter_info *iter_info)
        return fsid;
 }

+static int fanotify_merge_error_event(struct fsnotify_group *group,
+                                     struct fsnotify_event *event)
+{
+       struct fanotify_event *fae = FANOTIFY_E(event);
+       struct fanotify_error_event *fee = FANOTIFY_EE(fae);
+
+       /*
+        * When err_count > 0, the reporting slot is full.  Just account
+        * the additional error and abort the insertion.
+        */
+       if (fee->err_count) {
+               fee->err_count++;
+               return 1;
+       }
+
+       return 0;
+}
+
+static void fanotify_insert_error_event(struct fsnotify_group *group,
+                                       struct fsnotify_event *event,
+                                       const void *data)
+{
+       struct fanotify_event *fae = FANOTIFY_E(event);
+       const struct fsnotify_event_info *ei =
+               (struct fsnotify_event_info *) data;
+       const struct fs_error_report *report =
+               (struct fs_error_report *) ei->data;
+       struct inode *inode = report->inode;
+       struct fanotify_error_event *fee;
+       int fh_len;
+
+       /* This might be an unexpected type of event (i.e. overflow). */
+       if (!fanotify_is_error_event(fae->mask))
+               return;
+
+       fee = FANOTIFY_EE(fae);
+       fee->fae.type = FANOTIFY_EVENT_TYPE_FS_ERROR;
+       fee->error = report->error;
+       fee->fsid = fee->sb_mark->fsn_mark.connector->fsid;
+       fee->err_count = 1;
+
+       /*
+        * Error reporting needs to happen in atomic context.  If this
+        * inode's file handler is more than we initially predicted,
+        * there is nothing better we can do than report the error with
+        * a bad FH.
+        */
+       fh_len = inode ? fanotify_encode_fh_len(inode) : FANOTIFY_EMPTY_FH_LEN;
if we decide to call fanotify_encode_null_fh() from fanotify_encode_fh(),
we should move this logic into fanotify_encode_fh_len() and convert the
only possible caller of fanotify_encode_fh_len(NULL) to:

        unsigned int child_fh_len = child ? fanotify_encode_fh_len(child) : 0;

There are arguments for making this change and against it, so I'd like
to hear where Jan stands on this matter.

+       if (fh_len > fee->max_fh_len)
+               return;
You need to report a NULL/EMPTY fh in this case as your comment above
states, so why return?
max_fh_len must not be allowed to be less than FANOTIFY_NULL_FH_LEN.
+
+       if (inode)
+               fanotify_encode_fh(&fee->object_fh, inode, fh_len, NULL, 0);
+       else
+               fanotify_encode_empty_fh(&fee->object_fh, fee->max_fh_len);
+}
+
So this could be non conditional call to fanotify_encode_fh().

FYI, this is where my attention span on this patch review ends
and I need to take a break.

I think you should be able to split a prep patch for null fh support and
then the FAN_FS_ERROR patch would just use fanotify_encode_fh()
blindly without caring if inode is NULL or not.

Thanks,
Amir.

Re: [PATCH v4 13/16] fanotify: Introduce FAN_FS_ERROR event

From: Amir Goldstein <amir73il@gmail.com>
Date: 2021-07-21 09:47:39

On Tue, Jul 20, 2021 at 7:00 PM Gabriel Krisman Bertazi
[off-list ref] wrote:
The FAN_FS_ERROR event is a new inode event used by filesystem wide
monitoring tools to receive notifications of type FS_ERROR_EVENT,
emitted directly by filesystems when a problem is detected.  The error
notification includes a generic error descriptor and a FID identifying
the file affected.

FID is sent for every FAN_FS_ERROR. Errors not linked to a regular inode
are reported against the root inode.
commit message is out dated.
An error reporting structure is attached per-mark, and only a single
error can be stored at a time.  This is ok, since once an error occurs,
it is common for a stream of related errors to be reported.  We only log
accumulate the total of errors occurred since the last notification.

Signed-off-by: Gabriel Krisman Bertazi <redacted>

---
Part #2 of review:

[...]
quoted hunk
diff --git a/fs/notify/fanotify/fanotify_user.c b/fs/notify/fanotify/fanotify_user.c
index 0696f2121781..bfc6bf6be197 100644
--- a/fs/notify/fanotify/fanotify_user.c
+++ b/fs/notify/fanotify/fanotify_user.c
@@ -107,6 +107,8 @@ struct kmem_cache *fanotify_perm_event_cachep __read_mostly;
 #define FANOTIFY_EVENT_ALIGN 4
 #define FANOTIFY_INFO_HDR_LEN \
        (sizeof(struct fanotify_event_info_fid) + sizeof(struct file_handle))
+#define FANOTIFY_INFO_ERROR_LEN \
+       (sizeof(struct fanotify_event_info_error))

 static int fanotify_fid_info_len(int fh_len, int name_len)
 {
@@ -130,6 +132,9 @@ static size_t fanotify_event_len(struct fanotify_event *event,
        if (!fid_mode)
                return event_len;

+       if (fanotify_is_error_event(event->mask))
+               event_len += FANOTIFY_INFO_ERROR_LEN;
+
        info = fanotify_event_info(event);
        dir_fh_len = fanotify_event_dir_fh_len(event);
        fh_len = fanotify_event_object_fh_len(event);
@@ -167,6 +172,90 @@ static void fanotify_unhash_event(struct fsnotify_group *group,
        hlist_del_init(&event->merge_list);
 }

+static struct fanotify_error_event *fanotify_alloc_error_event(
+                                       struct fanotify_sb_mark *sb_mark,
+                                       int fh_len)
+{
+       struct fanotify_error_event *fee;
+       struct super_block *sb;
+
+       if (!fh_len) {
+               /*
+                * The FH buffer size is predicted to be the same size
+                * as the root inode file handler.  This should work for
+                * file systems without variable sized FH.
+                */
+               sb = container_of(sb_mark->fsn_mark.connector->obj,
+                                 struct super_block, s_fsnotify_marks);
+               fh_len = fanotify_encode_fh_len(sb->s_root->d_inode);
We need to make sure that fh_len is at least 8 bytes.
We could also take care of that inside fanotify_encode_fh_len.
+       }
+
+       fee = kzalloc(sizeof(*fee) + fh_len, GFP_KERNEL_ACCOUNT);
+       if (!fee)
+               return NULL;
+
+       fanotify_init_event(&fee->fae, 0, FS_ERROR);
+       fee->sb_mark = sb_mark;
+       fee->max_fh_len = fh_len;

I don't understand this logic.
I think we need to store max_fh_len in fanotify_sb_mark struct
and maybe rename to error event member to fh_buf_len.

fanotify_add_mark() should initialize max_fh_len of the sb mark according
to sb->s_root->d_inode.

When insert_error_event fails to encode fh because it does not fit in the
event allocated fh_buf_len, it needs to update the sb_mark's max_fh_len
(with notification lock held).

The next event read will use the new max_fh_len to allocate an event
with a larger buffer and resolve the error condition.

I may be missing something but I don't see how your implementation
resolves the error condition?
+
+       return fee;
+}
+
+/*
+ * Replace a mark's error event with a new structure in preparation for
+ * it to be dequeued.  This is a bit annoying since we need to drop the
+ * lock, so another thread might just steal the event from us.
+ */
+static struct fanotify_event *fanotify_replace_fs_error_event(
+                                       struct fsnotify_group *group,
+                                       struct fanotify_event *fae)
+{
+       struct fanotify_error_event *new, *fee = FANOTIFY_EE(fae);
+       struct fanotify_sb_mark *sb_mark = fee->sb_mark;
+       struct fsnotify_event *fse;
+       int max_fh_len = fee->max_fh_len;
+       int fh_len = fanotify_event_object_fh_len(fae);
+
+       pr_debug("%s: event=%p\n", __func__, fae);
+
+       assert_spin_locked(&group->notification_lock);
+
+       spin_unlock(&group->notification_lock);
+       new = fanotify_alloc_error_event(sb_mark, fee->max_fh_len);
+       spin_lock(&group->notification_lock);
+
+       if (!new)
+               return ERR_PTR(-ENOMEM);
+
+       /*
+        * Since we temporarily dropped the notification_lock, the event
+        * might have been taken from under us and reported by another
+        * reader.  Peek again prior to removal.
+        *
+        * Maybe this is not the same event we started handling.  But as
+        * long as it is also a same size error event for the same
+        * filesystem, it is safe to reuse the allocated memory.
+        */
I don't like this optimization. It doesn't gain much and adds complexity.
If it's not the same event we started handling please return EAGAIN.
quoted hunk
+       fse = fsnotify_peek_first_event(group);
+       if (!fse || !fanotify_is_error_event(FANOTIFY_E(fse)->mask))
+               goto fail;
+
+       fae = FANOTIFY_E(fse);
+       fee = FANOTIFY_EE(fae);
+       if (fee->sb_mark != sb_mark || max_fh_len != fee->max_fh_len  ||
+           fh_len < fanotify_event_object_fh_len(fae))
+               goto fail;
+
+       sb_mark->error_event = new;
+
+       return fae;
+
+fail:
+       kfree(new);
+
+       return ERR_PTR(-EAGAIN);
+}
+
 /*
  * Get an fanotify notification event if one exists and is small
  * enough to fit in "count". Return an error pointer if the count
@@ -196,9 +285,20 @@ static struct fanotify_event *get_one_event(struct fsnotify_group *group,
                goto out;
        }

+       if (fanotify_is_error_event(event->mask)) {
+               /*
+                * Recreate the error event ahead of dequeueing so we
+                * don't need to handle a incorrectly dequeued event.
+                */
+               event = fanotify_replace_fs_error_event(group, event);
+               if (IS_ERR(event))
+                       goto out;
+       }
+
        /*
-        * Held the notification_lock the whole time, so this is the
-        * same event we peeked above.
+        * This might not be the same event peeked above, if
+        * fanotify_recreate_fs_error raced with another reader. It is
+        * guaranteed to succeed, though.
I don't think we need to drop this assumption.
quoted hunk
         */
        fsnotify_remove_first_event(group);
        if (fanotify_is_perm_event(event->mask))
@@ -310,6 +410,28 @@ static int process_access_response(struct fsnotify_group *group,
        return -ENOENT;
 }

+static size_t copy_error_info_to_user(struct fanotify_event *event,
+                                     char __user *buf, int count)
+{
+       struct fanotify_event_info_error info;
+       struct fanotify_error_event *fee = FANOTIFY_EE(event);
+
+       info.hdr.info_type = FAN_EVENT_INFO_TYPE_ERROR;
+       info.hdr.pad = 0;
+       info.hdr.len = sizeof(struct fanotify_event_info_error);
+
+       if (WARN_ON(count < info.hdr.len))
+               return -EFAULT;
+
+       info.error = fee->error;
+       info.error_count = fee->err_count;
+
+       if (copy_to_user(buf, &info, sizeof(info)))
+               return -EFAULT;
+
+       return info.hdr.len;
+}
+
 static int copy_info_to_user(__kernel_fsid_t *fsid, struct fanotify_fh *fh,
                             int info_type, const char *name, size_t name_len,
                             char __user *buf, size_t count)
@@ -468,6 +590,14 @@ static ssize_t copy_event_to_user(struct fsnotify_group *group,
        if (f)
                fd_install(fd, f);

+       if (fanotify_is_error_event(event->mask)) {
+               ret = copy_error_info_to_user(event, buf, count);
+               if (ret < 0)
+                       goto out_close_fd;
+               buf += ret;
+               count -= ret;
+       }
+
        /* Event info records order is: dir fid + name, child fid */
        if (fanotify_event_dir_fh_len(event)) {
                info_type = info->name_len ? FAN_EVENT_INFO_TYPE_DFID_NAME :
@@ -580,6 +710,8 @@ static ssize_t fanotify_read(struct file *file, char __user *buf,
                event = get_one_event(group, count);
                if (IS_ERR(event)) {
                        ret = PTR_ERR(event);
+                       if (ret == -EAGAIN)
+                               continue;
                        break;
                }
@@ -993,7 +1125,9 @@ static int fanotify_add_mark(struct fsnotify_group *group,
                             __kernel_fsid_t *fsid)
 {
        struct fsnotify_mark *fsn_mark;
+       struct fanotify_sb_mark *sb_mark;
        __u32 added;
+       int ret = 0;

        mutex_lock(&group->mark_mutex);
        fsn_mark = fsnotify_find_mark(connp, group);
@@ -1004,13 +1138,34 @@ static int fanotify_add_mark(struct fsnotify_group *group,
                        return PTR_ERR(fsn_mark);
                }
        }
+
+       /*
+        * Error events are allocated per super-block mark, but only if
+        * strictly needed (i.e. FAN_FS_ERROR was requested).
+        */
+       if (type == FSNOTIFY_OBJ_TYPE_SB && !(flags & FAN_MARK_IGNORED_MASK) &&
+           (mask & FAN_FS_ERROR)) {
+               sb_mark = FANOTIFY_SB_MARK(fsn_mark);
+
+               if (!sb_mark->error_event) {
+                       sb_mark->error_event =
+                               fanotify_alloc_error_event(sb_mark, 0);
+                       if (!sb_mark->error_event) {
+                               ret = -ENOMEM;
+                               goto out;
+                       }
+               }
+       }
+
        added = fanotify_mark_add_to_mask(fsn_mark, mask, flags);
        if (added & ~fsnotify_conn_mask(fsn_mark->connector))
                fsnotify_recalc_mask(fsn_mark->connector);
+
+out:
        mutex_unlock(&group->mark_mutex);

        fsnotify_put_mark(fsn_mark);
-       return 0;
+       return ret;
 }

 static int fanotify_add_vfsmount_mark(struct fsnotify_group *group,
@@ -1382,14 +1537,14 @@ static int do_fanotify_mark(int fanotify_fd, unsigned int flags, __u64 mask,
                goto fput_and_out;

        /*
-        * Events with data type inode do not carry enough information to report
-        * event->fd, so we do not allow setting a mask for inode events unless
-        * group supports reporting fid.
-        * inode events are not supported on a mount mark, because they do not
-        * carry enough information (i.e. path) to be filtered by mount point.
-        */
+       * Events that do not carry enough information to report
+       * event->fd require a group that supports reporting fid.  Those
+       * events are not supported on a mount mark, because they do not
+       * carry enough information (i.e. path) to be filtered by mount
+       * point.
+       */
        fid_mode = FAN_GROUP_FLAG(group, FANOTIFY_FID_BITS);
-       if (mask & FANOTIFY_INODE_EVENTS &&
+       if (!(mask & FANOTIFY_FD_EVENTS) &&
            (!fid_mode || mark_type == FAN_MARK_MOUNT))
                goto fput_and_out;
[...]
quoted hunk
@@ -81,9 +81,13 @@ extern struct ctl_table fanotify_table[]; /* for sysctl */
  */
 #define FANOTIFY_DIRENT_EVENTS (FAN_MOVE | FAN_CREATE | FAN_DELETE)

-/* Events that can only be reported with data type FSNOTIFY_EVENT_INODE */
+/* Events that can be reported with event->fd */
+#define FANOTIFY_FD_EVENTS (FANOTIFY_PATH_EVENTS | FANOTIFY_PERM_EVENTS)
+
+/* Events that can only be reported to groups that support FID mode */
 #define FANOTIFY_INODE_EVENTS  (FANOTIFY_DIRENT_EVENTS | \
I know this macro is unused now, but let's call it FANOTIFY_FID_EVENTS please,
because FAN_FS_ERROR does not have data type INODE.

Thanks,
Amir.

[fsnotify] 4c40d6efc8: unixbench.score -3.3% regression

From: kernel test robot <hidden>
Date: 2021-07-31 06:20:14


Greeting,

FYI, we noticed a -3.3% regression of unixbench.score due to commit:


commit: 4c40d6efc8b22b88a45c335ffd6d25b55d769f5b ("[PATCH v4 08/16] fsnotify: pass arguments of fsnotify() in struct fsnotify_event_info")
url: https://github.com/0day-ci/linux/commits/Gabriel-Krisman-Bertazi/File-system-wide-monitoring/20210721-001444
base: https://git.kernel.org/cgit/linux/kernel/git/jack/linux-fs.git fsnotify

in testcase: unixbench
on test machine: 96 threads 2 sockets Intel(R) Xeon(R) CPU @ 2.30GHz with 128G memory
with following parameters:

	runtime: 300s
	nr_task: 1
	test: pipe
	cpufreq_governor: performance
	ucode: 0x4003006

test-description: UnixBench is the original BYTE UNIX benchmark suite aims to test performance of Unix-like system.
test-url: https://github.com/kdlucas/byte-unixbench

In addition to that, the commit also has significant impact on the following tests:

+------------------+-------------------------------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_thread_ops -1.3% regression                        |
| test machine     | 192 threads 4 sockets Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory |
| test parameters  | cpufreq_governor=performance                                                        |
|                  | mode=thread                                                                         |
|                  | nr_task=100%                                                                        |
|                  | test=eventfd1                                                                       |
|                  | ucode=0x5003006                                                                     |
+------------------+-------------------------------------------------------------------------------------+


If you fix the issue, kindly add following tag
Reported-by: kernel test robot <redacted>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install                job.yaml  # job file is attached in this email
        bin/lkp split-job --compatible job.yaml  # generate the yaml file for lkp run
        bin/lkp run                    generated-yaml-file

=========================================================================================
compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase/ucode:
  gcc-9/performance/x86_64-rhel-8.3/1/debian-10.4-x86_64-20200603.cgz/300s/lkp-csl-2sp4/pipe/unixbench/0x4003006

commit: 
  263b74f276 ("fsnotify: Add helper to detect overflow_event")
  4c40d6efc8 ("fsnotify: pass arguments of fsnotify() in struct fsnotify_event_info")

263b74f2761d777d 4c40d6efc8b22b88a45c335ffd6 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
      1554            -3.3%       1503        unixbench.score
 7.551e+08            -3.4%  7.295e+08        unixbench.workload
      0.00 ±158%   +4075.0%       0.02 ± 37%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.copy_page_from_iter.pipe_write.new_sync_write
      0.00 ±158%   +4075.0%       0.02 ± 37%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.copy_page_from_iter.pipe_write.new_sync_write
    318916 ±169%     -95.4%      14689 ±  5%  perf-stat.i.dTLB-load-misses
   5208687 ±  7%     +26.3%    6579325 ±  2%  perf-stat.i.iTLB-load-misses
    991.91 ±  5%     -17.0%     822.95 ±  2%  perf-stat.i.instructions-per-iTLB-miss
      0.02 ±170%      -0.0        0.00 ±  5%  perf-stat.overall.dTLB-load-miss-rate%
    931.13 ±  7%     -22.5%     721.29 ±  2%  perf-stat.overall.instructions-per-iTLB-miss
      2491            +1.8%       2535        perf-stat.overall.path-length
    318091 ±169%     -95.4%      14652 ±  5%  perf-stat.ps.dTLB-load-misses
   5195369 ±  7%     +26.3%    6562257 ±  2%  perf-stat.ps.iTLB-load-misses
      0.82 ± 13%      +0.2        1.02 ±  8%  perf-profile.calltrace.cycles-pp.mutex_lock.pipe_write.new_sync_write.vfs_write.ksys_write
      0.69 ±  8%      +0.2        0.91 ± 11%  perf-profile.calltrace.cycles-pp.security_file_permission.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.05 ± 10%      +0.3        1.33 ±  6%  perf-profile.calltrace.cycles-pp.security_file_permission.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.39 ± 63%      +0.3        0.69 ± 11%  perf-profile.calltrace.cycles-pp.common_file_perm.security_file_permission.vfs_write.ksys_write.do_syscall_64
      0.16 ±158%      +0.5        0.68 ±  6%  perf-profile.calltrace.cycles-pp.common_file_perm.security_file_permission.vfs_read.ksys_read.do_syscall_64
      0.64 ± 16%      -0.2        0.46 ± 17%  perf-profile.children.cycles-pp.anon_pipe_buf_release
      0.34 ± 47%      -0.2        0.19 ± 10%  perf-profile.children.cycles-pp.wait_for_xmitr
      0.35 ± 46%      -0.2        0.20 ± 10%  perf-profile.children.cycles-pp.serial8250_console_write
      0.33 ± 47%      -0.1        0.19 ±  9%  perf-profile.children.cycles-pp.serial8250_console_putchar
      0.21 ± 29%      -0.1        0.13 ± 22%  perf-profile.children.cycles-pp.enqueue_hrtimer
      0.18 ± 31%      -0.1        0.11 ± 31%  perf-profile.children.cycles-pp.timerqueue_add
      0.30 ± 14%      -0.1        0.23 ± 15%  perf-profile.children.cycles-pp.update_blocked_averages
      0.10 ± 21%      -0.1        0.04 ± 91%  perf-profile.children.cycles-pp.menu_reflect
      1.37 ±  8%      +0.4        1.76 ±  6%  perf-profile.children.cycles-pp.common_file_perm
      2.27 ±  8%      +0.5        2.82 ±  5%  perf-profile.children.cycles-pp.security_file_permission
      0.00            +1.5        1.47 ± 14%  perf-profile.children.cycles-pp.__fsnotify
      0.63 ± 16%      -0.2        0.45 ± 17%  perf-profile.self.cycles-pp.anon_pipe_buf_release
      0.11 ± 32%      -0.0        0.07 ± 23%  perf-profile.self.cycles-pp.sysvec_apic_timer_interrupt
      0.47 ±  9%      +0.1        0.58 ± 12%  perf-profile.self.cycles-pp.new_sync_write
      0.43 ± 14%      +0.1        0.58 ± 12%  perf-profile.self.cycles-pp.security_file_permission
      0.86 ± 17%      +0.4        1.28 ±  8%  perf-profile.self.cycles-pp.common_file_perm
      0.00            +1.5        1.45 ± 14%  perf-profile.self.cycles-pp.__fsnotify


                                                                                
                                  unixbench.score                               
                                                                                
  1580 +--------------------------------------------------------------------+   
       |                                                                    |   
  1560 |-+                     .+...+...+..  ..+                            |   
       |                   ..+.            +.                               |   
  1540 |-+              .+.                                                 |   
       |...+..+...+...+.                                                    |   
  1520 |-+                                                                  |   
       |                        O   O   O  O                 O   O          |   
  1500 |-+                                     O   O  O   O          O  O   |   
       |                                                                    |   
  1480 |-+            O                                                     |   
       |   O  O   O          O                                              |   
  1460 |-+               O                                                  |   
       |                                                                    |   
  1440 +--------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample

***************************************************************************************************
lkp-csl-2ap2: 192 threads 4 sockets Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
  gcc-9/performance/x86_64-rhel-8.3/thread/100%/debian-10.4-x86_64-20200603.cgz/lkp-csl-2ap2/eventfd1/will-it-scale/0x5003006

commit: 
  263b74f276 ("fsnotify: Add helper to detect overflow_event")
  4c40d6efc8 ("fsnotify: pass arguments of fsnotify() in struct fsnotify_event_info")

263b74f2761d777d 4c40d6efc8b22b88a45c335ffd6 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
 3.014e+08            -1.3%  2.974e+08        will-it-scale.192.threads
   1569651            -1.3%    1549107        will-it-scale.per_thread_ops
 3.014e+08            -1.3%  2.974e+08        will-it-scale.workload
    747.17 ± 37%   +1300.3%      10462 ±145%  softirqs.CPU13.NET_RX
      1212 ± 41%   +1341.6%      17474 ±133%  interrupts.34:PCI-MSI.524292-edge.eth0-TxRx-3
      1212 ± 41%   +1341.6%      17474 ±133%  interrupts.CPU13.34:PCI-MSI.524292-edge.eth0-TxRx-3
      0.01 ±  8%     +32.5%       0.02 ± 15%  perf-sched.sch_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      2.86 ±162%    +256.5%      10.20 ± 38%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.__do_fault.do_fault.__handle_mm_fault
     38371            -1.5%      37785        proc-vmstat.nr_slab_reclaimable
     71698            -1.8%      70437        proc-vmstat.nr_slab_unreclaimable
      5621 ±  6%     -12.4%       4926 ±  3%  slabinfo.Acpi-State.active_objs
      5621 ±  6%     -12.4%       4926 ±  3%  slabinfo.Acpi-State.num_objs
      5395 ±  7%     -13.4%       4670 ±  7%  slabinfo.files_cache.active_objs
      5395 ±  7%     -13.4%       4670 ±  7%  slabinfo.files_cache.num_objs
   1070171 ±  2%      +8.4%    1160265 ±  2%  perf-stat.i.cache-misses
   7414136 ±  2%     +11.7%    8284549 ±  2%  perf-stat.i.cache-references
    741668 ±  3%     -12.8%     647061 ±  4%  perf-stat.i.cycles-between-cache-misses
 1.598e+11            +1.5%  1.621e+11        perf-stat.i.dTLB-loads
    165125            -2.3%     161301        perf-stat.i.dTLB-store-misses
 5.501e+08 ±  3%      +4.5%  5.746e+08        perf-stat.i.iTLB-load-misses
      1005 ±  4%      -4.3%     962.30        perf-stat.i.instructions-per-iTLB-miss
    237876 ±  2%     +10.3%     262365 ±  3%  perf-stat.i.node-load-misses
     91893 ±  2%      +4.9%      96434        perf-stat.i.node-store-misses
      0.01 ±  2%     +11.4%       0.02 ±  2%  perf-stat.overall.MPKI
    497110 ±  3%      -6.9%     462766 ±  2%  perf-stat.overall.cycles-between-cache-misses
      0.00            -0.0        0.00        perf-stat.overall.dTLB-store-miss-rate%
      1004 ±  4%      -4.5%     960.10        perf-stat.overall.instructions-per-iTLB-miss
    551166            +1.3%     558404        perf-stat.overall.path-length
   1107917 ±  3%      +7.4%    1190240 ±  2%  perf-stat.ps.cache-misses
   7527216 ±  2%     +11.4%    8382817 ±  2%  perf-stat.ps.cache-references
 1.592e+11            +1.5%  1.616e+11        perf-stat.ps.dTLB-loads
    164932            -2.3%     161085        perf-stat.ps.dTLB-store-misses
 5.483e+08 ±  3%      +4.5%  5.729e+08        perf-stat.ps.iTLB-load-misses
    237049 ±  2%     +10.4%     261604 ±  4%  perf-stat.ps.node-load-misses
     91568 ±  2%      +4.9%      96075        perf-stat.ps.node-store-misses
      9.39            -0.3        9.05        perf-profile.calltrace.cycles-pp.eventfd_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     11.74            -0.2       11.57        perf-profile.calltrace.cycles-pp.eventfd_read.new_sync_read.vfs_read.ksys_read.do_syscall_64
      7.75            -0.1        7.65        perf-profile.calltrace.cycles-pp.__entry_text_start.__libc_read
      7.76            -0.1        7.66        perf-profile.calltrace.cycles-pp.__entry_text_start.__libc_write
      7.46            -0.1        7.37        perf-profile.calltrace.cycles-pp._copy_to_iter.eventfd_read.new_sync_read.vfs_read.ksys_read
      1.54            -0.1        1.45        perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.__libc_write
      1.54            -0.1        1.46        perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.__libc_read
      2.59            -0.1        2.53        perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
      1.82            -0.0        1.78        perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
      2.58            -0.0        2.54        perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
      1.81            -0.0        1.78        perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
      1.06            -0.0        1.03        perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string._copy_from_user.eventfd_write.vfs_write.ksys_write
      1.74            -0.0        1.70        perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.eventfd_read.new_sync_read.vfs_read.ksys_read
      1.34            -0.0        1.32        perf-profile.calltrace.cycles-pp.fput_many.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
      0.55            -0.0        0.53        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_safe_stack.__libc_read
      0.64            +0.0        0.67        perf-profile.calltrace.cycles-pp.__might_sleep.__might_fault._copy_from_user.eventfd_write.vfs_write
      1.91            +0.0        1.95        perf-profile.calltrace.cycles-pp.__might_fault._copy_to_iter.eventfd_read.new_sync_read.vfs_read
      1.99            +0.1        2.04        perf-profile.calltrace.cycles-pp.__might_fault._copy_from_user.eventfd_write.vfs_write.ksys_write
     42.27            +0.1       42.38        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_read
     46.96            +0.1       47.07        perf-profile.calltrace.cycles-pp.__libc_write
      0.68 ±  6%      +0.1        0.82 ±  4%  perf-profile.calltrace.cycles-pp.__x64_sys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
     40.77            +0.2       40.94        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
     36.01            +0.3       36.27        perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
     26.10            +0.3       26.36        perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
     32.26            +0.3       32.55        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_write
     30.79            +0.3       31.13        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
     28.76            +0.6       29.33        perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
      4.33 ±  4%      +0.8        5.08 ±  2%  perf-profile.calltrace.cycles-pp.security_file_permission.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     18.63 ±  2%      +0.8       19.40        perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
      7.72 ±  3%      +0.8        8.49        perf-profile.calltrace.cycles-pp.security_file_permission.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.7        1.66 ±  3%  perf-profile.calltrace.cycles-pp.__fsnotify.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +2.0        1.96        perf-profile.calltrace.cycles-pp.__fsnotify.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +2.0        2.03 ±  2%  perf-profile.calltrace.cycles-pp.__fsnotify.security_file_permission.vfs_read.ksys_read.do_syscall_64
      7.05 ±  2%      -7.0        0.00        perf-profile.children.cycles-pp.fsnotify
      9.54            -0.3        9.22        perf-profile.children.cycles-pp.eventfd_write
      8.66            -0.2        8.42        perf-profile.children.cycles-pp.syscall_return_via_sysret
     11.96            -0.2       11.78        perf-profile.children.cycles-pp.eventfd_read
     57.15            -0.2       56.98        perf-profile.children.cycles-pp.__libc_read
     10.01            -0.1        9.88        perf-profile.children.cycles-pp.__entry_text_start
      3.74            -0.1        3.66        perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      5.51            -0.1        5.43        perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      3.62            -0.1        3.56        perf-profile.children.cycles-pp._raw_spin_lock_irq
      2.30            -0.0        2.26        perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      3.27            -0.0        3.22        perf-profile.children.cycles-pp.copy_user_generic_unrolled
      2.64            -0.0        2.62        perf-profile.children.cycles-pp.fput_many
      2.06            +0.0        2.09        perf-profile.children.cycles-pp.___might_sleep
      1.27            +0.0        1.32        perf-profile.children.cycles-pp.__might_sleep
      4.17            +0.1        4.26        perf-profile.children.cycles-pp.__might_fault
     47.14            +0.1       47.25        perf-profile.children.cycles-pp.__libc_write
      0.72 ±  5%      +0.1        0.84 ±  3%  perf-profile.children.cycles-pp.__x64_sys_write
     26.35            +0.2       26.59        perf-profile.children.cycles-pp.ksys_write
     36.20            +0.3       36.48        perf-profile.children.cycles-pp.ksys_read
     74.75            +0.4       75.16        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.48 ±  3%      +0.5        1.00 ±  3%  perf-profile.children.cycles-pp.apparmor_file_permission
     71.83            +0.6       72.38        perf-profile.children.cycles-pp.do_syscall_64
     18.86 ±  2%      +0.7       19.59        perf-profile.children.cycles-pp.vfs_write
     12.35 ±  3%      +1.4       13.79        perf-profile.children.cycles-pp.security_file_permission
      0.00            +5.8        5.80        perf-profile.children.cycles-pp.__fsnotify
      6.73 ±  2%      -6.7        0.00        perf-profile.self.cycles-pp.fsnotify
      2.20 ±  2%      -0.3        1.93 ±  2%  perf-profile.self.cycles-pp.eventfd_write
      8.57            -0.2        8.34        perf-profile.self.cycles-pp.syscall_return_via_sysret
      2.96            -0.1        2.81        perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      1.51            -0.1        1.41 ±  2%  perf-profile.self.cycles-pp.ksys_write
      3.32            -0.1        3.25        perf-profile.self.cycles-pp.exit_to_user_mode_prepare
      4.50            -0.1        4.43        perf-profile.self.cycles-pp.__entry_text_start
      2.57            -0.1        2.50        perf-profile.self.cycles-pp.eventfd_read
      2.53            -0.1        2.46        perf-profile.self.cycles-pp._copy_to_iter
      3.45            -0.1        3.39        perf-profile.self.cycles-pp._raw_spin_lock_irq
      2.52            -0.0        2.47        perf-profile.self.cycles-pp.fput_many
      0.83            -0.0        0.79        perf-profile.self.cycles-pp._copy_from_user
      2.10            -0.0        2.06        perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      0.92            -0.0        0.89        perf-profile.self.cycles-pp.syscall_exit_to_user_mode_prepare
      2.01            +0.0        2.04        perf-profile.self.cycles-pp.___might_sleep
      1.11            +0.0        1.16        perf-profile.self.cycles-pp.__might_sleep
      0.70 ±  4%      +0.1        0.80 ±  2%  perf-profile.self.cycles-pp.__x64_sys_write
      2.99 ±  2%      +0.4        3.36 ±  2%  perf-profile.self.cycles-pp.vfs_read
      2.53 ±  3%      +0.5        3.03 ±  4%  perf-profile.self.cycles-pp.vfs_write
      0.38 ±  4%      +0.5        0.89 ±  3%  perf-profile.self.cycles-pp.apparmor_file_permission
      2.64 ± 13%      +0.9        3.54 ±  9%  perf-profile.self.cycles-pp.security_file_permission
      0.00            +5.6        5.59        perf-profile.self.cycles-pp.__fsnotify





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


---
0DAY/LKP+ Test Infrastructure                   Open Source Technology Center
https://lists.01.org/hyperkitty/list/lkp@lists.01.org       Intel Corporation

Thanks,
Oliver Sang

Re: [PATCH v4 15/16] samples: Add fs error monitoring example

From: kernel test robot <hidden>
Date: 2021-07-31 07:12:28

Hi Gabriel,

I love your patch! Yet something to improve:

[auto build test ERROR on ext3/fsnotify]
[also build test ERROR on ext4/dev ext3/for_next linus/master v5.14-rc3 next-20210730]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Gabriel-Krisman-Bertazi/File-system-wide-monitoring/20210721-001444
base:   https://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs.git fsnotify
config: x86_64-allyesconfig (attached as .config)
compiler: gcc-9 (Debian 9.3.0-22) 9.3.0
reproduce (this is a W=1 build):
        # https://github.com/0day-ci/linux/commit/d65dd026fced2d19e194ceca1c490f32f1610a5d
        git remote add linux-review https://github.com/0day-ci/linux
        git fetch --no-tags linux-review Gabriel-Krisman-Bertazi/File-system-wide-monitoring/20210721-001444
        git checkout d65dd026fced2d19e194ceca1c490f32f1610a5d
        # save the attached .config to linux build tree
        mkdir build_dir
        make W=1 O=build_dir ARCH=x86_64 SHELL=/bin/bash samples/

If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <redacted>

All errors (new ones prefixed by >>):

   samples/fanotify/fs-monitor.c: In function 'print_fh':
quoted
samples/fanotify/fs-monitor.c:43:25: error: 'FILEID_INO32_GEN' undeclared (first use in this function)
      43 |  if (fh->handle_type == FILEID_INO32_GEN)
         |                         ^~~~~~~~~~~~~~~~
   samples/fanotify/fs-monitor.c:43:25: note: each undeclared identifier is reported only once for each function it appears in
quoted
samples/fanotify/fs-monitor.c:45:30: error: 'FILEID_INVALID' undeclared (first use in this function)
      45 |  else if (fh->handle_type == FILEID_INVALID && !h[0] && !h[1])
         |                              ^~~~~~~~~~~~~~


vim +/FILEID_INO32_GEN +43 samples/fanotify/fs-monitor.c

    31	
    32	static void print_fh(struct file_handle *fh)
    33	{
    34		int i;
    35		uint32_t *h = (uint32_t *) fh->f_handle;
    36	
    37		printf("\tfh: ");
    38		for (i = 0; i < fh->handle_bytes; i++)
    39			printf("%hhx", fh->f_handle[i]);
    40		printf("\n");
    41	
    42		printf("\tdecoded fh: ");
  > 43		if (fh->handle_type == FILEID_INO32_GEN)
    44			printf("inode=%u gen=%u\n", h[0], h[1]);
  > 45		else if (fh->handle_type == FILEID_INVALID && !h[0] && !h[1])
    46			printf("Type %d (Superblock error)\n", fh->handle_type);
    47		else
    48			printf("Type %d (Unknown)\n", fh->handle_type);
    49	

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org

Re: [fsnotify] 4c40d6efc8: unixbench.score -3.3% regression

From: Amir Goldstein <amir73il@gmail.com>
Date: 2021-07-31 09:27:58

On Sat, Jul 31, 2021 at 9:20 AM kernel test robot [off-list ref] wrote:


Greeting,

FYI, we noticed a -3.3% regression of unixbench.score due to commit:


commit: 4c40d6efc8b22b88a45c335ffd6d25b55d769f5b ("[PATCH v4 08/16] fsnotify: pass arguments of fsnotify() in struct fsnotify_event_info")
url: https://github.com/0day-ci/linux/commits/Gabriel-Krisman-Bertazi/File-system-wide-monitoring/20210721-001444
base: https://git.kernel.org/cgit/linux/kernel/git/jack/linux-fs.git fsnotify

in testcase: unixbench
on test machine: 96 threads 2 sockets Intel(R) Xeon(R) CPU @ 2.30GHz with 128G memory
with following parameters:

        runtime: 300s
        nr_task: 1
        test: pipe
        cpufreq_governor: performance
        ucode: 0x4003006

test-description: UnixBench is the original BYTE UNIX benchmark suite aims to test performance of Unix-like system.
test-url: https://github.com/kdlucas/byte-unixbench

In addition to that, the commit also has significant impact on the following tests:

+------------------+-------------------------------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_thread_ops -1.3% regression                        |
| test machine     | 192 threads 4 sockets Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory |
| test parameters  | cpufreq_governor=performance                                                        |
|                  | mode=thread                                                                         |
|                  | nr_task=100%                                                                        |
|                  | test=eventfd1                                                                       |
|                  | ucode=0x5003006                                                                     |
+------------------+-------------------------------------------------------------------------------------+


If you fix the issue, kindly add following tag
Reported-by: kernel test robot <redacted>
Gabriel,

It looks like my change throws away much of the performance gain for
small IO on pipes without any watches that was achieved by commit
71d734103edf ("fsnotify: Rearrange fast path to minimise overhead
when there is no watcher").

I think the way to fix it is to lift the optimization in __fsnotify()
to the fsnotify_parent() inline wrapper as Mel considered doing
but was not sure it was worth the effort at the time.

It's not completely trivial. I think it requires setting a flag
MNT_FSNOTIFY_WATCHED when there are watches on the
vfsmount. I will look into it.

Thanks,
Amir.

Re: [fsnotify] 4c40d6efc8: unixbench.score -3.3% regression

From: Amir Goldstein <amir73il@gmail.com>
Date: 2021-07-31 16:27:33

On Sat, Jul 31, 2021 at 12:27 PM Amir Goldstein [off-list ref] wrote:
On Sat, Jul 31, 2021 at 9:20 AM kernel test robot [off-list ref] wrote:
quoted


Greeting,

FYI, we noticed a -3.3% regression of unixbench.score due to commit:


commit: 4c40d6efc8b22b88a45c335ffd6d25b55d769f5b ("[PATCH v4 08/16] fsnotify: pass arguments of fsnotify() in struct fsnotify_event_info")
url: https://github.com/0day-ci/linux/commits/Gabriel-Krisman-Bertazi/File-system-wide-monitoring/20210721-001444
base: https://git.kernel.org/cgit/linux/kernel/git/jack/linux-fs.git fsnotify

in testcase: unixbench
on test machine: 96 threads 2 sockets Intel(R) Xeon(R) CPU @ 2.30GHz with 128G memory
with following parameters:

        runtime: 300s
        nr_task: 1
        test: pipe
        cpufreq_governor: performance
        ucode: 0x4003006

test-description: UnixBench is the original BYTE UNIX benchmark suite aims to test performance of Unix-like system.
test-url: https://github.com/kdlucas/byte-unixbench

In addition to that, the commit also has significant impact on the following tests:

+------------------+-------------------------------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_thread_ops -1.3% regression                        |
| test machine     | 192 threads 4 sockets Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory |
| test parameters  | cpufreq_governor=performance                                                        |
|                  | mode=thread                                                                         |
|                  | nr_task=100%                                                                        |
|                  | test=eventfd1                                                                       |
|                  | ucode=0x5003006                                                                     |
+------------------+-------------------------------------------------------------------------------------+


If you fix the issue, kindly add following tag
Reported-by: kernel test robot <redacted>
Gabriel,

It looks like my change throws away much of the performance gain for
small IO on pipes without any watches that was achieved by commit
71d734103edf ("fsnotify: Rearrange fast path to minimise overhead
when there is no watcher").

I think the way to fix it is to lift the optimization in __fsnotify()
to the fsnotify_parent() inline wrapper as Mel considered doing
but was not sure it was worth the effort at the time.

It's not completely trivial. I think it requires setting a flag
MNT_FSNOTIFY_WATCHED when there are watches on the
vfsmount. I will look into it.
Oliver,

Would it be possible to request a re-test with the branch:
https://github.com/amir73il/linux fsnotify-perf

The patch at the tip of that branch is the one this regression report
has blamed.

My expectation is that the patch at fsnotify-perf^ ("fsnotify: optimize the
case of no marks of any type") will improve performance of the test case
compared to baseline (v5.14-rc3) and that the patch at the tip of fsnotify-perf
would not regress performance.

Thanks,
Amir.

Re: [fsnotify] 4c40d6efc8: unixbench.score -3.3% regression

From: Oliver Sang <hidden>
Date: 2021-08-03 14:04:32

Hi Amir,

On Sat, Jul 31, 2021 at 07:27:19PM +0300, Amir Goldstein wrote:
On Sat, Jul 31, 2021 at 12:27 PM Amir Goldstein [off-list ref] wrote:
quoted
On Sat, Jul 31, 2021 at 9:20 AM kernel test robot [off-list ref] wrote:
quoted


Greeting,

FYI, we noticed a -3.3% regression of unixbench.score due to commit:


commit: 4c40d6efc8b22b88a45c335ffd6d25b55d769f5b ("[PATCH v4 08/16] fsnotify: pass arguments of fsnotify() in struct fsnotify_event_info")
url: https://github.com/0day-ci/linux/commits/Gabriel-Krisman-Bertazi/File-system-wide-monitoring/20210721-001444
base: https://git.kernel.org/cgit/linux/kernel/git/jack/linux-fs.git fsnotify

in testcase: unixbench
on test machine: 96 threads 2 sockets Intel(R) Xeon(R) CPU @ 2.30GHz with 128G memory
with following parameters:

        runtime: 300s
        nr_task: 1
        test: pipe
        cpufreq_governor: performance
        ucode: 0x4003006

test-description: UnixBench is the original BYTE UNIX benchmark suite aims to test performance of Unix-like system.
test-url: https://github.com/kdlucas/byte-unixbench

In addition to that, the commit also has significant impact on the following tests:

+------------------+-------------------------------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_thread_ops -1.3% regression                        |
| test machine     | 192 threads 4 sockets Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory |
| test parameters  | cpufreq_governor=performance                                                        |
|                  | mode=thread                                                                         |
|                  | nr_task=100%                                                                        |
|                  | test=eventfd1                                                                       |
|                  | ucode=0x5003006                                                                     |
+------------------+-------------------------------------------------------------------------------------+


If you fix the issue, kindly add following tag
Reported-by: kernel test robot <redacted>
Gabriel,

It looks like my change throws away much of the performance gain for
small IO on pipes without any watches that was achieved by commit
71d734103edf ("fsnotify: Rearrange fast path to minimise overhead
when there is no watcher").

I think the way to fix it is to lift the optimization in __fsnotify()
to the fsnotify_parent() inline wrapper as Mel considered doing
but was not sure it was worth the effort at the time.

It's not completely trivial. I think it requires setting a flag
MNT_FSNOTIFY_WATCHED when there are watches on the
vfsmount. I will look into it.
Oliver,

Would it be possible to request a re-test with the branch:
https://github.com/amir73il/linux fsnotify-perf

The patch at the tip of that branch is the one this regression report
has blamed.

My expectation is that the patch at fsnotify-perf^ ("fsnotify: optimize the
case of no marks of any type") will improve performance of the test case
compared to baseline (v5.14-rc3) and that the patch at the tip of fsnotify-perf
would not regress performance.
we tested this branch and the results meet your expectation.

fsnotify-perf^ improves performance comparing to v5.14-rc3. tip is a little worse
than its parent (-3.3%), but still better than v5.14-rc3.

below is detail data.


=========================================================================================
compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase/ucode:
  gcc-9/performance/x86_64-rhel-8.3/1/debian-10.4-x86_64-20200603.cgz/300s/lkp-csl-2sp4/pipe/unixbench/0x4003006

commit: 
  v5.14-rc3
  23050d041 ("fsnotify: optimize the case of no marks of any type")
  7446ba772 ("fsnotify: pass arguments of fsnotify() in struct fsnotify_event_info")

       v5.14-rc3 23050d0419441a02185e4ed5170 7446ba772ae107ab937cd04e880 
---------------- --------------------------- --------------------------- 
         %stddev     %change         %stddev     %change         %stddev
             \          |                \          |                \  
      1562            +8.0%       1688            +4.5%       1633        unixbench.score
    390.37            +0.0%     390.37            -0.0%     390.36        unixbench.time.elapsed_time
    390.37            +0.0%     390.37            -0.0%     390.36        unixbench.time.elapsed_time.max
    113.78 ± 91%    -100.0%       0.00           -84.4%      17.78 ±282%  unixbench.time.file_system_inputs
      3445 ± 89%    -100.0%       0.00           -82.6%     600.89 ±282%  unixbench.time.file_system_outputs
    560.89 ± 10%      -1.4%     553.00 ±  6%      -5.8%     528.22 ±  3%  unixbench.time.involuntary_context_switches
     10032            -0.3%      10001            -0.2%      10008        unixbench.time.maximum_resident_set_size
     50139            +0.1%      50173            -0.0%      50119        unixbench.time.minor_page_faults
      4096            +0.0%       4096            +0.0%       4096        unixbench.time.page_size
     76.00            +0.0%      76.00            +0.0%      76.00        unixbench.time.percent_of_cpu_this_job_got
    261.71            -0.9%     259.28            -0.8%     259.60        unixbench.time.system_time
     36.96            +6.5%      39.38            +5.5%      39.01        unixbench.time.user_time
    903.67 ±  5%      -6.7%     843.17            -5.1%     858.00 ±  4%  unixbench.time.voluntary_context_switches
 7.585e+08            +8.1%  8.198e+08            +4.5%  7.928e+08        unixbench.workload
 3.646e+10            -0.2%   3.64e+10            +0.5%  3.665e+10        cpuidle..time
  72938593 ±  9%      -2.4%   71214588 ± 11%      +2.7%   74876854        cpuidle..usage
    427.95            -0.0%     427.80            +0.0%     428.09        uptime.boot
     39776            -0.2%      39703            +0.4%      39920        uptime.idle
     35.32            -0.1%      35.28            +0.1%      35.37        boot-time.boot
     16.63            +0.1%      16.64            +0.3%      16.68        boot-time.dhcp
      2897            -0.2%       2890            +1.0%       2926        boot-time.idle
      0.97            +0.3%       0.98            +0.2%       0.98        boot-time.smp_boot
     97.59            -0.2       97.44            +0.3       97.93        mpstat.cpu.all.idle%
      0.00 ±  5%      -0.0        0.00 ±  5%      +0.0        0.00 ±  5%  mpstat.cpu.all.iowait%
      1.48 ± 25%      +0.1        1.62 ± 19%      -0.3        1.16 ±  6%  mpstat.cpu.all.irq%
      0.08 ± 25%      +0.0        0.09 ± 23%      -0.0        0.06 ±  6%  mpstat.cpu.all.soft%
      0.72            -0.0        0.72            -0.0        0.72        mpstat.cpu.all.sys%
      0.12            +0.0        0.13            +0.0        0.13        mpstat.cpu.all.usr%
      0.00          -100.0%       0.00          -100.0%       0.00        numa-numastat.node0.interleave_hit
    508271 ± 22%      -3.5%     490587 ± 15%      +2.8%     522300 ± 14%  numa-numastat.node0.local_node
    568022 ± 18%      -3.8%     546713 ± 13%      +3.8%     589704 ± 14%  numa-numastat.node0.numa_hit
     59789 ± 50%      -6.0%      56188 ± 47%     +12.7%      67391 ± 34%  numa-numastat.node0.other_node
      0.00          -100.0%       0.00          -100.0%       0.00        numa-numastat.node1.interleave_hit
    433905 ± 25%      +4.2%     452129 ± 16%      -3.3%     419730 ± 18%  numa-numastat.node1.local_node
    460898 ± 22%      +4.7%     482594 ± 15%      -4.7%     439025 ± 19%  numa-numastat.node1.numa_hit
     26971 ±112%     +13.3%      30571 ± 86%     -28.1%      19405 ±118%  numa-numastat.node1.other_node
    390.37            +0.0%     390.37            -0.0%     390.36        time.elapsed_time
    390.37            +0.0%     390.37            -0.0%     390.36        time.elapsed_time.max
    113.78 ± 91%    -100.0%       0.00           -84.4%      17.78 ±282%  time.file_system_inputs
      3445 ± 89%    -100.0%       0.00           -82.6%     600.89 ±282%  time.file_system_outputs
    560.89 ± 10%      -1.4%     553.00 ±  6%      -5.8%     528.22 ±  3%  time.involuntary_context_switches
     10032            -0.3%      10001            -0.2%      10008        time.maximum_resident_set_size
     50139            +0.1%      50173            -0.0%      50119        time.minor_page_faults
      4096            +0.0%       4096            +0.0%       4096        time.page_size
     76.00            +0.0%      76.00            +0.0%      76.00        time.percent_of_cpu_this_job_got
    261.71            -0.9%     259.28            -0.8%     259.60        time.system_time
     36.96            +6.5%      39.38            +5.5%      39.01        time.user_time
    903.67 ±  5%      -6.7%     843.17            -5.1%     858.00 ±  4%  time.voluntary_context_switches
     96.78            -0.5%      96.33            +0.3%      97.11        vmstat.cpu.id
      1.56 ± 31%      +7.1%       1.67 ± 28%     -35.7%       1.00        vmstat.cpu.sy
      0.00          -100.0%       0.00          -100.0%       0.00        vmstat.cpu.us
      9.44 ± 89%    -100.0%       0.00           -80.0%       1.89 ±282%  vmstat.io.bi
      4.11 ± 89%    -100.0%       0.00           -81.1%       0.78 ±282%  vmstat.io.bo
      7.89 ± 44%     -49.3%       4.00           -39.4%       4.78 ± 46%  vmstat.memory.buff
   2508417            -0.1%    2504846            -0.1%    2505666        vmstat.memory.cache
 1.278e+08            +0.0%  1.278e+08            +0.0%  1.278e+08        vmstat.memory.free
      0.00          -100.0%       0.00          -100.0%       0.00        vmstat.procs.b
      0.00          -100.0%       0.00          -100.0%       0.00        vmstat.procs.r
      1488 ±  2%      +1.4%       1509 ±  9%      -0.7%       1478 ±  3%  vmstat.system.cs
    185829 ±  9%      -1.7%     182747 ± 11%      +3.3%     192023        vmstat.system.in
     67.11 ± 13%      +2.2%      68.58 ± 12%     -14.9%      57.11 ±  2%  turbostat.Avg_MHz
      3.87 ± 24%      +0.3        4.17 ± 20%      -0.9        2.97 ±  4%  turbostat.Busy%
      1794 ± 13%      -6.4%       1679 ±  9%      +7.4%       1927 ±  5%  turbostat.Bzy_MHz
    774994 ±129%     +47.6%    1143588 ±194%     -11.3%     687474 ±264%  turbostat.C1
      0.33 ±169%      +0.1        0.39 ±308%      -0.1        0.19 ±271%  turbostat.C1%
  52685178 ± 37%     -10.0%   47392459 ± 44%     +40.0%   73744498 ±  2%  turbostat.C1E
     53.64 ± 70%     -10.1       43.52 ± 87%     +42.8       96.44        turbostat.C1E%
  19397931 ± 87%     +16.4%   22575610 ± 70%     -98.1%     371152 ± 18%  turbostat.C6
     42.98 ± 88%      +9.9       52.84 ± 70%     -42.2        0.77 ± 26%  turbostat.C6%
     86.37 ± 12%      -4.2%      82.74 ± 12%     +12.3%      97.00        turbostat.CPU%c1
      9.76 ±100%     +34.1%      13.09 ± 74%     -99.7%       0.03 ± 41%  turbostat.CPU%c6
     54.56 ±  2%      -3.0%      52.92 ±  3%      -1.2%      53.89 ±  3%  turbostat.CoreTmp
  72993878 ±  9%      -1.7%   71782914 ± 11%      +3.4%   75452294        turbostat.IRQ
      6.37 ±110%     +36.5%       8.69 ± 80%     -99.9%       0.00 ±282%  turbostat.Pkg%pc2
      1.16 ±121%     +52.8%       1.78 ± 82%    -100.0%       0.00        turbostat.Pkg%pc6
     54.89 ±  2%      -3.3%      53.08 ±  3%      -1.8%      53.89 ±  3%  turbostat.PkgTmp
    107.25 ±  4%      -2.5%     104.52            +0.3%     107.56 ±  2%  turbostat.PkgWatt
     40.75            -0.6%      40.52            +0.1%      40.80        turbostat.RAMWatt
      2295            -0.0%       2294            +0.0%       2295        turbostat.TSC_MHz
     11595 ±  4%      -4.0%      11130 ±  3%      -4.1%      11123 ±  3%  meminfo.Active
     10946 ±  2%      +0.1%      10953 ±  3%      -0.9%      10846 ±  3%  meminfo.Active(anon)
    648.56 ± 63%     -72.8%     176.58 ±  2%     -57.2%     277.44 ± 87%  meminfo.Active(file)
    184807            +0.1%     184995            +0.6%     185867        meminfo.AnonHugePages
    260664            +0.4%     261637            +0.6%     262146        meminfo.AnonPages
      7.89 ± 44%     -49.3%       4.00           -39.4%       4.78 ± 46%  meminfo.Buffers
   2404339            -0.1%    2400789            -0.1%    2401307        meminfo.Cached
    201222            +0.1%     201365            +0.1%     201340        meminfo.CmaFree
    204800            +0.0%     204800            +0.0%     204800        meminfo.CmaTotal
  65839596            +0.0%   65839596            +0.0%   65839596        meminfo.CommitLimit
    365007            +0.1%     365193            -0.0%     364843        meminfo.Committed_AS
 1.221e+08            -0.1%   1.22e+08            -0.7%  1.213e+08        meminfo.DirectMap1G
  13069880 ± 14%      +1.1%   13207381 ± 15%      +6.3%   13888398 ± 17%  meminfo.DirectMap2M
    822421 ±  4%      -2.6%     801429 ±  3%      -0.4%     819463 ±  5%  meminfo.DirectMap4k
      1.11 ± 89%    -100.0%       0.00           -80.0%       0.22 ±282%  meminfo.Dirty
      2048            +0.0%       2048            +0.0%       2048        meminfo.Hugepagesize
    273336            -0.8%     271017            -0.6%     271801        meminfo.Inactive
    270008            +0.3%     270869            +0.4%     271018        meminfo.Inactive(anon)
      3327 ± 85%     -95.6%     147.25           -76.5%     782.89 ±229%  meminfo.Inactive(file)
    104056            -0.0%     104033            +0.3%     104340        meminfo.KReclaimable
     16407            +0.6%      16499            +0.8%      16546        meminfo.KernelStack
     35129            +0.3%      35223            +0.1%      35172        meminfo.Mapped
 1.272e+08            +0.0%  1.272e+08            +0.0%  1.272e+08        meminfo.MemAvailable
 1.278e+08            +0.0%  1.278e+08            +0.0%  1.278e+08        meminfo.MemFree
 1.317e+08            +0.0%  1.317e+08            +0.0%  1.317e+08        meminfo.MemTotal
   3891419            -0.1%    3887765            -0.1%    3888354        meminfo.Memused
      1100 ± 65%     +18.3%       1301 ± 55%      +0.1%       1101 ± 65%  meminfo.Mlocked
      4027            +0.0%       4028            -0.3%       4014        meminfo.PageTables
     47685            -0.0%      47665            -0.3%      47541        meminfo.Percpu
    104056            -0.0%     104033            +0.3%     104340        meminfo.SReclaimable
    195896            -0.2%     195533            +0.4%     196756        meminfo.SUnreclaim
     20527            +0.0%      20537            -0.8%      20362        meminfo.Shmem
    299953            -0.1%     299567            +0.4%     301097        meminfo.Slab
   2380001            +0.0%    2380119            +0.0%    2380047        meminfo.Unevictable
 3.436e+10            +0.0%  3.436e+10            +0.0%  3.436e+10        meminfo.VmallocTotal
    170334            +0.1%     170435            +0.1%     170510        meminfo.VmallocUsed
      0.11 ±282%    -100.0%       0.00          -100.0%       0.00        meminfo.Writeback
   4253511            +0.0%    4253640            -0.1%    4248176        meminfo.max_used_kB
      3331 ± 46%     +11.1%       3702 ± 41%     +12.9%       3761 ± 42%  numa-meminfo.node0.Active
      2685 ± 53%     +31.8%       3539 ± 42%     +31.3%       3526 ± 41%  numa-meminfo.node0.Active(anon)
    645.44 ± 63%     -74.9%     161.92 ± 30%     -63.6%     234.78 ±115%  numa-meminfo.node0.Active(file)
    176640 ±  4%      -9.6%     159633 ± 30%     -14.9%     150241 ± 38%  numa-meminfo.node0.AnonHugePages
    235558 ±  3%      -8.4%     215743 ± 27%     -13.9%     202732 ± 35%  numa-meminfo.node0.AnonPages
    243183 ±  4%      -6.7%     226936 ± 26%     -13.7%     209848 ± 32%  numa-meminfo.node0.AnonPages.max
      0.44 ±187%    -100.0%       0.00          -100.0%       0.00        numa-meminfo.node0.Dirty
   1649175 ± 58%     +20.8%    1992434 ± 38%     +27.9%    2109828 ± 31%  numa-meminfo.node0.FilePages
    244464 ±  3%     -10.0%     220102 ± 28%     -14.8%     208186 ± 33%  numa-meminfo.node0.Inactive
    242388 ±  3%      -9.3%     219966 ± 28%     -14.4%     207445 ± 33%  numa-meminfo.node0.Inactive(anon)
      2075 ±128%     -93.5%     134.92 ± 30%     -64.3%     740.33 ±240%  numa-meminfo.node0.Inactive(file)
     65344 ± 23%      +6.2%      69380 ± 24%      +7.1%      70004 ± 18%  numa-meminfo.node0.KReclaimable
      9492 ±  4%      -1.6%       9338 ±  5%      -8.6%       8674 ±  7%  numa-meminfo.node0.KernelStack
     25629 ± 41%      +8.4%      27785 ± 34%     +12.1%      28733 ± 29%  numa-meminfo.node0.Mapped
  63052050            -0.4%   62775986            -0.6%   62655763        numa-meminfo.node0.MemFree
  65665512            +0.0%   65665512            +0.0%   65665512        numa-meminfo.node0.MemTotal
   2613460 ± 38%     +10.6%    2889524 ± 28%     +15.2%    3009747 ± 23%  numa-meminfo.node0.MemUsed
    831.11 ± 92%     +45.9%       1212 ± 64%     +26.2%       1049 ± 73%  numa-meminfo.node0.Mlocked
      3019 ±  5%      -6.8%       2813 ± 21%     -13.4%       2615 ± 26%  numa-meminfo.node0.PageTables
     65344 ± 23%      +6.2%      69380 ± 24%      +7.1%      70004 ± 18%  numa-meminfo.node0.SReclaimable
    114422 ±  7%      -5.1%     108617 ± 12%      -7.7%     105637 ±  8%  numa-meminfo.node0.SUnreclaim
      9968 ± 44%     -17.9%       8180 ± 59%     -12.7%       8702 ± 57%  numa-meminfo.node0.Shmem
    179767 ±  8%      -1.0%     177998 ± 15%      -2.3%     175641 ± 11%  numa-meminfo.node0.Slab
   1636644 ± 59%     +21.2%    1984121 ± 38%     +28.3%    2100300 ± 31%  numa-meminfo.node0.Unevictable
      0.11 ±282%    -100.0%       0.00          -100.0%       0.00        numa-meminfo.node0.Writeback
      8243 ± 16%     -10.1%       7411 ± 21%     -10.9%       7347 ± 22%  numa-meminfo.node1.Active
      8240 ± 16%     -10.2%       7397 ± 21%     -11.4%       7304 ± 22%  numa-meminfo.node1.Active(anon)
      3.11 ±187%    +371.4%      14.67 ±331%   +1271.4%      42.67 ±187%  numa-meminfo.node1.Active(file)
      8134 ± 95%    +211.4%      25330 ±193%    +337.7%      35606 ±163%  numa-meminfo.node1.AnonHugePages
     25161 ± 30%     +82.6%      45942 ±131%    +136.3%      59456 ±120%  numa-meminfo.node1.AnonPages
     51771 ± 22%     +39.7%      72336 ± 82%     +67.2%      86537 ± 80%  numa-meminfo.node1.AnonPages.max
      0.67 ±141%    -100.0%       0.00           -66.7%       0.22 ±282%  numa-meminfo.node1.Dirty
    755159 ±128%     -45.9%     408353 ±189%     -61.4%     291472 ±224%  numa-meminfo.node1.FilePages
     28934 ± 29%     +76.2%      50974 ±121%    +120.0%      63661 ±109%  numa-meminfo.node1.Inactive
     27681 ± 30%     +84.1%      50962 ±121%    +129.8%      63618 ±108%  numa-meminfo.node1.Inactive(anon)
      1252 ±185%     -99.0%      12.33 ±331%     -96.6%      42.44 ±146%  numa-meminfo.node1.Inactive(file)
     38709 ± 37%     -10.5%      34650 ± 50%     -11.3%      34333 ± 39%  numa-meminfo.node1.KReclaimable
      6917 ±  5%      +3.5%       7160 ±  8%     +13.8%       7872 ±  8%  numa-meminfo.node1.KernelStack
      9521 ±110%     -21.6%       7460 ±125%     -32.2%       6452 ±132%  numa-meminfo.node1.Mapped
  64734839            +0.4%   65014696            +0.6%   65134389        numa-meminfo.node1.MemFree
  66013684            +0.0%   66013684            +0.0%   66013684        numa-meminfo.node1.MemTotal
   1278843 ± 79%     -21.9%     998986 ± 83%     -31.2%     879293 ± 79%  numa-meminfo.node1.MemUsed
    267.67 ±189%     -67.2%      87.67 ±175%     -81.2%      50.33 ±240%  numa-meminfo.node1.Mlocked
      1007 ± 16%     +20.5%       1213 ± 50%     +38.9%       1399 ± 49%  numa-meminfo.node1.PageTables
     38709 ± 37%     -10.5%      34650 ± 50%     -11.3%      34333 ± 39%  numa-meminfo.node1.SReclaimable
     81477 ± 12%      +6.7%      86919 ± 18%     +11.8%      91120 ± 11%  numa-meminfo.node1.SUnreclaim
     10546 ± 40%     +17.1%      12352 ± 38%     +10.4%      11647 ± 41%  numa-meminfo.node1.Shmem
    120186 ± 12%      +1.2%     121570 ± 24%      +4.4%     125454 ± 18%  numa-meminfo.node1.Slab
    743356 ±130%     -46.7%     395996 ±195%     -62.4%     279745 ±233%  numa-meminfo.node1.Unevictable
    392.00            +0.0%     392.00            +0.0%     392.00        proc-vmstat.cma_alloc_success
      0.00          -100.0%       0.00          -100.0%       0.00        proc-vmstat.compact_isolated
    385.67 ±  5%      -2.7%     375.42 ±  4%      -0.4%     384.22 ±  5%  proc-vmstat.direct_map_level2_splits
      9.56 ± 18%      +0.3%       9.58 ± 22%      +7.0%      10.22 ± 22%  proc-vmstat.direct_map_level3_splits
      2734 ±  2%      +0.1%       2737 ±  3%      -0.8%       2711 ±  3%  proc-vmstat.nr_active_anon
    161.78 ± 63%     -73.1%      43.58 ±  2%     -57.5%      68.78 ± 88%  proc-vmstat.nr_active_file
     65169            +0.4%      65410            +0.6%      65536        proc-vmstat.nr_anon_pages
     89.78            -0.0%      89.75            +0.7%      90.44        proc-vmstat.nr_anon_transparent_hugepages
    444.00 ± 89%    -100.0%       0.00           -82.5%      77.78 ±282%  proc-vmstat.nr_dirtied
      0.00          -100.0%       0.00          -100.0%       0.00        proc-vmstat.nr_dirty
   3174707            +0.0%    3174712            +0.0%    3174717        proc-vmstat.nr_dirty_background_threshold
   6357177            +0.0%    6357187            +0.0%    6357197        proc-vmstat.nr_dirty_threshold
    601084            -0.1%     600197            -0.1%     600327        proc-vmstat.nr_file_pages
     50305            +0.1%      50341            +0.1%      50335        proc-vmstat.nr_free_cma
  31946873            +0.0%   31947836            +0.0%   31947701        proc-vmstat.nr_free_pages
     67505            +0.3%      67718            +0.4%      67754        proc-vmstat.nr_inactive_anon
    831.33 ± 85%     -95.6%      36.25           -76.5%     195.33 ±229%  proc-vmstat.nr_inactive_file
     16409            +0.5%      16497            +0.8%      16547        proc-vmstat.nr_kernel_stack
      8782            +0.3%       8805            +0.1%       8793        proc-vmstat.nr_mapped
    274.44 ± 65%     +18.4%     324.83 ± 55%      +0.2%     274.89 ± 65%  proc-vmstat.nr_mlock
      1006            -0.0%       1006            -0.3%       1003        proc-vmstat.nr_page_table_pages
      5129            +0.1%       5133            -0.8%       5090        proc-vmstat.nr_shmem
     26013            -0.0%      26007            +0.3%      26084        proc-vmstat.nr_slab_reclaimable
     48974            -0.2%      48883            +0.4%      49188        proc-vmstat.nr_slab_unreclaimable
    595000            +0.0%     595029            +0.0%     595011        proc-vmstat.nr_unevictable
    444.00 ± 89%    -100.0%       0.00           -82.5%      77.78 ±282%  proc-vmstat.nr_written
      2734 ±  2%      +0.1%       2737 ±  3%      -0.8%       2711 ±  3%  proc-vmstat.nr_zone_active_anon
    161.78 ± 63%     -73.1%      43.58 ±  2%     -57.5%      68.78 ± 88%  proc-vmstat.nr_zone_active_file
     67505            +0.3%      67718            +0.4%      67754        proc-vmstat.nr_zone_inactive_anon
    831.33 ± 85%     -95.6%      36.25           -76.5%     195.33 ±229%  proc-vmstat.nr_zone_inactive_file
    595000            +0.0%     595029            +0.0%     595011        proc-vmstat.nr_zone_unevictable
      0.00          -100.0%       0.00          -100.0%       0.00        proc-vmstat.nr_zone_write_pending
    528.00 ±166%    +144.6%       1291 ±202%      +7.7%     568.56 ±251%  proc-vmstat.numa_hint_faults
    297.78 ±238%    +117.2%     646.75 ±301%     -82.6%      51.78 ± 33%  proc-vmstat.numa_hint_faults_local
   1033052            +0.1%    1034106            +0.5%    1037901        proc-vmstat.numa_hit
     58.67 ± 35%      -1.8%      57.58 ± 37%      -5.3%      55.56 ± 36%  proc-vmstat.numa_huge_pte_updates
      0.00          -100.0%       0.00          -100.0%       0.00        proc-vmstat.numa_interleave
    946308            +0.2%     948540            +0.5%     951201        proc-vmstat.numa_local
     86760            -0.0%      86760            +0.0%      86797        proc-vmstat.numa_other
    684.11 ±252%    +224.6%       2220 ±241%    +257.7%       2447 ±280%  proc-vmstat.numa_pages_migrated
     32541 ± 38%      +6.3%      34605 ± 48%      -4.9%      30940 ± 53%  proc-vmstat.numa_pte_updates
      3208 ±  8%      -6.3%       3005 ±  9%      -4.6%       3060 ±  8%  proc-vmstat.pgactivate
      0.00          -100.0%       0.00          -100.0%       0.00        proc-vmstat.pgalloc_dma
      0.00          -100.0%       0.00          -100.0%       0.00        proc-vmstat.pgalloc_dma32
   1035736            +0.0%    1035978            +0.0%    1035903        proc-vmstat.pgalloc_normal
   1168271            +0.2%    1170241            +0.1%    1169796        proc-vmstat.pgfault
   1070341            +0.1%    1071645            +0.1%    1071935        proc-vmstat.pgfree
    684.11 ±252%    +224.6%       2220 ±241%    +257.7%       2447 ±280%  proc-vmstat.pgmigrate_success
      3763 ± 89%    -100.0%       0.00           -80.1%     750.22 ±282%  proc-vmstat.pgpgin
      1789 ± 89%    -100.0%       0.00           -82.5%     313.78 ±282%  proc-vmstat.pgpgout
     83209            +0.0%      83237            +0.7%      83775        proc-vmstat.pgreuse
     97.22            -0.3%      96.92            -0.1%      97.11        proc-vmstat.thp_collapse_alloc
      0.78 ±157%     -35.7%       0.50 ±223%     -85.7%       0.11 ±282%  proc-vmstat.thp_deferred_split_page
     12.33 ±  5%      +0.7%      12.42 ±  5%      -2.7%      12.00        proc-vmstat.thp_fault_alloc
      0.89 ±245%    +246.9%       3.08 ±322%    +325.0%       3.78 ±282%  proc-vmstat.thp_migration_success
      0.78 ±157%     -35.7%       0.50 ±223%     -85.7%       0.11 ±282%  proc-vmstat.thp_split_pmd
      0.00          -100.0%       0.00          -100.0%       0.00        proc-vmstat.thp_zero_page_alloc
    113.11            +0.9%     114.17            +0.7%     113.89        proc-vmstat.unevictable_pgs_culled
    586.00            +0.0%     586.00            +0.0%     586.00        proc-vmstat.unevictable_pgs_mlocked
    671.00 ± 53%     +31.8%     884.58 ± 42%     +31.3%     881.11 ± 41%  numa-vmstat.node0.nr_active_anon
    161.00 ± 63%     -75.2%      39.92 ± 30%     -63.8%      58.22 ±116%  numa-vmstat.node0.nr_active_file
     58889 ±  3%      -8.4%      53935 ± 27%     -13.9%      50683 ± 35%  numa-vmstat.node0.nr_anon_pages
     85.67 ±  4%      -9.6%      77.42 ± 30%     -14.9%      72.89 ± 39%  numa-vmstat.node0.nr_anon_transparent_hugepages
    376.67 ± 89%    -100.0%       0.00           -82.7%      65.11 ±282%  numa-vmstat.node0.nr_dirtied
      0.00          -100.0%       0.00          -100.0%       0.00        numa-vmstat.node0.nr_dirty
    412293 ± 58%     +20.8%     498108 ± 38%     +27.9%     527456 ± 31%  numa-vmstat.node0.nr_file_pages
     50305            +0.1%      50341            +0.1%      50335        numa-vmstat.node0.nr_free_cma
  15763023            -0.4%   15693998            -0.6%   15663929        numa-vmstat.node0.nr_free_pages
     60596 ±  3%      -9.2%      54991 ± 28%     -14.4%      51861 ± 33%  numa-vmstat.node0.nr_inactive_anon
    518.78 ±128%     -93.6%      33.17 ± 30%     -64.4%     184.78 ±241%  numa-vmstat.node0.nr_inactive_file
      0.00          -100.0%       0.00          -100.0%       0.00        numa-vmstat.node0.nr_isolated_anon
      9491 ±  4%      -1.6%       9341 ±  5%      -8.6%       8676 ±  7%  numa-vmstat.node0.nr_kernel_stack
      6408 ± 41%      +8.4%       6946 ± 34%     +12.1%       7184 ± 29%  numa-vmstat.node0.nr_mapped
    207.56 ± 93%     +45.9%     302.83 ± 64%     +26.2%     261.89 ± 73%  numa-vmstat.node0.nr_mlock
    753.89 ±  5%      -6.7%     703.67 ± 21%     -13.3%     653.78 ± 26%  numa-vmstat.node0.nr_page_table_pages
      2492 ± 44%     -17.9%       2044 ± 59%     -12.7%       2175 ± 57%  numa-vmstat.node0.nr_shmem
     16335 ± 23%      +6.2%      17344 ± 24%      +7.1%      17500 ± 18%  numa-vmstat.node0.nr_slab_reclaimable
     28605 ±  7%      -5.1%      27153 ± 12%      -7.7%      26408 ±  8%  numa-vmstat.node0.nr_slab_unreclaimable
    409160 ± 59%     +21.2%     496029 ± 38%     +28.3%     525074 ± 31%  numa-vmstat.node0.nr_unevictable
      0.00          -100.0%       0.00          -100.0%       0.00        numa-vmstat.node0.nr_writeback
    376.44 ± 89%    -100.0%       0.00           -82.7%      65.11 ±282%  numa-vmstat.node0.nr_written
    671.00 ± 53%     +31.8%     884.58 ± 42%     +31.3%     881.11 ± 41%  numa-vmstat.node0.nr_zone_active_anon
    161.00 ± 63%     -75.2%      39.92 ± 30%     -63.8%      58.22 ±116%  numa-vmstat.node0.nr_zone_active_file
     60596 ±  3%      -9.2%      54991 ± 28%     -14.4%      51861 ± 33%  numa-vmstat.node0.nr_zone_inactive_anon
    518.78 ±128%     -93.6%      33.17 ± 30%     -64.4%     184.78 ±241%  numa-vmstat.node0.nr_zone_inactive_file
    409160 ± 59%     +21.2%     496029 ± 38%     +28.3%     525074 ± 31%  numa-vmstat.node0.nr_zone_unevictable
      0.00          -100.0%       0.00          -100.0%       0.00        numa-vmstat.node0.nr_zone_write_pending
   1661451 ± 13%      +2.6%    1704304 ± 15%      +5.4%    1751623 ± 12%  numa-vmstat.node0.numa_hit
      3894 ±  8%      +0.3%       3905 ±  5%      -0.6%       3872 ±  6%  numa-vmstat.node0.numa_interleave
   1598537 ± 14%      +2.9%    1645037 ± 16%      +5.1%    1680768 ± 12%  numa-vmstat.node0.numa_local
     62954 ± 49%      -4.3%      60267 ± 44%     +12.6%      70856 ± 30%  numa-vmstat.node0.numa_other
      2060 ± 16%     -10.2%       1849 ± 21%     -11.4%       1825 ± 22%  numa-vmstat.node1.nr_active_anon
      0.67 ±187%    +450.0%       3.67 ±331%   +1500.0%      10.67 ±187%  numa-vmstat.node1.nr_active_file
      6291 ± 30%     +82.6%      11487 ±131%    +136.3%      14866 ±120%  numa-vmstat.node1.nr_anon_pages
      3.44 ±105%    +243.5%      11.83 ±201%    +383.9%      16.67 ±171%  numa-vmstat.node1.nr_anon_transparent_hugepages
      7.00 ±141%    -100.0%       0.00           -66.7%       2.33 ±282%  numa-vmstat.node1.nr_dirtied
      0.00          -100.0%       0.00          -100.0%       0.00        numa-vmstat.node1.nr_dirty
    188791 ±128%     -45.9%     102090 ±189%     -61.4%      72868 ±224%  numa-vmstat.node1.nr_file_pages
  16183708            +0.4%   16253674            +0.6%   16283582        numa-vmstat.node1.nr_free_pages
      6923 ± 30%     +84.1%      12744 ±121%    +129.8%      15907 ±108%  numa-vmstat.node1.nr_inactive_anon
    313.11 ±185%     -99.0%       3.08 ±331%     -96.6%      10.56 ±147%  numa-vmstat.node1.nr_inactive_file
      6916 ±  5%      +3.5%       7159 ±  8%     +13.8%       7874 ±  8%  numa-vmstat.node1.nr_kernel_stack
      2381 ±110%     -21.6%       1866 ±125%     -32.2%       1613 ±132%  numa-vmstat.node1.nr_mapped
     66.89 ±189%     -67.2%      21.92 ±175%     -81.2%      12.56 ±241%  numa-vmstat.node1.nr_mlock
    251.33 ± 15%     +20.5%     302.75 ± 50%     +39.3%     350.11 ± 49%  numa-vmstat.node1.nr_page_table_pages
      2638 ± 40%     +17.1%       3089 ± 38%     +10.4%       2912 ± 41%  numa-vmstat.node1.nr_shmem
      9677 ± 37%     -10.5%       8662 ± 50%     -11.3%       8582 ± 39%  numa-vmstat.node1.nr_slab_reclaimable
     20368 ± 12%      +6.7%      21729 ± 18%     +11.8%      22779 ± 11%  numa-vmstat.node1.nr_slab_unreclaimable
    185838 ±130%     -46.7%      98998 ±195%     -62.4%      69935 ±233%  numa-vmstat.node1.nr_unevictable
      7.00 ±141%    -100.0%       0.00           -66.7%       2.33 ±282%  numa-vmstat.node1.nr_written
      2060 ± 16%     -10.2%       1849 ± 21%     -11.4%       1825 ± 22%  numa-vmstat.node1.nr_zone_active_anon
      0.67 ±187%    +450.0%       3.67 ±331%   +1500.0%      10.67 ±187%  numa-vmstat.node1.nr_zone_active_file
      6923 ± 30%     +84.1%      12743 ±121%    +129.8%      15907 ±108%  numa-vmstat.node1.nr_zone_inactive_anon
    313.11 ±185%     -99.0%       3.08 ±331%     -96.6%      10.56 ±147%  numa-vmstat.node1.nr_zone_inactive_file
    185838 ±130%     -46.7%      98998 ±195%     -62.4%      69935 ±233%  numa-vmstat.node1.nr_zone_unevictable
      0.00          -100.0%       0.00          -100.0%       0.00        numa-vmstat.node1.nr_zone_write_pending
    923522 ± 24%      -4.1%     886107 ± 29%      -9.2%     838695 ± 26%  numa-vmstat.node1.numa_hit
      3652 ±  8%      -0.1%       3650 ±  6%      +0.9%       3685 ±  6%  numa-vmstat.node1.numa_interleave
    886782 ± 26%      -4.5%     846839 ± 32%      -8.7%     810030 ± 25%  numa-vmstat.node1.numa_local
     36727 ± 84%      +7.2%      39378 ± 68%     -21.7%      28741 ± 76%  numa-vmstat.node1.numa_other
     13.36 ± 85%     +28.8%      17.20 ± 67%     -89.9%       1.36 ±  3%  perf-stat.i.MPKI
 9.529e+08            -1.6%   9.38e+08            -2.1%  9.327e+08        perf-stat.i.branch-instructions
      2.12 ± 53%      +0.3        2.37 ± 48%      -1.2        0.92        perf-stat.i.branch-miss-rate%
  12593796 ± 34%      -0.6%   12517767 ± 34%     -37.8%    7836536        perf-stat.i.branch-misses
      5.54 ± 36%      -0.5        5.07 ± 41%      +2.6        8.12 ±  5%  perf-stat.i.cache-miss-rate%
   1100829 ± 63%     +15.7%    1273764 ± 52%     -69.8%     332573 ±  7%  perf-stat.i.cache-misses
  27486243 ± 77%     +23.0%   33803685 ± 62%     -85.7%    3933550 ±  3%  perf-stat.i.cache-references
      1422 ±  2%      +1.5%       1443 ±  9%      -0.6%       1414 ±  3%  perf-stat.i.context-switches
      1.94 ± 26%      +6.6%       2.07 ± 25%     -30.5%       1.35 ±  3%  perf-stat.i.cpi
     96002            +0.0%      96003            +0.0%      96002        perf-stat.i.cpu-clock
 5.879e+09 ± 14%      +1.5%  5.968e+09 ± 14%     -17.4%  4.854e+09 ±  3%  perf-stat.i.cpu-cycles
     98.53            +0.0%      98.54            +0.1%      98.64        perf-stat.i.cpu-migrations
     10513 ± 68%     -18.8%       8541 ± 74%     +61.8%      17015 ±  8%  perf-stat.i.cycles-between-cache-misses
      0.15 ± 89%      +0.0        0.18 ± 71%      -0.1        0.00 ±  9%  perf-stat.i.dTLB-load-miss-rate%
    882684 ± 88%     +16.5%    1027981 ± 71%     -98.4%      14258 ±  8%  perf-stat.i.dTLB-load-misses
 1.433e+09            +0.0%  1.434e+09            -0.7%  1.423e+09        perf-stat.i.dTLB-loads
      0.03 ± 86%      +0.0        0.04 ± 69%      -0.0        0.00 ±  2%  perf-stat.i.dTLB-store-miss-rate%
    112250 ± 83%     +15.1%     129192 ± 67%     -93.3%       7499        perf-stat.i.dTLB-store-misses
  8.75e+08            -3.8%  8.418e+08            -5.3%  8.285e+08        perf-stat.i.dTLB-stores
     76.52 ±  6%      -0.5       75.98 ±  6%      -3.5       73.02        perf-stat.i.iTLB-load-miss-rate%
   5320929           -18.5%    4337810 ±  4%      +0.8%    5364196        perf-stat.i.iTLB-load-misses
   1210384 ± 24%     -11.2%    1074564 ± 26%     +19.4%    1444799 ±  2%  perf-stat.i.iTLB-loads
  4.83e+09            -1.1%  4.775e+09            -1.7%  4.747e+09        perf-stat.i.instructions
    980.79 ±  2%     +19.7%       1174 ±  6%      -2.7%     954.36        perf-stat.i.instructions-per-iTLB-miss
      0.80 ± 12%      -1.9%       0.79 ± 13%     +13.9%       0.92 ±  2%  perf-stat.i.ipc
      0.61 ± 22%      +5.4%       0.64 ± 17%      +5.0%       0.64 ± 23%  perf-stat.i.major-faults
      0.06 ± 14%      +1.5%       0.06 ± 14%     -17.4%       0.05 ±  3%  perf-stat.i.metric.GHz
    313.74 ± 75%     +23.1%     386.17 ± 61%     -82.1%      56.26 ±  2%  perf-stat.i.metric.K/sec
     33.96            -1.5%      33.45            -2.3%      33.17        perf-stat.i.metric.M/sec
      2694            -0.0%       2694            +0.1%       2696        perf-stat.i.minor-faults
     86.80 ±  2%      +0.5       87.28 ±  2%      +3.7       90.50        perf-stat.i.node-load-miss-rate%
     58546 ± 16%      +5.7%      61885 ± 17%     +24.1%      72659 ±  8%  perf-stat.i.node-load-misses
     11768 ±  8%      -1.6%      11585 ±  8%     -15.6%       9928 ± 16%  perf-stat.i.node-loads
     73.58 ± 16%      -0.4       73.15 ± 25%     +13.8       87.41 ± 17%  perf-stat.i.node-store-miss-rate%
     25184 ± 14%      +6.3%      26770 ± 28%     +20.4%      30317 ± 19%  perf-stat.i.node-store-misses
     11083 ± 36%      +3.4%      11464 ± 50%     -44.3%       6175 ± 72%  perf-stat.i.node-stores
      2695            -0.0%       2694            +0.1%       2697        perf-stat.i.page-faults
     96002            +0.0%      96003            +0.0%      96002        perf-stat.i.task-clock
      5.71 ± 77%     +24.1%       7.09 ± 62%     -85.5%       0.83 ±  3%  perf-stat.overall.MPKI
      1.32 ± 34%      +0.0        1.34 ± 34%      -0.5        0.84        perf-stat.overall.branch-miss-rate%
      5.70 ± 38%      -0.5        5.18 ± 43%      +2.7        8.45 ±  5%  perf-stat.overall.cache-miss-rate%
      1.22 ± 14%      +2.6%       1.25 ± 14%     -16.0%       1.02 ±  2%  perf-stat.overall.cpi
      9179 ± 65%     -18.8%       7449 ± 70%     +59.8%      14669 ±  7%  perf-stat.overall.cycles-between-cache-misses
      0.06 ± 88%      +0.0        0.07 ± 71%      -0.1        0.00 ±  8%  perf-stat.overall.dTLB-load-miss-rate%
      0.01 ± 83%      +0.0        0.02 ± 67%      -0.0        0.00        perf-stat.overall.dTLB-store-miss-rate%
     81.65 ±  4%      -1.2       80.41 ±  4%      -2.9       78.78        perf-stat.overall.iTLB-load-miss-rate%
    907.89           +21.6%       1103 ±  5%      -2.5%     885.40 ±  2%  perf-stat.overall.instructions-per-iTLB-miss
      0.84 ± 14%      -2.5%       0.82 ± 15%     +16.7%       0.98 ±  2%  perf-stat.overall.ipc
     82.90 ±  3%      +1.0       83.91 ±  3%      +5.0       87.92 ±  2%  perf-stat.overall.node-load-miss-rate%
     69.61 ± 15%      -0.0       69.61 ± 23%     +13.0       82.57 ± 17%  perf-stat.overall.node-store-miss-rate%
      2484            -8.6%       2271            -6.0%       2335        perf-stat.overall.path-length
 9.504e+08            -1.6%  9.354e+08            -2.1%  9.303e+08        perf-stat.ps.branch-instructions
  12563461 ± 34%      -0.6%   12486690 ± 34%     -37.8%    7817966        perf-stat.ps.branch-misses
   1098102 ± 63%     +15.7%    1270644 ± 52%     -69.8%     331694 ±  7%  perf-stat.ps.cache-misses
  27418760 ± 77%     +23.0%   33722634 ± 62%     -85.7%    3923602 ±  3%  perf-stat.ps.cache-references
      1418 ±  2%      +1.5%       1439 ±  9%      -0.6%       1410 ±  3%  perf-stat.ps.context-switches
     95755            +0.0%      95755            +0.0%      95755        perf-stat.ps.cpu-clock
 5.864e+09 ± 14%      +1.5%  5.952e+09 ± 14%     -17.4%  4.841e+09 ±  3%  perf-stat.ps.cpu-cycles
     98.28            +0.0%      98.28            +0.1%      98.39        perf-stat.ps.cpu-migrations
    880507 ± 88%     +16.5%    1025511 ± 71%     -98.4%      14222 ±  8%  perf-stat.ps.dTLB-load-misses
 1.429e+09            +0.0%   1.43e+09            -0.7%   1.42e+09        perf-stat.ps.dTLB-loads
    111971 ± 83%     +15.1%     128879 ± 67%     -93.3%       7478        perf-stat.ps.dTLB-store-misses
 8.727e+08            -3.8%  8.394e+08            -5.3%  8.264e+08        perf-stat.ps.dTLB-stores
   5306748           -18.5%    4326031 ±  4%      +0.8%    5350615        perf-stat.ps.iTLB-load-misses
   1207238 ± 24%     -11.2%    1071729 ± 26%     +19.4%    1441083 ±  2%  perf-stat.ps.iTLB-loads
 4.817e+09            -1.2%  4.762e+09            -1.7%  4.735e+09        perf-stat.ps.instructions
      0.61 ± 22%      +5.4%       0.64 ± 17%      +5.1%       0.64 ± 23%  perf-stat.ps.major-faults
      2687            -0.0%       2686            +0.1%       2689        perf-stat.ps.minor-faults
     58400 ± 16%      +5.7%      61728 ± 17%     +24.1%      72475 ±  8%  perf-stat.ps.node-load-misses
     11737 ±  8%      -1.6%      11551 ±  8%     -15.6%       9900 ± 16%  perf-stat.ps.node-loads
     25118 ± 14%      +6.3%      26701 ± 28%     +20.4%      30237 ± 19%  perf-stat.ps.node-store-misses
     11051 ± 36%      +3.4%      11430 ± 50%     -44.3%       6155 ± 72%  perf-stat.ps.node-stores
      2687            -0.0%       2687            +0.1%       2689        perf-stat.ps.page-faults
     95755            +0.0%      95755            +0.0%      95755        perf-stat.ps.task-clock
 1.885e+12            -1.2%  1.862e+12            -1.8%  1.852e+12        perf-stat.total.instructions
      0.01 ± 50%     +10.2%       0.01 ± 46%     -20.5%       0.01 ± 31%  perf-sched.sch_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      0.00 ±161%    +221.9%       0.01 ±104%     -41.7%       0.00 ±142%  perf-sched.sch_delay.avg.ms.cleaner_kthread.kthread.ret_from_fork
      0.00        +1.7e+98%       0.00 ±331%  +3.3e+98%       0.00 ±282%  perf-sched.sch_delay.avg.ms.d_alloc_parallel.__lookup_slow.walk_component.link_path_walk.part
      0.02 ±129%     -45.2%       0.01 ± 19%     -50.4%       0.01 ± 11%  perf-sched.sch_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      0.01 ± 49%      +6.5%       0.01 ± 52%     -36.4%       0.01 ± 22%  perf-sched.sch_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      0.01 ± 14%     -12.3%       0.01 ± 32%      +7.5%       0.01 ± 30%  perf-sched.sch_delay.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
      0.01 ± 24%   +5694.8%       0.34 ±325%     -28.3%       0.00 ±  9%  perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.01 ± 60%     +17.4%       0.01 ± 55%     -47.8%       0.00        perf-sched.sch_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.00 ±282%    -100.0%       0.00          +250.0%       0.00 ±282%  perf-sched.sch_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault.[unknown]
      0.00          +1e+99%       0.00 ±212%  +5.6e+98%       0.00 ±191%  perf-sched.sch_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      0.00 ± 35%      +3.1%       0.00 ± 91%     -25.0%       0.00 ± 35%  perf-sched.sch_delay.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00 ±282%    -100.0%       0.00           +75.0%       0.00 ±282%  perf-sched.sch_delay.avg.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
      0.00        +3.3e+98%       0.00 ±331%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.io_schedule.__lock_page.pagecache_get_page.shmem_getpage_gfp
      0.00 ±282%     +82.1%       0.00 ±169%    +200.0%       0.00 ±170%  perf-sched.sch_delay.avg.ms.io_schedule.__lock_page_killable.filemap_fault.__do_fault
      0.00 ± 52%     +22.4%       0.00 ± 48%     -52.6%       0.00        perf-sched.sch_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      0.00 ±168%     -51.4%       0.00 ±224%     -73.0%       0.00 ±282%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.__flush_work.lru_add_drain_all.khugepaged
      0.00 ±282%    -100.0%       0.00          -100.0%       0.00        perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.copy_page_to_iter.pipe_read.new_sync_read
      0.00        +2.5e+98%       0.00 ±331%  +3.3e+98%       0.00 ±282%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
      0.00 ±282%    +800.0%       0.00 ±316%      +0.0%       0.00 ±282%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.mutex_lock.pipe_read.new_sync_read
      0.00 ±282%    +275.0%       0.00 ±234%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.mutex_lock.pipe_write.new_sync_write
      0.00        +1.7e+98%       0.00 ±331%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.sched_exec.bprm_execve
      0.01 ±111%     +51.3%       0.02 ± 90%     -49.6%       0.01 ± 17%  perf-sched.sch_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
      0.00          -100.0%       0.00        +7.8e+98%       0.00 ±282%  perf-sched.sch_delay.avg.ms.rwsem_down_read_slowpath.down_read_killable.__access_remote_vm.proc_pid_cmdline_read
      0.00          -100.0%       0.00          +1e+99%       0.00 ±282%  perf-sched.sch_delay.avg.ms.rwsem_down_read_slowpath.down_read_killable.mm_access.proc_mem_open
      0.01 ± 34%     +19.6%       0.01 ± 36%     -15.1%       0.01 ± 28%  perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      0.01 ±101%    +137.9%       0.02 ± 94%     -33.3%       0.01 ± 27%  perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
      0.01 ± 27%      +4.5%       0.01 ± 33%     -26.2%       0.01 ± 16%  perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
      0.03 ± 62%     -20.1%       0.02 ± 32%     -46.2%       0.02 ± 37%  perf-sched.sch_delay.avg.ms.schedule_timeout.__skb_wait_for_more_packets.unix_dgram_recvmsg.__sys_recvfrom
      0.01 ± 40%      -9.0%       0.01 ± 37%     -32.5%       0.01 ± 21%  perf-sched.sch_delay.avg.ms.schedule_timeout.io_schedule_timeout.wait_for_completion_io_timeout.blk_execute_rq
      0.01 ± 33%     +11.4%       0.01 ± 29%     -32.3%       0.01 ±  6%  perf-sched.sch_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.00        +7.2e+99%       0.01 ± 85%  +3.7e+99%       0.00 ±145%  perf-sched.sch_delay.avg.ms.schedule_timeout.khugepaged.kthread.ret_from_fork
      0.02 ± 81%      -0.2%       0.02 ± 79%     -50.0%       0.01 ± 27%  perf-sched.sch_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      0.00 ±208%    +228.1%       0.01 ± 86%     -37.5%       0.00 ±144%  perf-sched.sch_delay.avg.ms.schedule_timeout.transaction_kthread.kthread.ret_from_fork
      0.01 ± 91%     +71.2%       0.02 ± 69%      +0.0%       0.01 ± 37%  perf-sched.sch_delay.avg.ms.schedule_timeout.wait_for_completion.__flush_work.lru_add_drain_all
      0.00 ± 36%      -0.6%       0.00 ± 33%      -7.5%       0.00 ± 35%  perf-sched.sch_delay.avg.ms.schedule_timeout.wait_for_completion_interruptible.usb_stor_control_thread.kthread
      0.01 ± 25%     +27.4%       0.01 ± 70%     -19.4%       0.01 ± 18%  perf-sched.sch_delay.avg.ms.schedule_timeout.wait_for_completion_interruptible_timeout.usb_stor_msg_common.usb_stor_bulk_transfer_buf
      0.01 ± 35%     -10.9%       0.01 ± 13%     -26.6%       0.01 ±  7%  perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
      0.00 ± 19%      +1.1%       0.00 ± 19%     -17.4%       0.00 ± 14%  perf-sched.sch_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
      0.01 ± 11%      +8.2%       0.01 ± 14%     -12.4%       0.01 ±  4%  perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork
      0.02 ± 32%      +4.7%       0.02 ± 40%     -38.5%       0.01 ± 11%  perf-sched.sch_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      0.00 ±161%    +221.9%       0.01 ±104%     -41.7%       0.00 ±142%  perf-sched.sch_delay.max.ms.cleaner_kthread.kthread.ret_from_fork
      0.00        +1.7e+98%       0.00 ±331%  +3.3e+98%       0.00 ±282%  perf-sched.sch_delay.max.ms.d_alloc_parallel.__lookup_slow.walk_component.link_path_walk.part
      0.02 ±129%     -47.5%       0.01 ± 19%     -51.0%       0.01 ±  8%  perf-sched.sch_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      0.02 ± 54%      +1.7%       0.02 ± 32%     -36.2%       0.01 ± 28%  perf-sched.sch_delay.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      0.01 ± 21%     -13.1%       0.01 ± 37%      +6.3%       0.01 ± 41%  perf-sched.sch_delay.max.ms.do_syslog.part.0.kmsg_read.vfs_read
      0.04 ± 45%  +1.9e+05%      83.41 ±331%     -49.1%       0.02 ± 26%  perf-sched.sch_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.06 ± 91%     +50.4%       0.09 ± 65%     -82.7%       0.01 ± 21%  perf-sched.sch_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.00 ±282%    -100.0%       0.00          +250.0%       0.00 ±282%  perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault.[unknown]
      0.00        +1.1e+99%       0.00 ±197%  +6.7e+98%       0.00 ±187%  perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      0.01 ± 18%     +71.2%       0.01 ±129%      -5.7%       0.01 ± 38%  perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00 ±282%    -100.0%       0.00           +83.3%       0.00 ±282%  perf-sched.sch_delay.max.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
      0.00        +3.3e+98%       0.00 ±331%    -100.0%       0.00        perf-sched.sch_delay.max.ms.io_schedule.__lock_page.pagecache_get_page.shmem_getpage_gfp
      0.00 ±282%     +82.1%       0.00 ±169%    +200.0%       0.00 ±170%  perf-sched.sch_delay.max.ms.io_schedule.__lock_page_killable.filemap_fault.__do_fault
      0.09 ± 76%     +23.3%       0.11 ± 60%     -77.4%       0.02 ± 31%  perf-sched.sch_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      0.00 ±168%     -51.4%       0.00 ±224%     -73.0%       0.00 ±282%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.__flush_work.lru_add_drain_all.khugepaged
      0.00 ±282%    -100.0%       0.00          -100.0%       0.00        perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.copy_page_to_iter.pipe_read.new_sync_read
      0.00        +2.5e+98%       0.00 ±331%  +3.3e+98%       0.00 ±282%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
      0.00 ±282%    +512.5%       0.00 ±310%     -33.3%       0.00 ±282%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.mutex_lock.pipe_read.new_sync_read
      0.00 ±282%     +50.0%       0.00 ±204%    -100.0%       0.00        perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.mutex_lock.pipe_write.new_sync_write
      0.00        +1.7e+98%       0.00 ±331%    -100.0%       0.00        perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.sched_exec.bprm_execve
      0.03 ± 66%      +5.3%       0.03 ± 72%     -53.1%       0.01 ±  7%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.wait_for_completion.affine_move_task.__set_cpus_allowed_ptr
      0.04 ±106%   +1724.8%       0.71 ±308%     -64.7%       0.01 ± 31%  perf-sched.sch_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork
      0.00          -100.0%       0.00        +7.8e+98%       0.00 ±282%  perf-sched.sch_delay.max.ms.rwsem_down_read_slowpath.down_read_killable.__access_remote_vm.proc_pid_cmdline_read
      0.00          -100.0%       0.00          +1e+99%       0.00 ±282%  perf-sched.sch_delay.max.ms.rwsem_down_read_slowpath.down_read_killable.mm_access.proc_mem_open
      0.03 ± 64%     +52.1%       0.05 ± 73%     -39.5%       0.02 ± 22%  perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      0.02 ±122%    +164.8%       0.05 ± 95%     -25.6%       0.01 ± 64%  perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
      0.02 ± 32%     +56.4%       0.04 ± 60%     -16.3%       0.02 ± 29%  perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
      0.03 ± 62%     -20.1%       0.02 ± 32%     -46.2%       0.02 ± 37%  perf-sched.sch_delay.max.ms.schedule_timeout.__skb_wait_for_more_packets.unix_dgram_recvmsg.__sys_recvfrom
      0.02 ± 23%     +25.0%       0.02 ± 32%     -12.7%       0.02 ± 40%  perf-sched.sch_delay.max.ms.schedule_timeout.io_schedule_timeout.wait_for_completion_io_timeout.blk_execute_rq
      0.02 ± 46%      +9.2%       0.02 ± 35%     -27.5%       0.02 ± 23%  perf-sched.sch_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.00        +7.2e+99%       0.01 ± 85%  +3.7e+99%       0.00 ±145%  perf-sched.sch_delay.max.ms.schedule_timeout.khugepaged.kthread.ret_from_fork
      1.37 ±190%     -93.8%       0.09 ± 62%     -58.2%       0.57 ±273%  perf-sched.sch_delay.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      0.00 ±208%    +228.1%       0.01 ± 86%     -37.5%       0.00 ±144%  perf-sched.sch_delay.max.ms.schedule_timeout.transaction_kthread.kthread.ret_from_fork
      0.01 ± 86%     +66.9%       0.02 ± 63%      -9.8%       0.01 ± 37%  perf-sched.sch_delay.max.ms.schedule_timeout.wait_for_completion.__flush_work.lru_add_drain_all
      0.01 ± 28%     +11.9%       0.01 ± 18%     -13.4%       0.01 ± 22%  perf-sched.sch_delay.max.ms.schedule_timeout.wait_for_completion_interruptible.usb_stor_control_thread.kthread
      0.02 ± 43%     +39.0%       0.03 ± 76%     -34.3%       0.01 ± 14%  perf-sched.sch_delay.max.ms.schedule_timeout.wait_for_completion_interruptible_timeout.usb_stor_msg_common.usb_stor_bulk_transfer_buf
      1.26 ±274%     -97.3%       0.03 ± 21%     -97.6%       0.03 ± 10%  perf-sched.sch_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
      0.01 ± 46%     +54.7%       0.02 ± 88%     -51.0%       0.01 ± 48%  perf-sched.sch_delay.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
      3.19            +0.2%       3.19            +0.6%       3.21        perf-sched.sch_delay.max.ms.worker_thread.kthread.ret_from_fork
      0.01 ± 44%    +226.9%       0.02 ±215%     -32.1%       0.00 ± 11%  perf-sched.total_sch_delay.average.ms
      4.66 ± 57%   +1759.5%      86.69 ±317%     -27.0%       3.40 ± 16%  perf-sched.total_sch_delay.max.ms
    204.07 ±  6%      +0.8%     205.63 ±  8%      -1.6%     200.85 ± 10%  perf-sched.total_wait_and_delay.average.ms
      7199 ±  7%      -0.4%       7167 ±  9%      +2.2%       7355 ± 11%  perf-sched.total_wait_and_delay.count.ms
      9988            +0.0%       9992            -0.4%       9945        perf-sched.total_wait_and_delay.max.ms
    204.07 ±  6%      +0.8%     205.61 ±  8%      -1.6%     200.85 ± 10%  perf-sched.total_wait_time.average.ms
      9988            +0.0%       9992            -0.4%       9945        perf-sched.total_wait_time.max.ms
    899.74            -0.0%     899.72            -0.0%     899.71        perf-sched.wait_and_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
    682.12 ±  4%      -0.3%     680.00 ±  3%      -0.8%     676.62 ±  4%  perf-sched.wait_and_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
    272.57 ±  3%      +1.5%     276.71 ±  3%      +1.2%     275.98 ±  3%  perf-sched.wait_and_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      4.63 ±  3%      +1.1%       4.68 ±  3%      -2.7%       4.50        perf-sched.wait_and_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
     62.06            -0.3%      61.89            -1.2%      61.34        perf-sched.wait_and_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
     17.16 ± 17%      +5.3%      18.07            -7.1%      15.94 ± 24%  perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.wait_for_completion.affine_move_task.__set_cpus_allowed_ptr
      3.50 ± 36%      +6.8%       3.74 ±  8%      -0.4%       3.49 ± 36%  perf-sched.wait_and_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
      0.00          -100.0%       0.00       +6.3e+104%     634.38 ±282%  perf-sched.wait_and_delay.avg.ms.rwsem_down_read_slowpath.down_read_killable.__access_remote_vm.proc_pid_cmdline_read
    307.19 ± 57%     -18.0%     251.87 ± 73%     -11.1%     272.95 ± 54%  perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
    950.15            -0.2%     948.36            -0.3%     947.48        perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
    431.49 ± 29%      -8.9%     393.21            +8.0%     466.20 ± 30%  perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
      1428 ± 20%     -14.7%       1218 ± 42%      -2.9%       1386 ± 21%  perf-sched.wait_and_delay.avg.ms.schedule_timeout.io_schedule_timeout.wait_for_completion_io_timeout.blk_execute_rq
    478.66            -0.1%     478.15            +0.0%     478.80        perf-sched.wait_and_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
     12.17 ± 43%     +10.5%      13.44 ± 38%      +6.9%      13.01 ± 46%  perf-sched.wait_and_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
    829.27 ±  3%      -1.2%     819.18            -1.2%     819.16        perf-sched.wait_and_delay.avg.ms.schedule_timeout.wait_for_completion_interruptible_timeout.usb_stor_msg_common.usb_stor_bulk_transfer_buf
    747.87            +0.4%     751.10            -1.3%     737.79        perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
      0.00 ± 19%      +1.1%       0.00 ± 19%     -17.4%       0.00 ± 14%  perf-sched.wait_and_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
    368.31 ±  4%      -1.0%     364.47 ±  4%      +1.6%     374.25 ±  3%  perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork
     10.00            +0.0%      10.00            +0.0%      10.00        perf-sched.wait_and_delay.count.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
     13.22 ±  4%      +0.2%      13.25 ±  3%      +0.8%      13.33 ±  5%  perf-sched.wait_and_delay.count.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
    254.56            -0.4%     253.42            +0.1%     254.89        perf-sched.wait_and_delay.count.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
    246.44 ±  3%      -0.8%     244.58 ±  2%      +2.6%     252.89        perf-sched.wait_and_delay.count.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      2501            +0.2%       2507            +1.5%       2538        perf-sched.wait_and_delay.count.pipe_read.new_sync_read.vfs_read.ksys_read
      1045 ±  2%      +0.2%       1048 ±  2%      +1.0%       1056        perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.wait_for_completion.affine_move_task.__set_cpus_allowed_ptr
    156.89 ± 69%     +56.4%     245.33 ± 84%     +30.5%     204.78 ± 88%  perf-sched.wait_and_delay.count.rcu_gp_kthread.kthread.ret_from_fork
      0.00          -100.0%       0.00       +1.1e+101%       0.11 ±282%  perf-sched.wait_and_delay.count.rwsem_down_read_slowpath.down_read_killable.__access_remote_vm.proc_pid_cmdline_read
     20.00 ± 67%     -22.5%      15.50 ± 86%      -2.2%      19.56 ± 64%  perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      6.00            +0.0%       6.00            +0.0%       6.00        perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
     23.44 ±  5%      -2.3%      22.92            +0.9%      23.67 ±  6%  perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
     10.00            +0.0%      10.00            +0.0%      10.00        perf-sched.wait_and_delay.count.schedule_timeout.io_schedule_timeout.wait_for_completion_io_timeout.blk_execute_rq
     39.78            -2.0%      39.00 ±  2%      +0.6%      40.00        perf-sched.wait_and_delay.count.schedule_timeout.kcompactd.kthread.ret_from_fork
    986.56 ± 57%     -12.7%     861.25 ± 59%      +4.4%       1030 ± 72%  perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      9.89 ±  3%      +1.1%      10.00            +1.1%      10.00        perf-sched.wait_and_delay.count.schedule_timeout.wait_for_completion_interruptible_timeout.usb_stor_msg_common.usb_stor_bulk_transfer_buf
      1241 ±  2%      +0.2%       1244 ±  2%      +0.8%       1251        perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork
     71.56            -0.2%      71.42            +0.2%      71.67        perf-sched.wait_and_delay.count.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
    533.44 ±  3%      +0.1%     534.17 ±  2%      -0.1%     532.78 ±  2%  perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork
    999.83            +0.0%     999.84            -0.0%     999.81        perf-sched.wait_and_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      1000            +0.0%       1000            -0.0%       1000        perf-sched.wait_and_delay.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      2750 ± 55%      +7.2%       2948 ± 60%     +24.4%       3420 ± 51%  perf-sched.wait_and_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      1000            +0.0%       1000            -0.0%       1000        perf-sched.wait_and_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      3004            +0.0%       3004            -0.0%       3003        perf-sched.wait_and_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      8994 ± 31%     +11.1%       9992           -11.1%       7995 ± 46%  perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.wait_for_completion.affine_move_task.__set_cpus_allowed_ptr
      4.49 ± 35%     +18.2%       5.31 ± 15%      -0.9%       4.45 ± 35%  perf-sched.wait_and_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork
      0.00          -100.0%       0.00       +6.3e+104%     634.38 ±282%  perf-sched.wait_and_delay.max.ms.rwsem_down_read_slowpath.down_read_killable.__access_remote_vm.proc_pid_cmdline_read
      2091 ±124%     -33.3%       1395 ±139%     -37.6%       1305 ±133%  perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      5700            -0.2%       5690            -0.3%       5684        perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
      1994 ±140%     -49.8%       1001           +47.9%       2950 ±123%  perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
      2047            -5.5%       1935 ± 14%      +0.0%       2047        perf-sched.wait_and_delay.max.ms.schedule_timeout.io_schedule_timeout.wait_for_completion_io_timeout.blk_execute_rq
    505.02            +0.0%     505.03            -0.0%     505.01        perf-sched.wait_and_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
    304.90 ± 44%      -9.6%     275.59 ± 51%      +0.0%     305.00 ± 52%  perf-sched.wait_and_delay.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      2048            +0.0%       2048            +0.0%       2048        perf-sched.wait_and_delay.max.ms.schedule_timeout.wait_for_completion_interruptible_timeout.usb_stor_msg_common.usb_stor_bulk_transfer_buf
      1585 ± 32%      +1.9%       1615 ± 31%      -7.3%       1469 ± 35%  perf-sched.wait_and_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
      0.01 ± 46%     +56.2%       0.02 ± 86%     -51.0%       0.01 ± 48%  perf-sched.wait_and_delay.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
      5388 ± 31%     -16.1%       4522 ± 36%     -11.8%       4751 ± 29%  perf-sched.wait_and_delay.max.ms.worker_thread.kthread.ret_from_fork
    899.73            -0.0%     899.71            -0.0%     899.70        perf-sched.wait_time.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      1.57 ± 10%      +6.1%       1.67 ±  8%      +1.6%       1.60 ± 11%  perf-sched.wait_time.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
    682.11 ±  4%      -0.3%     679.99 ±  3%      -0.8%     676.62 ±  4%  perf-sched.wait_time.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      1.59 ± 10%      +5.7%       1.68 ±  8%      +1.0%       1.60 ± 11%  perf-sched.wait_time.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
    272.57 ±  3%      +1.4%     276.37 ±  3%      +1.2%     275.97 ±  3%  perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      4.62 ±  3%      +1.1%       4.67 ±  3%      -2.6%       4.50        perf-sched.wait_time.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.01 ±106%     -60.5%       0.00 ±223%     -21.1%       0.01 ±113%  perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      0.01 ± 21%     +11.4%       0.01 ± 28%     +12.9%       0.01 ± 23%  perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.04 ±282%    -100.0%       0.00           -14.8%       0.03 ±282%  perf-sched.wait_time.avg.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
     62.06            -0.3%      61.88            -1.2%      61.33        perf-sched.wait_time.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      0.01 ±149%      +5.9%       0.01 ±133%     -39.7%       0.00 ±174%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.copy_page_from_iter.pipe_write.new_sync_write
      0.01 ± 87%     -56.4%       0.01 ±160%     -60.9%       0.00 ±179%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.copy_page_to_iter.pipe_read.new_sync_read
      0.01 ± 70%     -27.3%       0.01 ± 73%     -60.8%       0.01 ±141%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.mutex_lock.pipe_read.new_sync_read
      0.02 ± 61%     -16.4%       0.01 ± 88%     +21.6%       0.02 ± 40%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.mutex_lock.pipe_write.new_sync_write
     17.15 ± 17%      +5.3%      18.07            -7.1%      15.94 ± 24%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.wait_for_completion.affine_move_task.__set_cpus_allowed_ptr
      3.49 ± 36%      +6.7%       3.72 ±  8%     +10.8%       3.86 ±  7%  perf-sched.wait_time.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
      0.00          -100.0%       0.00       +6.3e+104%     634.38 ±282%  perf-sched.wait_time.avg.ms.rwsem_down_read_slowpath.down_read_killable.__access_remote_vm.proc_pid_cmdline_read
      0.00          -100.0%       0.00       +1.2e+100%       0.01 ±282%  perf-sched.wait_time.avg.ms.rwsem_down_read_slowpath.down_read_killable.mm_access.proc_mem_open
    314.20 ± 52%     -15.5%     265.53 ± 63%      -9.8%     283.48 ± 45%  perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
    950.14            -0.2%     948.34            -0.3%     947.47        perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
    431.49 ± 29%      -8.9%     393.20            +8.0%     466.20 ± 30%  perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
      5.50 ± 61%     -23.1%       4.23 ±107%      +4.9%       5.77 ± 96%  perf-sched.wait_time.avg.ms.schedule_timeout.__skb_wait_for_more_packets.unix_dgram_recvmsg.__sys_recvfrom
      1428 ± 20%     -14.7%       1218 ± 42%      -2.9%       1386 ± 21%  perf-sched.wait_time.avg.ms.schedule_timeout.io_schedule_timeout.wait_for_completion_io_timeout.blk_execute_rq
    478.65            -0.1%     478.14            +0.0%     478.80        perf-sched.wait_time.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.01 ± 33%     +20.3%       0.01 ± 63%      +6.3%       0.01 ± 60%  perf-sched.wait_time.avg.ms.schedule_timeout.khugepaged.kthread.ret_from_fork
     12.15 ± 43%     +10.5%      13.43 ± 38%      +7.0%      13.00 ± 46%  perf-sched.wait_time.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      0.00 ±188%     +41.7%       0.00 ±192%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_timeout.wait_for_completion.__flush_work.lru_add_drain_all
      0.00 ±191%     +88.6%       0.01 ±141%     -31.4%       0.00 ±206%  perf-sched.wait_time.avg.ms.schedule_timeout.wait_for_completion.affine_move_task.__set_cpus_allowed_ptr
      0.08 ± 18%      +2.2%       0.09 ±  7%      +7.5%       0.09 ±  2%  perf-sched.wait_time.avg.ms.schedule_timeout.wait_for_completion_interruptible.usb_stor_control_thread.kthread
    829.26 ±  3%      -1.2%     819.17            -1.2%     819.15        perf-sched.wait_time.avg.ms.schedule_timeout.wait_for_completion_interruptible_timeout.usb_stor_msg_common.usb_stor_bulk_transfer_buf
    747.87            +0.4%     751.09            -1.3%     737.78        perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
    368.29 ±  4%      -1.0%     364.46 ±  4%      +1.6%     374.24 ±  3%  perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork
    999.81            +0.0%     999.82            -0.0%     999.80        perf-sched.wait_time.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      3.14 ± 10%      +6.1%       3.33 ±  8%      +1.6%       3.19 ± 11%  perf-sched.wait_time.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      1000            +0.0%       1000            -0.0%       1000        perf-sched.wait_time.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      3.17 ± 10%      +5.7%       3.35 ±  8%      +1.0%       3.21 ± 11%  perf-sched.wait_time.max.ms.do_syslog.part.0.kmsg_read.vfs_read
      2750 ± 55%      +7.2%       2948 ± 60%     +24.4%       3420 ± 51%  perf-sched.wait_time.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      1000            +0.0%       1000            -0.0%       1000        perf-sched.wait_time.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.01 ± 98%     -62.5%       0.00 ±201%     -25.0%       0.01 ±112%  perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      0.03 ± 16%      -2.1%       0.03 ± 12%      -8.7%       0.03 ±  3%  perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.13 ±282%    -100.0%       0.00           -25.1%       0.10 ±282%  perf-sched.wait_time.max.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
      3004            +0.0%       3004            -0.0%       3003        perf-sched.wait_time.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      0.01 ±141%      -0.0%       0.01 ±129%     -49.4%       0.00 ±174%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.copy_page_from_iter.pipe_write.new_sync_write
      0.01 ± 84%     -44.1%       0.01 ±160%     -57.4%       0.01 ±178%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.copy_page_to_iter.pipe_read.new_sync_read
      0.02 ± 63%     -11.2%       0.02 ± 72%     -49.1%       0.01 ±122%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.mutex_lock.pipe_read.new_sync_read
      0.02 ± 54%     -30.1%       0.01 ± 83%      +8.4%       0.02 ± 37%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.mutex_lock.pipe_write.new_sync_write
      8994 ± 31%     +11.1%       9992           -11.1%       7995 ± 46%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.wait_for_completion.affine_move_task.__set_cpus_allowed_ptr
      4.48 ± 35%     +12.6%       5.04            +9.2%       4.89 ±  6%  perf-sched.wait_time.max.ms.rcu_gp_kthread.kthread.ret_from_fork
      0.00          -100.0%       0.00       +6.3e+104%     634.38 ±282%  perf-sched.wait_time.max.ms.rwsem_down_read_slowpath.down_read_killable.__access_remote_vm.proc_pid_cmdline_read
      0.00          -100.0%       0.00       +1.2e+100%       0.01 ±282%  perf-sched.wait_time.max.ms.rwsem_down_read_slowpath.down_read_killable.mm_access.proc_mem_open
      2156 ±118%     -29.2%       1525 ±121%     -34.9%       1403 ±119%  perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      5700            -0.2%       5690            -0.3%       5684        perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
      1994 ±140%     -49.8%       1001           +47.9%       2950 ±123%  perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
      5.50 ± 61%     -23.1%       4.23 ±107%      +4.9%       5.77 ± 96%  perf-sched.wait_time.max.ms.schedule_timeout.__skb_wait_for_more_packets.unix_dgram_recvmsg.__sys_recvfrom
      2047            -5.5%       1935 ± 14%      +0.0%       2047        perf-sched.wait_time.max.ms.schedule_timeout.io_schedule_timeout.wait_for_completion_io_timeout.blk_execute_rq
    505.01            +0.0%     505.01            -0.0%     505.00        perf-sched.wait_time.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.01 ± 33%     +20.3%       0.01 ± 63%      +6.3%       0.01 ± 60%  perf-sched.wait_time.max.ms.schedule_timeout.khugepaged.kthread.ret_from_fork
    304.86 ± 44%      -9.6%     275.56 ± 51%      +0.0%     304.99 ± 52%  perf-sched.wait_time.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      0.00 ±204%     +32.7%       0.00 ±186%    -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_timeout.wait_for_completion.__flush_work.lru_add_drain_all
      0.00 ±191%     +88.6%       0.01 ±141%     -31.4%       0.00 ±206%  perf-sched.wait_time.max.ms.schedule_timeout.wait_for_completion.affine_move_task.__set_cpus_allowed_ptr
      0.10 ±  6%      -3.1%       0.09 ±  5%      -1.4%       0.10 ±  4%  perf-sched.wait_time.max.ms.schedule_timeout.wait_for_completion_interruptible.usb_stor_control_thread.kthread
      2048            +0.0%       2048            +0.0%       2048        perf-sched.wait_time.max.ms.schedule_timeout.wait_for_completion_interruptible_timeout.usb_stor_msg_common.usb_stor_bulk_transfer_buf
      1585 ± 32%      +1.9%       1615 ± 31%      -7.3%       1469 ± 35%  perf-sched.wait_time.max.ms.smpboot_thread_fn.kthread.ret_from_fork
      0.00        +4.2e+98%       0.00 ±331%    -100.0%       0.00        perf-sched.wait_time.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
      5388 ± 31%     -16.1%       4522 ± 36%     -11.8%       4751 ± 29%  perf-sched.wait_time.max.ms.worker_thread.kthread.ret_from_fork
    190513            -0.0%     190442            -0.0%     190445        slabinfo.Acpi-Operand.active_objs
      3401            -0.0%       3400            -0.0%       3400        slabinfo.Acpi-Operand.active_slabs
    190523            -0.0%     190457            -0.0%     190457        slabinfo.Acpi-Operand.num_objs
      3401            -0.0%       3400            -0.0%       3400        slabinfo.Acpi-Operand.num_slabs
      1403 ± 11%      -0.3%       1399 ±  8%      -7.5%       1297 ±  8%  slabinfo.Acpi-Parse.active_objs
     19.22 ± 11%      -0.3%      19.17 ±  8%      -7.5%      17.78 ±  8%  slabinfo.Acpi-Parse.active_slabs
      1403 ± 11%      -0.3%       1399 ±  8%      -7.5%       1297 ±  8%  slabinfo.Acpi-Parse.num_objs
     19.22 ± 11%      -0.3%      19.17 ±  8%      -7.5%      17.78 ±  8%  slabinfo.Acpi-Parse.num_slabs
      4840 ±  4%      -3.2%       4686 ±  3%      +0.1%       4846 ±  3%  slabinfo.Acpi-State.active_objs
     94.56 ±  4%      -3.6%      91.17 ±  3%      -0.1%      94.44 ±  2%  slabinfo.Acpi-State.active_slabs
      4840 ±  4%      -3.2%       4686 ±  3%      +0.1%       4846 ±  3%  slabinfo.Acpi-State.num_objs
     94.56 ±  4%      -3.6%      91.17 ±  3%      -0.1%      94.44 ±  2%  slabinfo.Acpi-State.num_slabs
    202.67 ±  7%      -3.9%     194.67 ±  4%      -5.3%     192.00        slabinfo.RAW.active_objs
      6.33 ±  7%      -3.9%       6.08 ±  4%      -5.3%       6.00        slabinfo.RAW.active_slabs
    202.67 ±  7%      -3.9%     194.67 ±  4%      -5.3%     192.00        slabinfo.RAW.num_objs
      6.33 ±  7%      -3.9%       6.08 ±  4%      -5.3%       6.00        slabinfo.RAW.num_slabs
    104.00            +0.0%     104.00            +0.0%     104.00        slabinfo.RAWv6.active_objs
      4.00            +0.0%       4.00            +0.0%       4.00        slabinfo.RAWv6.active_slabs
    104.00            +0.0%     104.00            +0.0%     104.00        slabinfo.RAWv6.num_objs
      4.00            +0.0%       4.00            +0.0%       4.00        slabinfo.RAWv6.num_slabs
     56.00            -4.2%      53.67 ±  9%      +0.0%      56.00        slabinfo.TCP.active_objs
      4.00            -4.2%       3.83 ±  9%      +0.0%       4.00        slabinfo.TCP.active_slabs
     56.00            -4.2%      53.67 ±  9%      +0.0%      56.00        slabinfo.TCP.num_objs
      4.00            -4.2%       3.83 ±  9%      +0.0%       4.00        slabinfo.TCP.num_slabs
     39.00            -2.8%      37.92 ±  9%      +0.0%      39.00        slabinfo.TCPv6.active_objs
      3.00            -2.8%       2.92 ±  9%      +0.0%       3.00        slabinfo.TCPv6.active_slabs
     39.00            -2.8%      37.92 ±  9%      +0.0%      39.00        slabinfo.TCPv6.num_objs
      3.00            -2.8%       2.92 ±  9%      +0.0%       3.00        slabinfo.TCPv6.num_slabs
    162.11 ±  8%      +1.7%     164.83 ±  8%      -1.8%     159.22 ± 10%  slabinfo.UDPv6.active_objs
      6.22 ± 10%      +1.8%       6.33 ±  9%      -1.8%       6.11 ± 12%  slabinfo.UDPv6.active_slabs
    162.11 ±  8%      +1.7%     164.83 ±  8%      -1.8%     159.22 ± 10%  slabinfo.UDPv6.num_objs
      6.22 ± 10%      +1.8%       6.33 ±  9%      -1.8%       6.11 ± 12%  slabinfo.UDPv6.num_slabs
      1838 ± 15%      -1.3%       1815 ± 16%      +6.4%       1955 ± 12%  slabinfo.UNIX.active_objs
     60.89 ± 15%      -1.5%      60.00 ± 16%      +6.2%      64.67 ± 12%  slabinfo.UNIX.active_slabs
      1838 ± 15%      -1.3%       1815 ± 16%      +6.4%       1955 ± 12%  slabinfo.UNIX.num_objs
     60.89 ± 15%      -1.5%      60.00 ± 16%      +6.2%      64.67 ± 12%  slabinfo.UNIX.num_slabs
     25542 ±  4%      +2.5%      26173 ±  3%      +2.5%      26173 ±  4%  slabinfo.anon_vma.active_objs
    554.67 ±  4%      +2.5%     568.58 ±  3%      +2.5%     568.44 ±  4%  slabinfo.anon_vma.active_slabs
     25542 ±  4%      +2.5%      26190 ±  3%      +2.5%      26173 ±  4%  slabinfo.anon_vma.num_objs
    554.67 ±  4%      +2.5%     568.58 ±  3%      +2.5%     568.44 ±  4%  slabinfo.anon_vma.num_slabs
     60511 ± 10%      -1.1%      59846 ±  7%      +2.7%      62150 ±  7%  slabinfo.anon_vma_chain.active_objs
    947.78 ± 10%      -1.0%     938.00 ±  7%      +2.7%     973.67 ±  7%  slabinfo.anon_vma_chain.active_slabs
     60690 ± 10%      -1.0%      60066 ±  7%      +2.7%      62344 ±  7%  slabinfo.anon_vma_chain.num_objs
    947.78 ± 10%      -1.0%     938.00 ±  7%      +2.7%     973.67 ±  7%  slabinfo.anon_vma_chain.num_slabs
    231.00 ± 16%      -2.3%     225.75 ± 13%      -2.0%     226.33 ± 16%  slabinfo.bdev_cache.active_objs
     11.00 ± 16%      -2.3%      10.75 ± 13%      -2.0%      10.78 ± 16%  slabinfo.bdev_cache.active_slabs
    231.00 ± 16%      -2.3%     225.75 ± 13%      -2.0%     226.33 ± 16%  slabinfo.bdev_cache.num_objs
     11.00 ± 16%      -2.3%      10.75 ± 13%      -2.0%      10.78 ± 16%  slabinfo.bdev_cache.num_slabs
    105.33 ± 27%     -16.5%      88.00 ± 30%     -14.6%      90.00 ± 28%  slabinfo.biovec-128.active_objs
      6.22 ± 24%     -11.6%       5.50 ± 30%     -10.7%       5.56 ± 26%  slabinfo.biovec-128.active_slabs
    105.33 ± 27%     -16.5%      88.00 ± 30%     -14.6%      90.00 ± 28%  slabinfo.biovec-128.num_objs
      6.22 ± 24%     -11.6%       5.50 ± 30%     -10.7%       5.56 ± 26%  slabinfo.biovec-128.num_slabs
    304.89 ± 12%      +1.5%     309.33 ± 15%      +8.1%     329.67 ± 10%  slabinfo.biovec-64.active_objs
      9.33 ± 11%      +3.6%       9.67 ± 15%      +9.5%      10.22 ± 10%  slabinfo.biovec-64.active_slabs
    304.89 ± 12%      +1.5%     309.33 ± 15%      +8.1%     329.67 ± 10%  slabinfo.biovec-64.num_objs
      9.33 ± 11%      +3.6%       9.67 ± 15%      +9.5%      10.22 ± 10%  slabinfo.biovec-64.num_slabs
     97.78 ± 17%     -19.5%      78.67 ±  5%     -17.8%      80.33 ± 15%  slabinfo.biovec-max.active_objs
     12.00 ± 16%     -18.1%       9.83 ±  5%     -16.7%      10.00 ± 14%  slabinfo.biovec-max.active_slabs
     97.78 ± 17%     -19.5%      78.67 ±  5%     -17.8%      80.33 ± 15%  slabinfo.biovec-max.num_objs
     12.00 ± 16%     -18.1%       9.83 ±  5%     -16.7%      10.00 ± 14%  slabinfo.biovec-max.num_slabs
     49.89 ± 92%    -100.0%       0.00           -77.1%      11.44 ±282%  slabinfo.btrfs_delayed_tree_ref.active_objs
      0.89 ± 98%    -100.0%       0.00           -75.0%       0.22 ±282%  slabinfo.btrfs_delayed_tree_ref.active_slabs
     49.89 ± 92%    -100.0%       0.00           -77.1%      11.44 ±282%  slabinfo.btrfs_delayed_tree_ref.num_objs
      0.89 ± 98%    -100.0%       0.00           -75.0%       0.22 ±282%  slabinfo.btrfs_delayed_tree_ref.num_slabs
    167.44 ± 59%     -66.6%      56.00           -53.3%      78.22 ± 80%  slabinfo.btrfs_extent_map.active_objs
      2.67 ± 55%     -62.5%       1.00           -50.0%       1.33 ± 70%  slabinfo.btrfs_extent_map.active_slabs
    167.44 ± 59%     -66.6%      56.00           -53.3%      78.22 ± 80%  slabinfo.btrfs_extent_map.num_objs
      2.67 ± 55%     -62.5%       1.00           -50.0%       1.33 ± 70%  slabinfo.btrfs_extent_map.num_slabs
    145.89 ± 21%     -23.2%     112.00           -23.6%     111.44 ± 19%  slabinfo.btrfs_inode.active_objs
      4.78 ± 16%     -16.3%       4.00           -18.6%       3.89 ± 14%  slabinfo.btrfs_inode.active_slabs
    145.89 ± 21%     -23.2%     112.00           -23.6%     111.44 ± 19%  slabinfo.btrfs_inode.num_objs
      4.78 ± 16%     -16.3%       4.00           -18.6%       3.89 ± 14%  slabinfo.btrfs_inode.num_slabs
     29.22 ± 97%    -100.0%       0.00           -75.3%       7.22 ±282%  slabinfo.btrfs_ordered_extent.active_objs
      0.33 ±141%    -100.0%       0.00           -66.7%       0.11 ±282%  slabinfo.btrfs_ordered_extent.active_slabs
     29.22 ± 97%    -100.0%       0.00           -75.3%       7.22 ±282%  slabinfo.btrfs_ordered_extent.num_objs
      0.33 ±141%    -100.0%       0.00           -66.7%       0.11 ±282%  slabinfo.btrfs_ordered_extent.num_slabs
    517.22 ± 13%     -25.9%     383.33 ± 14%      -3.7%     498.22 ± 13%  slabinfo.buffer_head.active_objs
     12.78 ± 14%     -26.3%       9.42 ± 15%      -4.3%      12.22 ± 15%  slabinfo.buffer_head.active_slabs
    517.22 ± 13%     -25.9%     383.33 ± 14%      -3.7%     498.22 ± 13%  slabinfo.buffer_head.num_objs
     12.78 ± 14%     -26.3%       9.42 ± 15%      -4.3%      12.22 ± 15%  slabinfo.buffer_head.num_slabs
      6567 ±  3%      +2.9%       6761 ±  5%      -0.7%       6519 ±  3%  slabinfo.cred_jar.active_objs
    155.44 ±  3%      +3.0%     160.08 ±  5%      -0.6%     154.56 ±  3%  slabinfo.cred_jar.active_slabs
      6567 ±  3%      +2.9%       6761 ±  5%      -0.7%       6519 ±  3%  slabinfo.cred_jar.num_objs
    155.44 ±  3%      +3.0%     160.08 ±  5%      -0.6%     154.56 ±  3%  slabinfo.cred_jar.num_slabs
     42.00            +0.0%      42.00            +0.0%      42.00        slabinfo.dax_cache.active_objs
      1.00            +0.0%       1.00            +0.0%       1.00        slabinfo.dax_cache.active_slabs
     42.00            +0.0%      42.00            +0.0%      42.00        slabinfo.dax_cache.num_objs
      1.00            +0.0%       1.00            +0.0%       1.00        slabinfo.dax_cache.num_slabs
    129308 ±  2%      -0.5%     128608 ±  2%      +0.5%     129967 ±  3%  slabinfo.dentry.active_objs
      3115 ±  2%      -0.5%       3098 ±  3%      +0.7%       3137 ±  2%  slabinfo.dentry.active_slabs
    130891 ±  2%      -0.5%     130174 ±  3%      +0.7%     131776 ±  2%  slabinfo.dentry.num_objs
      3115 ±  2%      -0.5%       3098 ±  3%      +0.7%       3137 ±  2%  slabinfo.dentry.num_slabs
     32.00            +0.0%      32.00            +0.0%      32.00        slabinfo.dma-kmalloc-512.active_objs
      1.00            +0.0%       1.00            +0.0%       1.00        slabinfo.dma-kmalloc-512.active_slabs
     32.00            +0.0%      32.00            +0.0%      32.00        slabinfo.dma-kmalloc-512.num_objs
      1.00            +0.0%       1.00            +0.0%       1.00        slabinfo.dma-kmalloc-512.num_slabs
     30.00            +0.0%      30.00            +0.0%      30.00        slabinfo.dmaengine-unmap-128.active_objs
      1.00            +0.0%       1.00            +0.0%       1.00        slabinfo.dmaengine-unmap-128.active_slabs
     30.00            +0.0%      30.00            +0.0%      30.00        slabinfo.dmaengine-unmap-128.num_objs
      1.00            +0.0%       1.00            +0.0%       1.00        slabinfo.dmaengine-unmap-128.num_slabs
      1784 ± 11%     +12.8%       2012 ± 11%      -6.7%       1664 ± 14%  slabinfo.dmaengine-unmap-16.active_objs
     42.11 ± 11%     +12.6%      47.42 ± 12%      -6.9%      39.22 ± 14%  slabinfo.dmaengine-unmap-16.active_slabs
      1784 ± 11%     +12.8%       2012 ± 11%      -6.7%       1664 ± 14%  slabinfo.dmaengine-unmap-16.num_objs
     42.11 ± 11%     +12.6%      47.42 ± 12%      -6.9%      39.22 ± 14%  slabinfo.dmaengine-unmap-16.num_slabs
     15.00            +0.0%      15.00            +0.0%      15.00        slabinfo.dmaengine-unmap-256.active_objs
      1.00            +0.0%       1.00            +0.0%       1.00        slabinfo.dmaengine-unmap-256.active_slabs
     15.00            +0.0%      15.00            +0.0%      15.00        slabinfo.dmaengine-unmap-256.num_objs
      1.00            +0.0%       1.00            +0.0%       1.00        slabinfo.dmaengine-unmap-256.num_slabs
     20864 ± 11%      +1.5%      21184 ±  9%      -2.5%      20343 ± 16%  slabinfo.ep_head.active_objs
     81.33 ± 11%      +1.6%      82.67 ±  9%      -2.3%      79.44 ± 16%  slabinfo.ep_head.active_slabs
     20864 ± 11%      +1.5%      21184 ±  9%      -2.5%      20343 ± 16%  slabinfo.ep_head.num_objs
     81.33 ± 11%      +1.6%      82.67 ±  9%      -2.3%      79.44 ± 16%  slabinfo.ep_head.num_slabs
      1070 ±  8%      +6.5%       1141 ±  6%      +3.0%       1103 ±  8%  slabinfo.file_lock_cache.active_objs
     28.44 ±  8%      +6.6%      30.33 ±  6%      +3.1%      29.33 ±  9%  slabinfo.file_lock_cache.active_slabs
      1070 ±  8%      +6.5%       1141 ±  6%      +3.0%       1103 ±  8%  slabinfo.file_lock_cache.num_objs
     28.44 ±  8%      +6.6%      30.33 ±  6%      +3.1%      29.33 ±  9%  slabinfo.file_lock_cache.num_slabs
      4040 ±  5%      -1.8%       3965 ±  5%      -0.7%       4011 ±  4%  slabinfo.files_cache.active_objs
     87.11 ±  5%      -1.8%      85.50 ±  6%      -0.5%      86.67 ±  4%  slabinfo.files_cache.active_slabs
      4040 ±  5%      -1.8%       3965 ±  5%      -0.7%       4011 ±  4%  slabinfo.files_cache.num_objs
     87.11 ±  5%      -1.8%      85.50 ±  6%      -0.5%      86.67 ±  4%  slabinfo.files_cache.num_slabs
     27299 ±  3%      +0.8%      27509 ±  5%      +1.2%      27628 ±  5%  slabinfo.filp.active_objs
    874.11 ±  3%      +1.6%     888.33 ±  6%      +1.4%     886.11 ±  4%  slabinfo.filp.active_slabs
     27988 ±  3%      +1.6%      28442 ±  6%      +1.4%      28372 ±  4%  slabinfo.filp.num_objs
    874.11 ±  3%      +1.6%     888.33 ±  6%      +1.4%     886.11 ±  4%  slabinfo.filp.num_slabs
      3029 ± 14%      -1.3%       2990 ± 13%      -4.1%       2904 ± 14%  slabinfo.fsnotify_mark_connector.active_objs
     23.33 ± 15%      -2.1%      22.83 ± 12%      -3.8%      22.44 ± 14%  slabinfo.fsnotify_mark_connector.active_slabs
      3029 ± 14%      -1.3%       2990 ± 13%      -4.1%       2904 ± 14%  slabinfo.fsnotify_mark_connector.num_objs
     23.33 ± 15%      -2.1%      22.83 ± 12%      -3.8%      22.44 ± 14%  slabinfo.fsnotify_mark_connector.num_slabs
     30779            -0.1%      30762            -0.2%      30722        slabinfo.ftrace_event_field.active_objs
    362.11            -0.1%     361.92            -0.2%     361.44        slabinfo.ftrace_event_field.active_slabs
     30779            -0.1%      30762            -0.2%      30722        slabinfo.ftrace_event_field.num_objs
    362.11            -0.1%     361.92            -0.2%     361.44        slabinfo.ftrace_event_field.num_slabs
    106.00            +0.0%     106.00            +0.0%     106.00        slabinfo.hugetlbfs_inode_cache.active_objs
      2.00            +0.0%       2.00            +0.0%       2.00        slabinfo.hugetlbfs_inode_cache.active_slabs
    106.00            +0.0%     106.00            +0.0%     106.00        slabinfo.hugetlbfs_inode_cache.num_objs
      2.00            +0.0%       2.00            +0.0%       2.00        slabinfo.hugetlbfs_inode_cache.num_slabs
     85772            +0.1%      85889            +0.1%      85851        slabinfo.inode_cache.active_objs
      1562            +0.1%       1564            +0.1%       1563        slabinfo.inode_cache.active_slabs
     85947            +0.1%      86055            +0.1%      86029        slabinfo.inode_cache.num_objs
      1562            +0.1%       1564            +0.1%       1563        slabinfo.inode_cache.num_slabs
     85009            -0.1%      84887            -0.2%      84859        slabinfo.kernfs_node_cache.active_objs
      2656            -0.1%       2652            -0.2%       2651        slabinfo.kernfs_node_cache.active_slabs
     85009            -0.1%      84887            -0.2%      84859        slabinfo.kernfs_node_cache.num_objs
      2656            -0.1%       2652            -0.2%       2651        slabinfo.kernfs_node_cache.num_slabs
      1819 ± 14%      -2.2%       1780 ±  8%      -4.6%       1735 ±  9%  slabinfo.khugepaged_mm_slot.active_objs
     49.89 ± 14%      -2.3%      48.75 ±  8%      -5.1%      47.33 ±  9%  slabinfo.khugepaged_mm_slot.active_slabs
      1819 ± 14%      -2.2%       1780 ±  8%      -4.6%       1735 ±  9%  slabinfo.khugepaged_mm_slot.num_objs
     49.89 ± 14%      -2.3%      48.75 ±  8%      -5.1%      47.33 ±  9%  slabinfo.khugepaged_mm_slot.num_slabs
      5340            -1.0%       5289            -1.0%       5289        slabinfo.kmalloc-128.active_objs
    167.56            -0.9%     166.00            -0.9%     166.00        slabinfo.kmalloc-128.active_slabs
      5368            -1.0%       5315            -0.9%       5318        slabinfo.kmalloc-128.num_objs
    167.56            -0.9%     166.00            -0.9%     166.00        slabinfo.kmalloc-128.num_slabs
     35224            -0.0%      35211            -0.4%      35094        slabinfo.kmalloc-16.active_objs
    137.78            +0.0%     137.83            -0.2%     137.44        slabinfo.kmalloc-16.active_slabs
     35271            +0.0%      35285            -0.2%      35185        slabinfo.kmalloc-16.num_objs
    137.78            +0.0%     137.83            -0.2%     137.44        slabinfo.kmalloc-16.num_slabs
      5246            -0.0%       5245            +0.0%       5248        slabinfo.kmalloc-192.active_objs
    127.78            +0.6%     128.50            +0.7%     128.67        slabinfo.kmalloc-192.active_slabs
      5375            +0.4%       5397            +0.5%       5404        slabinfo.kmalloc-192.num_objs
    127.78            +0.6%     128.50            +0.7%     128.67        slabinfo.kmalloc-192.num_slabs
      5297 ±  3%      -1.4%       5220 ±  2%      -1.6%       5212        slabinfo.kmalloc-1k.active_objs
    166.11 ±  2%      -1.3%     163.92 ±  2%      -1.4%     163.78        slabinfo.kmalloc-1k.active_slabs
      5334 ±  3%      -1.4%       5260 ±  2%      -1.6%       5250        slabinfo.kmalloc-1k.num_objs
    166.11 ±  2%      -1.3%     163.92 ±  2%      -1.4%     163.78        slabinfo.kmalloc-1k.num_slabs
      9590 ±  4%      +1.4%       9720 ±  5%      +1.1%       9697 ±  4%  slabinfo.kmalloc-256.active_objs
    299.33 ±  4%      +1.3%     303.33 ±  5%      +1.2%     302.78 ±  4%  slabinfo.kmalloc-256.active_slabs
      9590 ±  4%      +1.4%       9720 ±  5%      +1.1%       9698 ±  4%  slabinfo.kmalloc-256.num_objs
    299.33 ±  4%      +1.3%     303.33 ±  5%      +1.2%     302.78 ±  4%  slabinfo.kmalloc-256.num_slabs
      5215 ±  3%      +2.1%       5325 ±  5%      +3.3%       5387 ±  3%  slabinfo.kmalloc-2k.active_objs
    329.89 ±  3%      +2.3%     337.42 ±  5%      +3.3%     340.78 ±  3%  slabinfo.kmalloc-2k.active_slabs
      5285 ±  3%      +2.3%       5404 ±  5%      +3.3%       5461 ±  3%  slabinfo.kmalloc-2k.num_objs
    329.89 ±  3%      +2.3%     337.42 ±  5%      +3.3%     340.78 ±  3%  slabinfo.kmalloc-2k.num_slabs
     67287            +0.9%      67884            +0.1%      67360        slabinfo.kmalloc-32.active_objs
    526.44            +0.9%     531.08            +0.0%     526.67        slabinfo.kmalloc-32.active_slabs
     67456            +0.9%      68072            +0.1%      67505        slabinfo.kmalloc-32.num_objs
    526.44            +0.9%     531.08            +0.0%     526.67        slabinfo.kmalloc-32.num_slabs
      1906            -0.7%       1893            -0.5%       1897        slabinfo.kmalloc-4k.active_objs
    242.89            -0.7%     241.25            -0.5%     241.67        slabinfo.kmalloc-4k.active_slabs
      1947            -0.6%       1935            -0.4%       1938        slabinfo.kmalloc-4k.num_objs
    242.89            -0.7%     241.25            -0.5%     241.67        slabinfo.kmalloc-4k.num_slabs
     15912 ±  4%      -0.5%      15837 ±  3%      -1.2%      15721 ±  2%  slabinfo.kmalloc-512.active_objs
    499.22 ±  4%      -0.5%     496.92 ±  3%      -1.1%     493.67 ±  2%  slabinfo.kmalloc-512.active_slabs
     15993 ±  4%      -0.5%      15914 ±  3%      -1.1%      15813 ±  2%  slabinfo.kmalloc-512.num_objs
    499.22 ±  4%      -0.5%     496.92 ±  3%      -1.1%     493.67 ±  2%  slabinfo.kmalloc-512.num_slabs
     50201            -0.5%      49952            -0.3%      50039        slabinfo.kmalloc-64.active_objs
    787.33            -0.5%     783.25            -0.4%     784.33        slabinfo.kmalloc-64.active_slabs
     50425            -0.5%      50169            -0.4%      50230        slabinfo.kmalloc-64.num_objs
    787.33            -0.5%     783.25            -0.4%     784.33        slabinfo.kmalloc-64.num_slabs
     53715            +0.2%      53832            -0.1%      53682        slabinfo.kmalloc-8.active_objs
    107.56            +0.6%     108.25 ±  2%      -0.1%     107.44 ±  2%  slabinfo.kmalloc-8.active_slabs
     55068            +0.7%      55466 ±  2%      +0.1%      55115        slabinfo.kmalloc-8.num_objs
    107.56            +0.6%     108.25 ±  2%      -0.1%     107.44 ±  2%  slabinfo.kmalloc-8.num_slabs
    779.44            -0.1%     778.75            -0.1%     778.78        slabinfo.kmalloc-8k.active_objs
    200.56            -0.1%     200.42            -0.1%     200.44        slabinfo.kmalloc-8k.active_slabs
    804.11            -0.1%     803.67            -0.1%     803.22        slabinfo.kmalloc-8k.num_objs
    200.56            -0.1%     200.42            -0.1%     200.44        slabinfo.kmalloc-8k.num_slabs
      7532 ±  3%      +3.7%       7807 ±  3%      +5.7%       7961 ±  2%  slabinfo.kmalloc-96.active_objs
    180.78 ±  3%      +3.7%     187.50 ±  3%      +5.8%     191.22 ±  3%  slabinfo.kmalloc-96.active_slabs
      7612 ±  3%      +3.7%       7896 ±  3%      +5.7%       8048 ±  2%  slabinfo.kmalloc-96.num_objs
    180.78 ±  3%      +3.7%     187.50 ±  3%      +5.8%     191.22 ±  3%  slabinfo.kmalloc-96.num_slabs
    362.44 ± 30%      +6.8%     387.25 ± 33%      -2.2%     354.33 ± 23%  slabinfo.kmalloc-cg-16.active_objs
      1.22 ± 34%     -11.4%       1.08 ± 25%     -18.2%       1.00        slabinfo.kmalloc-cg-16.active_slabs
    362.44 ± 30%      +6.8%     387.25 ± 33%      -2.2%     354.33 ± 23%  slabinfo.kmalloc-cg-16.num_objs
      1.22 ± 34%     -11.4%       1.08 ± 25%     -18.2%       1.00        slabinfo.kmalloc-cg-16.num_slabs
      3251 ±  8%      -0.4%       3239 ±  7%      +0.2%       3256 ±  6%  slabinfo.kmalloc-cg-192.active_objs
     76.78 ±  8%      -0.3%      76.58 ±  7%      +0.3%      77.00 ±  6%  slabinfo.kmalloc-cg-192.active_slabs
      3251 ±  8%      -0.4%       3239 ±  7%      +0.2%       3256 ±  6%  slabinfo.kmalloc-cg-192.num_objs
     76.78 ±  8%      -0.3%      76.58 ±  7%      +0.3%      77.00 ±  6%  slabinfo.kmalloc-cg-192.num_slabs
      3356 ±  7%      -1.8%       3296 ±  6%      -1.5%       3306 ±  6%  slabinfo.kmalloc-cg-1k.active_objs
    104.22 ±  7%      -1.0%     103.17 ±  6%      -1.3%     102.89 ±  6%  slabinfo.kmalloc-cg-1k.active_slabs
      3356 ±  7%      -1.1%       3320 ±  6%      -1.5%       3306 ±  6%  slabinfo.kmalloc-cg-1k.num_objs
    104.22 ±  7%      -1.0%     103.17 ±  6%      -1.3%     102.89 ±  6%  slabinfo.kmalloc-cg-1k.num_slabs
    502.22 ±  6%      +4.8%     526.25 ±  7%      +7.1%     537.78 ± 13%  slabinfo.kmalloc-cg-256.active_objs
     15.11 ±  7%      +4.2%      15.75 ±  8%      +7.4%      16.22 ± 13%  slabinfo.kmalloc-cg-256.active_slabs
    502.22 ±  6%      +4.8%     526.25 ±  7%      +7.1%     537.78 ± 13%  slabinfo.kmalloc-cg-256.num_objs
     15.11 ±  7%      +4.2%      15.75 ±  8%      +7.4%      16.22 ± 13%  slabinfo.kmalloc-cg-256.num_slabs
    305.22 ± 11%      -8.6%     278.83 ± 15%      -9.9%     275.11 ±  8%  slabinfo.kmalloc-cg-2k.active_objs
     18.56 ± 10%     -10.2%      16.67 ± 16%     -11.4%      16.44 ±  9%  slabinfo.kmalloc-cg-2k.active_slabs
    305.22 ± 11%      -8.6%     278.83 ± 15%      -9.9%     275.11 ±  8%  slabinfo.kmalloc-cg-2k.num_objs
     18.56 ± 10%     -10.2%      16.67 ± 16%     -11.4%      16.44 ±  9%  slabinfo.kmalloc-cg-2k.num_slabs
     10395 ± 10%      -0.7%      10326 ±  9%      +4.5%      10868 ±  6%  slabinfo.kmalloc-cg-32.active_objs
     80.67 ± 10%      -0.8%      80.00 ±  9%      +4.4%      84.22 ±  5%  slabinfo.kmalloc-cg-32.active_slabs
     10395 ± 10%      -0.7%      10326 ±  9%      +4.5%      10868 ±  6%  slabinfo.kmalloc-cg-32.num_objs
     80.67 ± 10%      -0.8%      80.00 ±  9%      +4.4%      84.22 ±  5%  slabinfo.kmalloc-cg-32.num_slabs
    768.00            +0.0%     768.00            +0.0%     768.00        slabinfo.kmalloc-cg-4k.active_objs
     96.00            +0.0%      96.00            +0.0%      96.00        slabinfo.kmalloc-cg-4k.active_slabs
    768.00            +0.0%     768.00            +0.0%     768.00        slabinfo.kmalloc-cg-4k.num_objs
     96.00            +0.0%      96.00            +0.0%      96.00        slabinfo.kmalloc-cg-4k.num_slabs
      3068            -0.2%       3063            +0.0%       3068        slabinfo.kmalloc-cg-512.active_objs
     95.89            -0.2%      95.67            +0.0%      95.89        slabinfo.kmalloc-cg-512.active_slabs
      3068            -0.2%       3063            +0.0%       3068        slabinfo.kmalloc-cg-512.num_objs
     95.89            -0.2%      95.67            +0.0%      95.89        slabinfo.kmalloc-cg-512.num_slabs
      1855 ±  9%      +0.7%       1869 ±  6%      -1.3%       1830 ±  8%  slabinfo.kmalloc-cg-64.active_objs
     28.22 ±  9%      +0.7%      28.42 ±  7%      -0.8%      28.00 ±  9%  slabinfo.kmalloc-cg-64.active_slabs
      1855 ±  9%      +0.7%       1869 ±  6%      -1.3%       1830 ±  8%  slabinfo.kmalloc-cg-64.num_objs
     28.22 ±  9%      +0.7%      28.42 ±  7%      -0.8%      28.00 ±  9%  slabinfo.kmalloc-cg-64.num_slabs
     49606            -0.2%      49523            -0.2%      49511        slabinfo.kmalloc-cg-8.active_objs
     96.67            -0.3%      96.42            -0.1%      96.56        slabinfo.kmalloc-cg-8.active_slabs
     49606            -0.2%      49523            -0.2%      49511        slabinfo.kmalloc-cg-8.num_objs
     96.67            -0.3%      96.42            -0.1%      96.56        slabinfo.kmalloc-cg-8.num_slabs
     35.00            -2.9%      34.00 ±  5%      -1.3%      34.56 ±  3%  slabinfo.kmalloc-cg-8k.active_objs
      8.00            -3.1%       7.75 ±  5%      -1.4%       7.89 ±  3%  slabinfo.kmalloc-cg-8k.active_slabs
     35.00            -2.9%      34.00 ±  5%      -1.3%      34.56 ±  3%  slabinfo.kmalloc-cg-8k.num_objs
      8.00            -3.1%       7.75 ±  5%      -1.4%       7.89 ±  3%  slabinfo.kmalloc-cg-8k.num_slabs
    447.00 ± 10%      -5.5%     422.50 ± 11%      -2.1%     437.67 ±  6%  slabinfo.kmalloc-cg-96.active_objs
      9.67 ± 11%      -6.0%       9.08 ± 12%      -2.3%       9.44 ±  7%  slabinfo.kmalloc-cg-96.active_slabs
    447.00 ± 10%      -5.5%     422.50 ± 11%      -2.1%     437.67 ±  6%  slabinfo.kmalloc-cg-96.num_objs
      9.67 ± 11%      -6.0%       9.08 ± 12%      -2.3%       9.44 ±  7%  slabinfo.kmalloc-cg-96.num_slabs
      1144 ± 24%      +6.2%       1216 ± 10%      +1.9%       1166 ± 17%  slabinfo.kmalloc-rcl-128.active_objs
     36.00 ± 22%      +5.6%      38.00 ± 10%      +1.2%      36.44 ± 17%  slabinfo.kmalloc-rcl-128.active_slabs
      1152 ± 22%      +5.6%       1216 ± 10%      +1.2%       1166 ± 17%  slabinfo.kmalloc-rcl-128.num_objs
     36.00 ± 22%      +5.6%      38.00 ± 10%      +1.2%      36.44 ± 17%  slabinfo.kmalloc-rcl-128.num_slabs
     93.33 ± 18%     -13.8%      80.50 ± 14%      -5.0%      88.67 ± 14%  slabinfo.kmalloc-rcl-192.active_objs
      2.22 ± 18%     -13.8%       1.92 ± 14%      -5.0%       2.11 ± 14%  slabinfo.kmalloc-rcl-192.active_slabs
     93.33 ± 18%     -13.8%      80.50 ± 14%      -5.0%      88.67 ± 14%  slabinfo.kmalloc-rcl-192.num_objs
      2.22 ± 18%     -13.8%       1.92 ± 14%      -5.0%       2.11 ± 14%  slabinfo.kmalloc-rcl-192.num_slabs
      7850 ±  4%      +1.5%       7969 ±  3%      +1.3%       7954 ±  4%  slabinfo.kmalloc-rcl-64.active_objs
    123.33 ±  4%      +0.9%     124.50 ±  3%      +1.6%     125.33 ±  3%  slabinfo.kmalloc-rcl-64.active_slabs
      7936 ±  4%      +1.1%       8021 ±  3%      +1.5%       8059 ±  3%  slabinfo.kmalloc-rcl-64.num_objs
    123.33 ±  4%      +0.9%     124.50 ±  3%      +1.6%     125.33 ±  3%  slabinfo.kmalloc-rcl-64.num_slabs
      2616 ± 20%      +0.5%       2630 ± 18%      -2.9%       2540 ± 19%  slabinfo.kmalloc-rcl-96.active_objs
     62.11 ± 20%      +0.2%      62.25 ± 18%      -2.9%      60.33 ± 19%  slabinfo.kmalloc-rcl-96.active_slabs
      2616 ± 20%      +0.5%       2630 ± 18%      -2.9%       2540 ± 19%  slabinfo.kmalloc-rcl-96.num_objs
     62.11 ± 20%      +0.2%      62.25 ± 18%      -2.9%      60.33 ± 19%  slabinfo.kmalloc-rcl-96.num_slabs
    387.56 ±  8%      -0.2%     386.67 ±  9%      +0.0%     387.56 ±  9%  slabinfo.kmem_cache.active_objs
     12.11 ±  8%      -0.2%      12.08 ±  9%      +0.0%      12.11 ±  9%  slabinfo.kmem_cache.active_slabs
    387.56 ±  8%      -0.2%     386.67 ±  9%      +0.0%     387.56 ±  9%  slabinfo.kmem_cache.num_objs
     12.11 ±  8%      -0.2%      12.08 ±  9%      +0.0%      12.11 ±  9%  slabinfo.kmem_cache.num_slabs
    787.78 ±  7%      +0.5%     791.33 ±  7%      +0.0%     787.78 ±  7%  slabinfo.kmem_cache_node.active_objs
     12.78 ±  7%      +0.4%      12.83 ±  7%      +0.0%      12.78 ±  7%  slabinfo.kmem_cache_node.active_slabs
    817.78 ±  7%      +0.4%     821.33 ±  7%      +0.0%     817.78 ±  7%  slabinfo.kmem_cache_node.num_objs
     12.78 ±  7%      +0.4%      12.83 ±  7%      +0.0%      12.78 ±  7%  slabinfo.kmem_cache_node.num_slabs
     18770 ±  2%      +0.4%      18836            +1.2%      18997        slabinfo.lsm_file_cache.active_objs
    109.44 ±  2%      +0.4%     109.83            +1.2%     110.78        slabinfo.lsm_file_cache.active_slabs
     18770 ±  2%      +0.4%      18836            +1.2%      18997        slabinfo.lsm_file_cache.num_objs
    109.44 ±  2%      +0.4%     109.83            +1.2%     110.78        slabinfo.lsm_file_cache.num_slabs
      2960 ±  2%      -0.5%       2944 ±  3%      +0.3%       2969 ±  3%  slabinfo.mm_struct.active_objs
     98.00 ±  2%      -0.3%      97.67 ±  3%      +0.3%      98.33 ±  2%  slabinfo.mm_struct.active_slabs
      2960 ±  2%      -0.5%       2944 ±  3%      +0.3%       2969 ±  3%  slabinfo.mm_struct.num_objs
     98.00 ±  2%      -0.3%      97.67 ±  3%      +0.3%      98.33 ±  2%  slabinfo.mm_struct.num_slabs
      1112 ±  6%      -9.1%       1010 ±  8%      -8.5%       1017 ±  8%  slabinfo.mnt_cache.active_objs
     21.11 ±  6%      -9.6%      19.08 ±  8%     -10.0%      19.00 ±  8%  slabinfo.mnt_cache.active_slabs
      1112 ±  6%      -9.1%       1010 ±  8%      -8.5%       1017 ±  8%  slabinfo.mnt_cache.num_objs
     21.11 ±  6%      -9.6%      19.08 ±  8%     -10.0%      19.00 ±  8%  slabinfo.mnt_cache.num_slabs
     34.00            +0.0%      34.00            +0.0%      34.00        slabinfo.mqueue_inode_cache.active_objs
      1.00            +0.0%       1.00            +0.0%       1.00        slabinfo.mqueue_inode_cache.active_slabs
     34.00            +0.0%      34.00            +0.0%      34.00        slabinfo.mqueue_inode_cache.num_objs
      1.00            +0.0%       1.00            +0.0%       1.00        slabinfo.mqueue_inode_cache.num_slabs
    768.00            +0.0%     768.00            +0.0%     768.00        slabinfo.names_cache.active_objs
     96.00            +0.0%      96.00            +0.0%      96.00        slabinfo.names_cache.active_slabs
    768.00            +0.0%     768.00            +0.0%     768.00        slabinfo.names_cache.num_objs
     96.00            +0.0%      96.00            +0.0%      96.00        slabinfo.names_cache.num_slabs
     34.00            +0.0%      34.00            +0.0%      34.00        slabinfo.nfs_read_data.active_objs
      1.00            +0.0%       1.00            +0.0%       1.00        slabinfo.nfs_read_data.active_slabs
     34.00            +0.0%      34.00            +0.0%      34.00        slabinfo.nfs_read_data.num_objs
      1.00            +0.0%       1.00            +0.0%       1.00        slabinfo.nfs_read_data.num_slabs
    124.00            +0.0%     124.00            +0.0%     124.00        slabinfo.numa_policy.active_objs
      2.00            +0.0%       2.00            +0.0%       2.00        slabinfo.numa_policy.active_slabs
    124.00            +0.0%     124.00            +0.0%     124.00        slabinfo.numa_policy.num_objs
      2.00            +0.0%       2.00            +0.0%       2.00        slabinfo.numa_policy.num_slabs
      8212 ±  9%      -0.8%       8145 ±  7%      +4.0%       8538 ±  5%  slabinfo.pde_opener.active_objs
     80.00 ±  9%      -1.0%      79.17 ±  7%      +3.9%      83.11 ±  5%  slabinfo.pde_opener.active_slabs
      8212 ±  9%      -0.8%       8145 ±  7%      +4.0%       8538 ±  5%  slabinfo.pde_opener.num_objs
     80.00 ±  9%      -1.0%      79.17 ±  7%      +3.9%      83.11 ±  5%  slabinfo.pde_opener.num_slabs
      4075            -0.9%       4037 ±  2%      +0.2%       4084        slabinfo.perf_event.active_objs
    153.89            -0.8%     152.67            +0.2%     154.22        slabinfo.perf_event.active_slabs
      4166            -0.8%       4133            +0.1%       4172        slabinfo.perf_event.num_objs
    153.89            -0.8%     152.67            +0.2%     154.22        slabinfo.perf_event.num_slabs
      4703 ±  3%      +3.2%       4852 ±  3%      +1.1%       4757 ±  4%  slabinfo.pid.active_objs
    146.11 ±  3%      +3.2%     150.83 ±  3%      +1.1%     147.78 ±  4%  slabinfo.pid.active_slabs
      4703 ±  3%      +3.2%       4852 ±  3%      +1.1%       4757 ±  4%  slabinfo.pid.num_objs
    146.11 ±  3%      +3.2%     150.83 ±  3%      +1.1%     147.78 ±  4%  slabinfo.pid.num_slabs
      1509 ±  8%      -4.0%       1449 ±  8%      +4.9%       1584 ±  5%  slabinfo.pool_workqueue.active_objs
     46.56 ±  8%      -3.7%      44.83 ±  8%      +5.5%      49.11 ±  6%  slabinfo.pool_workqueue.active_slabs
      1509 ±  8%      -3.8%       1451 ±  8%      +5.1%       1587 ±  5%  slabinfo.pool_workqueue.num_objs
     46.56 ±  8%      -3.7%      44.83 ±  8%      +5.5%      49.11 ±  6%  slabinfo.pool_workqueue.num_slabs
      2930            -1.5%       2887            -1.1%       2898 ±  2%  slabinfo.proc_dir_entry.active_objs
     69.78            -1.5%      68.75            -1.1%      69.00 ±  2%  slabinfo.proc_dir_entry.active_slabs
      2930            -1.5%       2887            -1.1%       2898 ±  2%  slabinfo.proc_dir_entry.num_objs
     69.78            -1.5%      68.75            -1.1%      69.00 ±  2%  slabinfo.proc_dir_entry.num_slabs
     14278            +1.0%      14427            +0.3%      14325        slabinfo.proc_inode_cache.active_objs
    290.89            +1.1%     294.17            +0.4%     292.00        slabinfo.proc_inode_cache.active_slabs
     14282            +1.0%      14431            +0.4%      14339        slabinfo.proc_inode_cache.num_objs
    290.89            +1.1%     294.17            +0.4%     292.00        slabinfo.proc_inode_cache.num_slabs
     27516            +0.2%      27564            -0.2%      27449        slabinfo.radix_tree_node.active_objs
    491.56            +0.0%     491.67            -0.4%     489.67        slabinfo.radix_tree_node.active_slabs
     27553            +0.0%      27564            -0.4%      27452        slabinfo.radix_tree_node.num_objs
    491.56            +0.0%     491.67            -0.4%     489.67        slabinfo.radix_tree_node.num_slabs
     78.33 ± 12%      +0.5%      78.75 ±  8%      -4.3%      75.00        slabinfo.request_queue.active_objs
      5.22 ± 12%      +0.5%       5.25 ±  8%      -4.3%       5.00        slabinfo.request_queue.active_slabs
     78.33 ± 12%      +0.5%      78.75 ±  8%      -4.3%      75.00        slabinfo.request_queue.num_objs
      5.22 ± 12%      +0.5%       5.25 ±  8%      -4.3%       5.00        slabinfo.request_queue.num_slabs
     51.00            +0.0%      51.00            +0.0%      51.00        slabinfo.rpc_inode_cache.active_objs
      1.00            +0.0%       1.00            +0.0%       1.00        slabinfo.rpc_inode_cache.active_slabs
     51.00            +0.0%      51.00            +0.0%      51.00        slabinfo.rpc_inode_cache.num_objs
      1.00            +0.0%       1.00            +0.0%       1.00        slabinfo.rpc_inode_cache.num_slabs
    924.44 ±  2%      +0.1%     925.33 ±  2%      -1.2%     913.78        slabinfo.scsi_sense_cache.active_objs
     28.89 ±  2%      +0.1%      28.92 ±  2%      -1.2%      28.56        slabinfo.scsi_sense_cache.active_slabs
    924.44 ±  2%      +0.1%     925.33 ±  2%      -1.2%     913.78        slabinfo.scsi_sense_cache.num_objs
     28.89 ±  2%      +0.1%      28.92 ±  2%      -1.2%      28.56        slabinfo.scsi_sense_cache.num_slabs
      3264            +0.0%       3264            +0.0%       3264        slabinfo.seq_file.active_objs
     96.00            +0.0%      96.00            +0.0%      96.00        slabinfo.seq_file.active_slabs
      3264            +0.0%       3264            +0.0%       3264        slabinfo.seq_file.num_objs
     96.00            +0.0%      96.00            +0.0%      96.00        slabinfo.seq_file.num_slabs
      4731 ±  6%      +3.3%       4887 ±  6%      +6.9%       5056 ±  7%  slabinfo.shmem_inode_cache.active_objs
    102.56 ±  7%      +3.0%     105.67 ±  6%      +6.7%     109.44 ±  7%  slabinfo.shmem_inode_cache.active_slabs
      4731 ±  6%      +3.3%       4887 ±  6%      +6.9%       5056 ±  7%  slabinfo.shmem_inode_cache.num_objs
    102.56 ±  7%      +3.0%     105.67 ±  6%      +6.7%     109.44 ±  7%  slabinfo.shmem_inode_cache.num_slabs
      2217 ±  4%      -2.7%       2156 ±  4%      -0.9%       2198 ±  4%  slabinfo.sighand_cache.active_objs
    148.11 ±  3%      -2.6%     144.25 ±  4%      -1.1%     146.56 ±  4%  slabinfo.sighand_cache.active_slabs
      2232 ±  3%      -2.6%       2173 ±  4%      -1.1%       2208 ±  4%  slabinfo.sighand_cache.num_objs
    148.11 ±  3%      -2.6%     144.25 ±  4%      -1.1%     146.56 ±  4%  slabinfo.sighand_cache.num_slabs
      4292 ±  4%      +1.2%       4344 ±  5%      +0.3%       4304 ±  4%  slabinfo.signal_cache.active_objs
    152.67 ±  4%      +1.8%     155.42 ±  5%      +0.3%     153.11 ±  4%  slabinfo.signal_cache.active_slabs
      4292 ±  4%      +1.9%       4372 ±  5%      +0.3%       4304 ±  4%  slabinfo.signal_cache.num_objs
    152.67 ±  4%      +1.8%     155.42 ±  5%      +0.3%     153.11 ±  4%  slabinfo.signal_cache.num_slabs
    313.00 ± 25%      -3.7%     301.33 ± 14%      +6.6%     333.67 ± 14%  slabinfo.skbuff_fclone_cache.active_objs
      9.33 ± 23%      +0.9%       9.42 ± 14%     +10.7%      10.33 ± 14%  slabinfo.skbuff_fclone_cache.active_slabs
    313.00 ± 25%      -3.7%     301.33 ± 14%      +6.6%     333.67 ± 14%  slabinfo.skbuff_fclone_cache.num_objs
      9.33 ± 23%      +0.9%       9.42 ± 14%     +10.7%      10.33 ± 14%  slabinfo.skbuff_fclone_cache.num_slabs
      4730 ±  7%      -4.1%       4535 ±  6%      -5.8%       4456 ±  4%  slabinfo.skbuff_head_cache.active_objs
    147.56 ±  7%      -4.1%     141.50 ±  6%      -5.7%     139.11 ±  4%  slabinfo.skbuff_head_cache.active_slabs
      4730 ±  7%      -4.1%       4535 ±  6%      -5.8%       4456 ±  4%  slabinfo.skbuff_head_cache.num_objs
    147.56 ±  7%      -4.1%     141.50 ±  6%      -5.7%     139.11 ±  4%  slabinfo.skbuff_head_cache.num_slabs
      3053 ± 14%      -0.2%       3048 ± 13%      +5.4%       3218 ±  9%  slabinfo.sock_inode_cache.active_objs
     78.00 ± 14%      +0.0%      78.00 ± 13%      +5.4%      82.22 ± 10%  slabinfo.sock_inode_cache.active_slabs
      3053 ± 14%      -0.2%       3048 ± 13%      +5.4%       3218 ±  9%  slabinfo.sock_inode_cache.num_objs
     78.00 ± 14%      +0.0%      78.00 ± 13%      +5.4%      82.22 ± 10%  slabinfo.sock_inode_cache.num_slabs
      1432 ± 14%      +3.3%       1479 ±  8%      +1.0%       1445 ± 12%  slabinfo.task_delay_info.active_objs
     27.89 ± 14%      +2.2%      28.50 ±  8%      +0.8%      28.11 ± 12%  slabinfo.task_delay_info.active_slabs
      1432 ± 14%      +3.3%       1479 ±  8%      +1.0%       1445 ± 12%  slabinfo.task_delay_info.num_objs
     27.89 ± 14%      +2.2%      28.50 ±  8%      +0.8%      28.11 ± 12%  slabinfo.task_delay_info.num_slabs
      1328 ±  7%      +2.9%       1366 ± 10%      +7.9%       1433 ±  4%  slabinfo.task_group.active_objs
     28.33 ±  7%      +3.2%      29.25 ± 10%      +7.8%      30.56 ±  5%  slabinfo.task_group.active_slabs
      1328 ±  7%      +2.9%       1366 ± 10%      +7.9%       1433 ±  4%  slabinfo.task_group.num_objs
     28.33 ±  7%      +3.2%      29.25 ± 10%      +7.8%      30.56 ±  5%  slabinfo.task_group.num_slabs
      1260            +0.5%       1265            +0.9%       1271        slabinfo.task_struct.active_objs
      1266            +0.5%       1272            +0.9%       1277        slabinfo.task_struct.active_slabs
      1266            +0.5%       1272            +0.9%       1277        slabinfo.task_struct.num_objs
      1266            +0.5%       1272            +0.9%       1277        slabinfo.task_struct.num_slabs
    294.33 ± 10%      -7.2%     273.00 ±  8%      -1.7%     289.33 ± 12%  slabinfo.taskstats.active_objs
      6.33 ± 10%      -7.9%       5.83 ±  9%      -1.8%       6.22 ± 12%  slabinfo.taskstats.active_slabs
    294.33 ± 10%      -7.2%     273.00 ±  8%      -1.7%     289.33 ± 12%  slabinfo.taskstats.num_objs
      6.33 ± 10%      -7.9%       5.83 ±  9%      -1.8%       6.22 ± 12%  slabinfo.taskstats.num_slabs
      3069 ±  5%      -1.7%       3017 ±  3%      -1.5%       3022 ±  3%  slabinfo.trace_event_file.active_objs
     66.22 ±  5%      -1.5%      65.25 ±  2%      -1.2%      65.44 ±  3%  slabinfo.trace_event_file.active_slabs
      3069 ±  5%      -1.7%       3017 ±  3%      -1.5%       3022 ±  3%  slabinfo.trace_event_file.num_objs
     66.22 ±  5%      -1.5%      65.25 ±  2%      -1.2%      65.44 ±  3%  slabinfo.trace_event_file.num_slabs
     33.00            +0.0%      33.00            +0.0%      33.00        slabinfo.tw_sock_TCP.active_objs
      1.00            +0.0%       1.00            +0.0%       1.00        slabinfo.tw_sock_TCP.active_slabs
     33.00            +0.0%      33.00            +0.0%      33.00        slabinfo.tw_sock_TCP.num_objs
      1.00            +0.0%       1.00            +0.0%       1.00        slabinfo.tw_sock_TCP.num_slabs
     54128 ±  7%      -3.8%      52087 ±  6%      +1.9%      55144 ±  5%  slabinfo.vm_area_struct.active_objs
      1355 ±  7%      -3.7%       1304 ±  6%      +2.0%       1382 ±  5%  slabinfo.vm_area_struct.active_slabs
     54236 ±  7%      -3.7%      52216 ±  6%      +2.0%      55300 ±  5%  slabinfo.vm_area_struct.num_objs
      1355 ±  7%      -3.7%       1304 ±  6%      +2.0%       1382 ±  5%  slabinfo.vm_area_struct.num_slabs
     21955            +0.4%      22037            -0.1%      21932 ±  2%  slabinfo.vmap_area.active_objs
    343.00            +0.4%     344.50            -0.0%     342.89 ±  2%  slabinfo.vmap_area.active_slabs
     21992            +0.4%      22071            -0.1%      21972 ±  2%  slabinfo.vmap_area.num_objs
    343.00            +0.4%     344.50            -0.0%     342.89 ±  2%  slabinfo.vmap_area.num_slabs
     42.00            +0.0%      42.00            +0.0%      42.00        slabinfo.xfrm_state.active_objs
      1.00            +0.0%       1.00            +0.0%       1.00        slabinfo.xfrm_state.active_slabs
     42.00            +0.0%      42.00            +0.0%      42.00        slabinfo.xfrm_state.num_objs
      1.00            +0.0%       1.00            +0.0%       1.00        slabinfo.xfrm_state.num_slabs
     66.46 ±  8%      -1.7       64.80 ±  9%      -6.3       60.12 ±  5%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     66.50 ±  8%      -1.6       64.85 ±  9%      -6.3       60.15 ±  5%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     66.50 ±  8%      -1.6       64.85 ±  9%      -6.3       60.15 ±  5%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     17.94 ±  8%      -1.0       16.93 ±  6%      -0.3       17.64 ±  8%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     60.15 ±  9%      -1.0       59.19 ± 10%      -6.6       53.59 ±  5%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     59.34 ±  9%      -1.0       58.37 ± 10%      -6.5       52.82 ±  6%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     14.53 ±  9%      -0.8       13.73 ±  5%      -0.6       13.97 ± 10%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
     67.00 ±  8%      -0.7       66.27 ±  6%      -6.3       60.69 ±  5%  perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
      4.98 ± 15%      -0.7        4.30 ± 14%      +0.4        5.37 ±  7%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      9.81 ±  8%      -0.5        9.31 ±  5%      -0.4        9.41 ± 10%  perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      2.54 ± 27%      -0.5        2.05 ± 33%      +0.5        2.99 ± 12%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary
      9.50 ±  8%      -0.5        9.02 ±  5%      -0.3        9.24 ± 10%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      1.98 ± 36%      -0.5        1.52 ± 44%      +0.5        2.52 ± 13%  perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry
      0.90 ± 70%      -0.4        0.53 ±127%      +0.6        1.46 ± 22%  perf-profile.calltrace.cycles-pp.timekeeping_max_deferment.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle
      8.52 ± 23%      -0.3        8.18 ± 13%      +1.1        9.66 ± 13%  perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.86 ± 58%      -0.3        0.57 ± 98%      +0.4        1.24 ± 23%  perf-profile.calltrace.cycles-pp.tick_nohz_irq_exit.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      2.65 ± 14%      -0.3        2.36 ± 12%      +0.2        2.80 ± 14%  perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      3.25 ± 15%      -0.3        2.97 ± 13%      +0.3        3.53 ± 15%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read
      3.14 ± 15%      -0.3        2.86 ± 13%      +0.3        3.42 ± 15%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      2.94 ± 15%      -0.3        2.66 ± 13%      +0.2        3.17 ± 14%  perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      4.00 ± 15%      -0.3        3.74 ± 13%      +0.4        4.43 ± 14%  perf-profile.calltrace.cycles-pp.read
      2.35 ± 34%      -0.3        2.10 ± 44%      +0.6        2.98 ± 15%  perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.24 ±112%      -0.2        0.00            +0.2        0.42 ± 72%  perf-profile.calltrace.cycles-pp.anon_pipe_buf_release.pipe_read.new_sync_read.vfs_read.ksys_read
      0.78 ± 71%      -0.2        0.55 ±101%      +0.4        1.21 ± 24%  perf-profile.calltrace.cycles-pp.ktime_get.tick_nohz_irq_exit.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      3.00 ± 12%      -0.2        2.78 ±  9%      -0.3        2.67 ±  8%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      9.50 ± 23%      -0.2        9.29 ± 13%      +1.4       10.91 ± 12%  perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      3.61 ± 12%      -0.2        3.41 ±  9%      -0.4        3.21 ±  8%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      0.19 ±141%      -0.2        0.00            -0.2        0.00        perf-profile.calltrace.cycles-pp.fsnotify.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.26 ± 54%      -0.2        1.08 ± 83%      +0.5        1.79 ± 26%  perf-profile.calltrace.cycles-pp.ktime_get.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      5.66 ± 13%      -0.2        5.49 ± 12%      -0.9        4.76 ±  7%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      3.10 ± 12%      -0.2        2.94 ± 13%      +0.6        3.65 ± 13%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      2.55 ± 13%      -0.2        2.39 ± 14%      +0.5        3.03 ± 12%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      0.82 ± 40%      -0.2        0.66 ± 14%      +0.1        0.90 ± 15%  perf-profile.calltrace.cycles-pp.security_file_permission.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      3.19 ± 13%      -0.2        3.04 ± 13%      +0.6        3.77 ± 13%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
      2.23 ± 12%      -0.2        2.08 ±  8%      -0.2        2.08 ±  7%  perf-profile.calltrace.cycles-pp.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      3.24 ± 13%      -0.1        3.09 ± 11%      -0.5        2.77 ±  8%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
      2.86 ± 13%      -0.1        2.72 ± 13%      +0.5        3.37 ± 12%  perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      3.93 ± 14%      -0.1        3.80 ± 13%      +0.7        4.67 ± 11%  perf-profile.calltrace.cycles-pp.write
      0.28 ±114%      -0.1        0.16 ±173%      +0.2        0.45 ± 74%  perf-profile.calltrace.cycles-pp.calc_global_load_tick.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer
      1.78 ± 12%      -0.1        1.66 ±  7%      -0.1        1.68 ±  8%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      0.62 ±  7%      -0.1        0.51 ± 46%      -0.1        0.53 ± 36%  perf-profile.calltrace.cycles-pp.load_balance.rebalance_domains.__softirqentry_text_start.irq_exit_rcu.sysvec_apic_timer_interrupt
      0.94 ± 22%      -0.1        0.83 ± 19%      +0.1        1.07 ±  7%  perf-profile.calltrace.cycles-pp.lapic_next_deadline.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      1.59 ± 10%      -0.1        1.51 ±  9%      -0.1        1.51 ± 10%  perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      0.12 ±187%      -0.1        0.04 ±331%      -0.1        0.00        perf-profile.calltrace.cycles-pp.rcu_idle_exit.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      1.01 ± 11%      -0.1        0.94 ±  9%      -0.0        0.98 ±  8%  perf-profile.calltrace.cycles-pp.rebalance_domains.__softirqentry_text_start.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.06 ±282%      -0.1        0.00            -0.1        0.00        perf-profile.calltrace.cycles-pp.fsnotify.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.06 ±282%      -0.1        0.00            -0.1        0.00        perf-profile.calltrace.cycles-pp._raw_spin_trylock.rebalance_domains.__softirqentry_text_start.irq_exit_rcu.sysvec_apic_timer_interrupt
      0.06 ±282%      -0.1        0.00            -0.1        0.00        perf-profile.calltrace.cycles-pp.update_rq_clock.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer
      0.18 ±142%      -0.0        0.14 ±173%      -0.2        0.00        perf-profile.calltrace.cycles-pp.hrtimer_update_next_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.12 ±187%      -0.0        0.09 ±223%      -0.1        0.06 ±282%  perf-profile.calltrace.cycles-pp.mutex_unlock.pipe_read.new_sync_read.vfs_read.ksys_read
      0.41 ± 99%      -0.0        0.39 ± 92%      +0.2        0.59 ± 62%  perf-profile.calltrace.cycles-pp.ktime_get_update_offsets_now.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.92 ± 13%      -0.0        0.90 ± 17%      -0.1        0.84 ± 13%  perf-profile.calltrace.cycles-pp.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      0.26 ±112%      +0.0        0.26 ±120%      -0.3        0.00        perf-profile.calltrace.cycles-pp.rcu_sched_clock_irq.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      0.18 ±142%      +0.0        0.19 ±142%      +0.1        0.31 ± 90%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.pipe_read.new_sync_read.vfs_read.ksys_read
      0.00            +0.0        0.04 ±331%      +0.2        0.18 ±142%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_safe_stack
      0.56 ± 38%      +0.0        0.60 ± 33%      +0.2        0.77 ± 37%  perf-profile.calltrace.cycles-pp.security_file_permission.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.12 ±282%      +0.0        0.16 ±223%      +0.2        0.28 ±144%  perf-profile.calltrace.cycles-pp.ktime_get.tick_irq_enter.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.00            +0.0        0.04 ±331%      +0.0        0.00        perf-profile.calltrace.cycles-pp.ktime_get.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
      0.00            +0.0        0.05 ±331%      +0.2        0.17 ±141%  perf-profile.calltrace.cycles-pp.mutex_unlock.pipe_write.new_sync_write.vfs_write.ksys_write
      0.00            +0.0        0.05 ±331%      +0.2        0.24 ±112%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.pipe_write.new_sync_write.vfs_write.ksys_write
      0.90 ± 26%      +0.1        0.95 ± 17%      -0.4        0.52 ± 82%  perf-profile.calltrace.cycles-pp.tick_irq_enter.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      0.96 ± 25%      +0.1        1.01 ± 15%      -0.2        0.73 ± 36%  perf-profile.calltrace.cycles-pp.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      0.40 ± 72%      +0.1        0.46 ± 61%      +0.2        0.65 ± 38%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.12 ±187%      +0.1        0.18 ±141%      +0.4        0.50 ± 55%  perf-profile.calltrace.cycles-pp.common_file_perm.security_file_permission.vfs_read.ksys_read.do_syscall_64
      0.78 ± 39%      +0.1        0.86 ± 17%      +0.2        0.99 ± 15%  perf-profile.calltrace.cycles-pp.atime_needs_update.touch_atime.pipe_read.new_sync_read.vfs_read
      0.00            +0.1        0.09 ±223%      +0.1        0.12 ±187%  perf-profile.calltrace.cycles-pp.find_busiest_group.load_balance.rebalance_domains.__softirqentry_text_start.irq_exit_rcu
      0.19 ±142%      +0.1        0.28 ±100%      +0.3        0.51 ± 55%  perf-profile.calltrace.cycles-pp.common_file_perm.security_file_permission.vfs_write.ksys_write.do_syscall_64
      0.76 ± 39%      +0.1        0.85 ± 18%      +0.2        0.96 ± 14%  perf-profile.calltrace.cycles-pp.mutex_lock.pipe_read.new_sync_read.vfs_read.ksys_read
      0.59 ± 36%      +0.1        0.69 ± 16%      +0.1        0.71 ± 38%  perf-profile.calltrace.cycles-pp.__fget_light.__fdget_pos.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.12 ±187%      +0.1        0.23 ±118%      -0.1        0.00        perf-profile.calltrace.cycles-pp.__remove_hrtimer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      0.72 ± 40%      +0.1        0.83 ± 20%      +0.3        1.03 ± 18%  perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.33 ± 91%      +0.1        0.46 ± 59%      +0.4        0.68 ± 10%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write
      0.66 ± 37%      +0.1        0.79 ± 17%      +0.2        0.84 ± 17%  perf-profile.calltrace.cycles-pp.__fdget_pos.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.43 ± 32%      +0.1        1.56 ± 24%      +0.4        1.86 ± 21%  perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin.copy_page_from_iter.pipe_write.new_sync_write
      0.60 ± 39%      +0.1        0.74 ± 13%      +0.2        0.83 ± 11%  perf-profile.calltrace.cycles-pp.__fget_light.__fdget_pos.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.68 ± 31%      +0.1        1.82 ± 25%      +0.7        2.38 ± 16%  perf-profile.calltrace.cycles-pp.copyin.copy_page_from_iter.pipe_write.new_sync_write.vfs_write
      0.67 ± 39%      +0.1        0.82 ± 13%      +0.3        0.93 ± 12%  perf-profile.calltrace.cycles-pp.__fdget_pos.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.89 ± 39%      +0.1        1.04 ± 16%      +0.3        1.18 ± 15%  perf-profile.calltrace.cycles-pp.touch_atime.pipe_read.new_sync_read.vfs_read.ksys_read
      0.67 ± 38%      +0.2        0.82 ± 16%      +0.3        0.99 ± 16%  perf-profile.calltrace.cycles-pp.mutex_lock.pipe_write.new_sync_write.vfs_write.ksys_write
      1.06 ± 25%      +0.2        1.21 ± 15%      +0.3        1.35 ± 15%  perf-profile.calltrace.cycles-pp.__entry_text_start
      0.21 ±143%      +0.2        0.37 ± 85%      -0.2        0.00        perf-profile.calltrace.cycles-pp.io_serial_in.wait_for_xmitr.serial8250_console_putchar.uart_console_write.serial8250_console_write
      0.92 ± 21%      +0.2        1.08 ± 14%      +0.3        1.23 ±  7%  perf-profile.calltrace.cycles-pp.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write.ksys_write
      0.35 ±114%      +0.2        0.52 ± 73%      -0.3        0.00        perf-profile.calltrace.cycles-pp.wait_for_xmitr.serial8250_console_putchar.uart_console_write.serial8250_console_write.console_unlock
      0.35 ±114%      +0.2        0.52 ± 73%      -0.3        0.00        perf-profile.calltrace.cycles-pp.serial8250_console_putchar.uart_console_write.serial8250_console_write.console_unlock.vprintk_emit
      0.36 ±114%      +0.2        0.54 ± 73%      -0.4        0.00        perf-profile.calltrace.cycles-pp.uart_console_write.serial8250_console_write.console_unlock.vprintk_emit.printk
      0.37 ±114%      +0.2        0.55 ± 73%      -0.4        0.00        perf-profile.calltrace.cycles-pp.serial8250_console_write.console_unlock.vprintk_emit.printk.irq_work_single
      0.38 ±114%      +0.2        0.57 ± 73%      -0.4        0.00        perf-profile.calltrace.cycles-pp.irq_work_run_list.irq_work_run.__sysvec_irq_work.sysvec_irq_work.asm_sysvec_irq_work
      0.38 ±114%      +0.2        0.57 ± 73%      -0.4        0.00        perf-profile.calltrace.cycles-pp.irq_work_run.__sysvec_irq_work.sysvec_irq_work.asm_sysvec_irq_work.cpuidle_enter_state
      0.38 ±114%      +0.2        0.57 ± 73%      -0.4        0.00        perf-profile.calltrace.cycles-pp.irq_work_single.irq_work_run_list.irq_work_run.__sysvec_irq_work.sysvec_irq_work
      0.38 ±114%      +0.2        0.57 ± 73%      -0.4        0.00        perf-profile.calltrace.cycles-pp.printk.irq_work_single.irq_work_run_list.irq_work_run.__sysvec_irq_work
      0.38 ±114%      +0.2        0.57 ± 73%      -0.4        0.00        perf-profile.calltrace.cycles-pp.vprintk_emit.printk.irq_work_single.irq_work_run_list.irq_work_run
      0.38 ±114%      +0.2        0.57 ± 73%      -0.4        0.00        perf-profile.calltrace.cycles-pp.console_unlock.vprintk_emit.printk.irq_work_single.irq_work_run_list
      1.82 ± 31%      +0.2        2.01 ± 24%      +0.5        2.27 ± 13%  perf-profile.calltrace.cycles-pp.copyout.copy_page_to_iter.pipe_read.new_sync_read.vfs_read
      0.38 ±114%      +0.2        0.57 ± 73%      -0.4        0.00        perf-profile.calltrace.cycles-pp.asm_sysvec_irq_work.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      0.38 ±114%      +0.2        0.57 ± 73%      -0.4        0.00        perf-profile.calltrace.cycles-pp.sysvec_irq_work.asm_sysvec_irq_work.cpuidle_enter_state.cpuidle_enter.do_idle
      0.38 ±114%      +0.2        0.57 ± 73%      -0.4        0.00        perf-profile.calltrace.cycles-pp.__sysvec_irq_work.sysvec_irq_work.asm_sysvec_irq_work.cpuidle_enter_state.cpuidle_enter
      0.20 ±141%      +0.2        0.41 ± 72%      +0.4        0.58 ± 37%  perf-profile.calltrace.cycles-pp.file_update_time.pipe_write.new_sync_write.vfs_write.ksys_write
      0.29 ±113%      +0.2        0.53 ± 50%      +0.3        0.61 ± 36%  perf-profile.calltrace.cycles-pp.syscall_return_via_sysret
      3.16 ± 20%      +0.3        3.41 ± 13%      +0.8        3.94 ± 11%  perf-profile.calltrace.cycles-pp.copy_page_from_iter.pipe_write.new_sync_write.vfs_write.ksys_write
      8.16 ± 19%      +0.3        8.42 ± 14%      +1.6        9.80 ±  8%  perf-profile.calltrace.cycles-pp.pipe_read.new_sync_read.vfs_read.ksys_read.do_syscall_64
      1.43 ± 27%      +0.3        1.70 ± 23%      +0.4        1.83 ± 17%  perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyout.copy_page_to_iter.pipe_read.new_sync_read
      3.19 ± 19%      +0.3        3.48 ± 12%      +0.5        3.74 ± 10%  perf-profile.calltrace.cycles-pp.copy_page_to_iter.pipe_read.new_sync_read.vfs_read.ksys_read
      8.69 ± 19%      +0.4        9.06 ± 14%      +1.7       10.44 ±  8%  perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      8.23 ± 21%      +0.5        8.72 ± 12%      +2.0       10.23 ± 13%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     20.32 ± 22%      +0.5       20.83 ± 13%      +4.0       24.32 ± 13%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
      9.23 ± 21%      +0.6        9.85 ± 13%      +2.2       11.43 ± 13%  perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     20.93 ± 22%      +0.7       21.59 ± 13%      +4.2       25.14 ± 13%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     40.56 ± 12%      +0.7       41.30 ±  9%      -5.7       34.83 ± 10%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      8.34 ± 19%      +0.8        9.10 ± 13%      +2.3       10.67 ±  8%  perf-profile.calltrace.cycles-pp.pipe_write.new_sync_write.vfs_write.ksys_write.do_syscall_64
      8.94 ± 18%      +0.8        9.74 ± 13%      +2.5       11.41 ±  8%  perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.30 ± 90%      +0.8        1.15 ±260%      -0.2        0.06 ±282%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_kernel
      0.31 ± 90%      +0.9        1.20 ±248%      -0.1        0.17 ±141%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_kernel.secondary_startup_64_no_verify
      0.41 ± 71%      +0.9        1.32 ±222%      -0.0        0.39 ± 71%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_kernel.secondary_startup_64_no_verify
      0.41 ± 71%      +0.9        1.32 ±222%      -0.0        0.39 ± 71%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_kernel.secondary_startup_64_no_verify
      0.41 ± 71%      +0.9        1.32 ±222%      -0.0        0.39 ± 71%  perf-profile.calltrace.cycles-pp.start_kernel.secondary_startup_64_no_verify
     66.50 ±  8%      -1.6       64.85 ±  9%      -6.3       60.15 ±  5%  perf-profile.children.cycles-pp.start_secondary
      1.46 ± 22%      -1.5        0.00            -1.5        0.00        perf-profile.children.cycles-pp.fsnotify
     16.66 ±  8%      -0.9       15.76 ±  5%      -0.4       16.29 ±  9%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
     14.78 ±  9%      -0.8       13.97 ±  5%      -0.5       14.25 ± 10%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
     67.00 ±  8%      -0.7       66.27 ±  6%      -6.3       60.69 ±  5%  perf-profile.children.cycles-pp.do_idle
     67.00 ±  8%      -0.7       66.27 ±  6%      -6.3       60.69 ±  5%  perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     67.00 ±  8%      -0.7       66.27 ±  6%      -6.3       60.69 ±  5%  perf-profile.children.cycles-pp.cpu_startup_entry
      5.07 ± 15%      -0.7        4.38 ± 14%      +0.4        5.45 ±  7%  perf-profile.children.cycles-pp.menu_select
     11.22 ± 19%      -0.6       10.58 ± 13%      +1.3       12.49 ±  8%  perf-profile.children.cycles-pp.vfs_read
      9.98 ±  8%      -0.5        9.48 ±  5%      -0.4        9.62 ± 10%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      2.58 ± 27%      -0.5        2.09 ± 33%      +0.5        3.04 ± 12%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      9.68 ±  8%      -0.5        9.20 ±  5%      -0.2        9.46 ± 10%  perf-profile.children.cycles-pp.hrtimer_interrupt
     12.49 ± 19%      -0.5       12.02 ± 13%      +1.6       14.11 ±  8%  perf-profile.children.cycles-pp.ksys_read
      2.04 ± 35%      -0.5        1.57 ± 43%      +0.5        2.56 ± 14%  perf-profile.children.cycles-pp.tick_nohz_next_event
      2.99 ± 42%      -0.4        2.59 ± 57%      +0.9        3.91 ± 29%  perf-profile.children.cycles-pp.ktime_get
      0.97 ± 59%      -0.3        0.67 ± 86%      +0.5        1.47 ± 22%  perf-profile.children.cycles-pp.timekeeping_max_deferment
      0.57 ± 23%      -0.3        0.30 ± 23%      +0.2        0.74 ± 17%  perf-profile.children.cycles-pp.anon_pipe_buf_release
      4.20 ± 15%      -0.2        3.95 ± 13%      +0.5        4.68 ± 13%  perf-profile.children.cycles-pp.read
      2.40 ± 33%      -0.2        2.16 ± 43%      +0.6        3.04 ± 15%  perf-profile.children.cycles-pp.clockevents_program_event
      1.92 ± 21%      -0.2        1.69 ± 13%      +0.3        2.23 ± 12%  perf-profile.children.cycles-pp.security_file_permission
      0.90 ± 52%      -0.2        0.67 ± 70%      +0.3        1.25 ± 23%  perf-profile.children.cycles-pp.tick_nohz_irq_exit
      3.08 ± 12%      -0.2        2.86 ±  9%      -0.3        2.76 ±  8%  perf-profile.children.cycles-pp.update_process_times
      3.69 ± 12%      -0.2        3.48 ±  9%      -0.4        3.31 ±  7%  perf-profile.children.cycles-pp.tick_sched_timer
      5.80 ± 13%      -0.2        5.61 ± 12%      -0.9        4.91 ±  7%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      2.29 ± 12%      -0.2        2.12 ±  8%      -0.2        2.13 ±  7%  perf-profile.children.cycles-pp.irq_exit_rcu
      3.30 ± 14%      -0.1        3.15 ± 11%      -0.5        2.85 ±  7%  perf-profile.children.cycles-pp.tick_sched_handle
      4.14 ± 15%      -0.1        3.99 ± 13%      +0.8        4.92 ± 11%  perf-profile.children.cycles-pp.write
      0.49 ± 34%      -0.1        0.35 ± 55%      +0.1        0.61 ± 23%  perf-profile.children.cycles-pp.calc_global_load_tick
      1.82 ± 12%      -0.1        1.69 ±  8%      -0.1        1.72 ±  7%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.73 ± 13%      -0.1        0.62 ± 19%      +0.1        0.79 ± 15%  perf-profile.children.cycles-pp.read_tsc
      0.98 ± 22%      -0.1        0.88 ± 20%      +0.2        1.13 ±  6%  perf-profile.children.cycles-pp.lapic_next_deadline
      1.65 ± 10%      -0.1        1.56 ±  9%      -0.1        1.56 ± 10%  perf-profile.children.cycles-pp.scheduler_tick
      0.90 ± 28%      -0.1        0.82 ± 30%      +0.2        1.05 ± 11%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.62 ± 38%      -0.1        0.55 ± 41%      +0.1        0.71 ± 32%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      1.02 ± 12%      -0.1        0.95 ±  9%      -0.0        1.01 ±  8%  perf-profile.children.cycles-pp.rebalance_domains
     60.60 ±  9%      -0.0       60.55 ±  7%      -6.5       54.06 ±  5%  perf-profile.children.cycles-pp.cpuidle_enter
      0.30 ± 29%      -0.0        0.25 ± 20%      +0.0        0.31 ± 24%  perf-profile.children.cycles-pp.tsc_verify_tsc_adjust
     60.57 ±  9%      -0.0       60.53 ±  7%      -6.6       54.02 ±  5%  perf-profile.children.cycles-pp.cpuidle_enter_state
      0.60 ±  8%      -0.0        0.56 ± 13%      -0.0        0.57 ±  8%  perf-profile.children.cycles-pp.irqtime_account_irq
      0.30 ± 28%      -0.0        0.26 ± 19%      +0.0        0.32 ± 23%  perf-profile.children.cycles-pp.arch_cpu_idle_enter
      0.29 ± 45%      -0.0        0.25 ± 18%      +0.0        0.34 ± 19%  perf-profile.children.cycles-pp._raw_spin_trylock
      0.03 ±282%      -0.0        0.00            -0.0        0.00        perf-profile.children.cycles-pp.__mod_lruvec_page_state
      0.66 ±  7%      -0.0        0.63 ± 11%      -0.0        0.64 ± 10%  perf-profile.children.cycles-pp.load_balance
      0.03 ±282%      -0.0        0.00            -0.0        0.00        perf-profile.children.cycles-pp.page_add_new_anon_rmap
      0.04 ± 75%      -0.0        0.02 ±173%      -0.0        0.02 ±141%  perf-profile.children.cycles-pp.sched_setaffinity
      0.03 ±282%      -0.0        0.00            -0.0        0.00        perf-profile.children.cycles-pp.__mod_lruvec_state
      0.03 ±282%      -0.0        0.00            -0.0        0.00        perf-profile.children.cycles-pp.__mod_node_page_state
      0.43 ± 10%      -0.0        0.40 ± 13%      +0.0        0.44 ± 12%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.17 ± 59%      -0.0        0.14 ± 41%      +0.0        0.19 ± 61%  perf-profile.children.cycles-pp.asm_exc_page_fault
      0.03 ±282%      -0.0        0.01 ±223%      -0.0        0.00        perf-profile.children.cycles-pp.wp_page_copy
      0.15 ± 64%      -0.0        0.13 ± 40%      +0.0        0.17 ± 68%  perf-profile.children.cycles-pp.do_user_addr_fault
      0.05 ± 57%      -0.0        0.02 ±119%      -0.0        0.03 ±117%  perf-profile.children.cycles-pp.schedule
      0.13 ± 75%      -0.0        0.11 ± 55%      +0.0        0.15 ± 69%  perf-profile.children.cycles-pp.__handle_mm_fault
      0.10 ± 52%      -0.0        0.08 ± 20%      +0.0        0.12 ± 71%  perf-profile.children.cycles-pp.mmput
      0.10 ± 52%      -0.0        0.08 ± 20%      +0.0        0.12 ± 71%  perf-profile.children.cycles-pp.exit_mmap
      0.15 ± 64%      -0.0        0.13 ± 40%      +0.0        0.17 ± 67%  perf-profile.children.cycles-pp.exc_page_fault
      0.03 ± 92%      -0.0        0.01 ±174%      -0.0        0.03 ±113%  perf-profile.children.cycles-pp.smpboot_thread_fn
      0.24 ± 30%      -0.0        0.22 ± 34%      -0.1        0.16 ± 10%  perf-profile.children.cycles-pp.cpuidle_governor_latency_req
      0.09 ± 24%      -0.0        0.07 ± 41%      -0.0        0.08 ± 49%  perf-profile.children.cycles-pp.__schedule
      0.14 ± 22%      -0.0        0.12 ± 36%      +0.0        0.14 ± 29%  perf-profile.children.cycles-pp.trigger_load_balance
      0.14 ± 69%      -0.0        0.12 ± 47%      +0.0        0.16 ± 65%  perf-profile.children.cycles-pp.handle_mm_fault
      0.37 ± 12%      -0.0        0.35 ± 17%      +0.0        0.38 ± 16%  perf-profile.children.cycles-pp._raw_spin_lock
      0.10 ± 21%      -0.0        0.08 ± 38%      -0.0        0.10 ± 23%  perf-profile.children.cycles-pp.idle_cpu
      0.95 ± 13%      -0.0        0.93 ± 17%      -0.1        0.87 ± 13%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.12 ± 49%      -0.0        0.10 ± 39%      -0.1        0.05 ± 56%  perf-profile.children.cycles-pp.irq_work_needs_cpu
      0.11 ± 21%      -0.0        0.09 ± 29%      +0.0        0.16 ± 64%  perf-profile.children.cycles-pp.bprm_execve
      0.37 ± 15%      -0.0        0.36 ± 18%      -0.0        0.37 ± 12%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.03 ±114%      -0.0        0.01 ±173%      -0.0        0.00        perf-profile.children.cycles-pp.__intel_pmu_disable_all
      0.44 ± 10%      -0.0        0.42 ± 13%      -0.1        0.37 ± 12%  perf-profile.children.cycles-pp.hrtimer_next_event_without
      0.08 ± 19%      -0.0        0.06 ± 37%      -0.0        0.06 ± 80%  perf-profile.children.cycles-pp.process_interval
      0.03 ±184%      -0.0        0.02 ±142%      -0.0        0.02 ±187%  perf-profile.children.cycles-pp.unmap_page_range
      0.17 ± 21%      -0.0        0.16 ± 22%      -0.0        0.14 ± 16%  perf-profile.children.cycles-pp.error_entry
      0.04 ± 60%      -0.0        0.03 ±101%      +0.0        0.05 ± 59%  perf-profile.children.cycles-pp.do_filp_open
      0.08 ± 20%      -0.0        0.06 ± 37%      -0.0        0.06 ± 65%  perf-profile.children.cycles-pp.__libc_start_main
      0.08 ± 20%      -0.0        0.06 ± 37%      -0.0        0.06 ± 65%  perf-profile.children.cycles-pp.run_builtin
      0.14 ± 22%      -0.0        0.13 ± 23%      +0.0        0.18 ± 57%  perf-profile.children.cycles-pp.do_execveat_common
      0.08 ± 19%      -0.0        0.06 ± 37%      -0.0        0.06 ± 80%  perf-profile.children.cycles-pp.cmd_stat
      0.08 ± 19%      -0.0        0.06 ± 37%      -0.0        0.06 ± 80%  perf-profile.children.cycles-pp.dispatch_events
      0.04 ± 91%      -0.0        0.02 ±118%      +0.1        0.09 ± 99%  perf-profile.children.cycles-pp.begin_new_exec
      0.09 ± 22%      -0.0        0.08 ± 30%      +0.0        0.14 ± 71%  perf-profile.children.cycles-pp.load_elf_binary
      0.05 ± 58%      -0.0        0.03 ±108%      -0.0        0.01 ±187%  perf-profile.children.cycles-pp.update_dl_rq_load_avg
      0.04 ± 57%      -0.0        0.03 ±101%      +0.0        0.05 ± 59%  perf-profile.children.cycles-pp.path_openat
      0.14 ± 22%      -0.0        0.13 ± 23%      +0.0        0.18 ± 56%  perf-profile.children.cycles-pp.__x64_sys_execve
      0.10 ± 22%      -0.0        0.08 ± 30%      +0.0        0.14 ± 71%  perf-profile.children.cycles-pp.exec_binprm
      0.07 ± 95%      -0.0        0.05 ± 49%      -0.0        0.05 ± 57%  perf-profile.children.cycles-pp.__x64_sys_exit_group
      0.07 ± 95%      -0.0        0.05 ± 49%      -0.0        0.05 ± 57%  perf-profile.children.cycles-pp.do_group_exit
      0.07 ± 95%      -0.0        0.05 ± 49%      -0.0        0.05 ± 57%  perf-profile.children.cycles-pp.do_exit
      0.46 ±  9%      -0.0        0.45 ± 14%      +0.0        0.46 ± 13%  perf-profile.children.cycles-pp.find_busiest_group
      0.08 ± 65%      -0.0        0.06 ± 91%      +0.0        0.10 ± 75%  perf-profile.children.cycles-pp.note_gp_changes
      0.07 ± 22%      -0.0        0.06 ± 37%      -0.0        0.05 ± 79%  perf-profile.children.cycles-pp.read_counters
      0.02 ±141%      -0.0        0.00 ±331%      -0.0        0.01 ±190%  perf-profile.children.cycles-pp.setlocale
      0.40 ± 23%      -0.0        0.39 ± 26%      -0.1        0.30 ± 17%  perf-profile.children.cycles-pp.rcu_idle_exit
      0.03 ±184%      -0.0        0.02 ±142%      -0.0        0.02 ±147%  perf-profile.children.cycles-pp.unmap_vmas
      0.09 ± 46%      -0.0        0.08 ± 40%      +0.0        0.09 ± 47%  perf-profile.children.cycles-pp.rcu_dynticks_eqs_enter
      0.02 ±282%      -0.0        0.01 ±223%      -0.0        0.02 ±187%  perf-profile.children.cycles-pp.zap_pte_range
      0.14 ± 21%      -0.0        0.13 ± 23%      +0.0        0.18 ± 57%  perf-profile.children.cycles-pp.execve
      0.04 ± 75%      -0.0        0.03 ±101%      +0.0        0.08 ± 22%  perf-profile.children.cycles-pp.rcu_read_unlock_strict
      0.01 ±282%      -0.0        0.00            -0.0        0.00        perf-profile.children.cycles-pp.asm_common_interrupt
      0.01 ±282%      -0.0        0.00            -0.0        0.00        perf-profile.children.cycles-pp.common_interrupt
      0.01 ±187%      -0.0        0.00            -0.0        0.00        perf-profile.children.cycles-pp.hrtimer_run_queues
      0.01 ±187%      -0.0        0.00            +0.0        0.01 ±188%  perf-profile.children.cycles-pp.try_to_wake_up
      0.32 ± 18%      -0.0        0.31 ± 21%      +0.0        0.36 ±  8%  perf-profile.children.cycles-pp.__intel_pmu_enable_all
      0.08 ± 62%      -0.0        0.07 ± 59%      -0.0        0.04 ± 94%  perf-profile.children.cycles-pp.x86_pmu_disable
      0.44 ± 19%      -0.0        0.44 ± 18%      -0.1        0.34 ± 13%  perf-profile.children.cycles-pp.__hrtimer_next_event_base
      0.01 ±190%      -0.0        0.01 ±331%      -0.0        0.00        perf-profile.children.cycles-pp.rcu_gp_kthread
      0.01 ±188%      -0.0        0.00 ±331%      +0.0        0.02 ±142%  perf-profile.children.cycles-pp.balance_fair
      0.01 ±188%      -0.0        0.00 ±331%      +0.0        0.02 ±147%  perf-profile.children.cycles-pp.newidle_balance
      0.10 ± 87%      -0.0        0.09 ± 80%      -0.1        0.03 ±113%  perf-profile.children.cycles-pp.irq_work_tick
      0.09 ± 29%      -0.0        0.09 ± 52%      -0.0        0.05 ± 56%  perf-profile.children.cycles-pp.menu_reflect
      0.15 ± 14%      -0.0        0.15 ± 17%      +0.0        0.16 ± 19%  perf-profile.children.cycles-pp.hrtimer_get_next_event
      0.01 ±187%      -0.0        0.00 ±331%      +0.0        0.02 ±112%  perf-profile.children.cycles-pp.can_stop_idle_tick
      0.05 ± 97%      -0.0        0.05 ± 76%      -0.0        0.02 ±142%  perf-profile.children.cycles-pp.do_nocb_deferred_wakeup
      0.65 ± 15%      -0.0        0.65 ± 17%      +0.1        0.75 ±  8%  perf-profile.children.cycles-pp.native_sched_clock
      0.01 ±282%      -0.0        0.00            -0.0        0.00        perf-profile.children.cycles-pp.net_rx_action
      0.01 ±282%      -0.0        0.00            -0.0        0.00        perf-profile.children.cycles-pp.__napi_poll
      0.01 ±282%      -0.0        0.00            -0.0        0.00        perf-profile.children.cycles-pp.i40e_napi_poll
      0.12 ± 19%      -0.0        0.12 ± 23%      +0.0        0.12 ± 28%  perf-profile.children.cycles-pp.main
      0.13 ± 38%      -0.0        0.13 ± 59%      +0.0        0.15 ± 54%  perf-profile.children.cycles-pp.rcu_core
      0.01 ±282%      -0.0        0.00            -0.0        0.00        perf-profile.children.cycles-pp.force_qs_rnp
      0.01 ±282%      -0.0        0.00            -0.0        0.00        perf-profile.children.cycles-pp.timerqueue_iterate_next
      0.01 ±282%      -0.0        0.00            -0.0        0.00        perf-profile.children.cycles-pp._dl_addr
      0.01 ±282%      -0.0        0.00            -0.0        0.00        perf-profile.children.cycles-pp.enqueue_task_fair
      0.01 ±282%      -0.0        0.00            -0.0        0.00        perf-profile.children.cycles-pp.copy_strings
      0.01 ±282%      -0.0        0.00            -0.0        0.00        perf-profile.children.cycles-pp.pick_next_task_fair
      0.01 ±282%      -0.0        0.00            +0.0        0.01 ±282%  perf-profile.children.cycles-pp.tlb_finish_mmu
      0.01 ±282%      -0.0        0.00            +0.0        0.01 ±282%  perf-profile.children.cycles-pp.tlb_flush_mmu
      0.09 ± 23%      -0.0        0.09 ± 28%      +0.0        0.10 ± 22%  perf-profile.children.cycles-pp.apparmor_file_permission
      0.07 ± 86%      -0.0        0.06 ± 56%      +0.0        0.11 ± 37%  perf-profile.children.cycles-pp.tick_sched_do_timer
      0.05 ±105%      -0.0        0.04 ± 96%      +0.0        0.08 ± 66%  perf-profile.children.cycles-pp.timekeeping_advance
      0.07 ± 53%      -0.0        0.07 ± 65%      -0.0        0.07 ± 44%  perf-profile.children.cycles-pp.get_cpu_device
      0.05 ± 72%      -0.0        0.05 ±108%      +0.0        0.07 ± 41%  perf-profile.children.cycles-pp.do_fault
      0.04 ± 78%      -0.0        0.04 ± 89%      +0.0        0.06 ± 66%  perf-profile.children.cycles-pp.do_sys_open
      0.01 ±282%      -0.0        0.00 ±331%      -0.0        0.00        perf-profile.children.cycles-pp.ksoftirqd_running
      0.11 ± 21%      -0.0        0.10 ± 23%      -0.0        0.11 ± 14%  perf-profile.children.cycles-pp._find_next_bit
      0.20 ± 22%      -0.0        0.20 ± 18%      +0.0        0.22 ± 13%  perf-profile.children.cycles-pp.rw_verify_area
      0.04 ± 79%      -0.0        0.04 ± 89%      +0.0        0.06 ± 66%  perf-profile.children.cycles-pp.do_sys_openat2
      0.24 ± 33%      -0.0        0.23 ± 39%      -0.1        0.14 ± 32%  perf-profile.children.cycles-pp.ret_from_fork
      0.01 ±188%      -0.0        0.01 ±223%      +0.0        0.03 ± 93%  perf-profile.children.cycles-pp.link_path_walk
      0.07 ± 68%      -0.0        0.06 ± 51%      -0.0        0.02 ±192%  perf-profile.children.cycles-pp.rcu_eqs_enter
      0.03 ±112%      -0.0        0.03 ±123%      +0.0        0.03 ± 90%  perf-profile.children.cycles-pp.vm_mmap_pgoff
      0.05 ± 94%      -0.0        0.05 ± 74%      -0.1        0.00        perf-profile.children.cycles-pp.intel_pmu_disable_all
      0.01 ±282%      -0.0        0.00 ±331%      -0.0        0.00        perf-profile.children.cycles-pp.timer_clear_idle
      0.01 ±282%      -0.0        0.00 ±331%      +0.0        0.01 ±282%  perf-profile.children.cycles-pp.tick_nohz_idle_retain_tick
      0.01 ±282%      -0.0        0.00 ±331%      +0.0        0.01 ±282%  perf-profile.children.cycles-pp.schedule_idle
      0.23 ± 35%      -0.0        0.23 ± 40%      -0.1        0.13 ± 33%  perf-profile.children.cycles-pp.kthread
      0.01 ±187%      -0.0        0.01 ±227%      +0.0        0.02 ±142%  perf-profile.children.cycles-pp.ksys_mmap_pgoff
      0.01 ±187%      -0.0        0.01 ±227%      +0.0        0.02 ±144%  perf-profile.children.cycles-pp.update_group_capacity
      0.01 ±282%      -0.0        0.01 ±331%      +0.0        0.01 ±187%  perf-profile.children.cycles-pp.next_uptodate_page
      0.01 ±282%      -0.0        0.01 ±331%      -0.0        0.00        perf-profile.children.cycles-pp.__note_gp_changes
      0.01 ±282%      -0.0        0.01 ±331%      -0.0        0.00        perf-profile.children.cycles-pp.tick_nohz_idle_got_tick
      0.03 ±113%      -0.0        0.02 ±122%      +0.0        0.03 ± 91%  perf-profile.children.cycles-pp.do_mmap
      0.01 ±187%      -0.0        0.01 ±230%      -0.0        0.00        perf-profile.children.cycles-pp.perf_ctx_unlock
      0.25 ± 24%      -0.0        0.25 ± 26%      -0.0        0.20 ± 23%  perf-profile.children.cycles-pp.rcu_eqs_exit
      0.00            +0.0        0.00            +0.0        0.01 ±282%  perf-profile.children.cycles-pp.read@plt
      0.00            +0.0        0.00            +0.0        0.01 ±282%  perf-profile.children.cycles-pp.page_remove_rmap
      0.00            +0.0        0.00            +0.0        0.01 ±282%  perf-profile.children.cycles-pp.raw_spin_rq_lock_nested
      0.00            +0.0        0.00            +0.0        0.01 ±282%  perf-profile.children.cycles-pp.asm_sysvec_call_function_single
      0.00            +0.0        0.00            +0.0        0.01 ±282%  perf-profile.children.cycles-pp.__lookup_slow
      0.00            +0.0        0.00            +0.0        0.01 ±282%  perf-profile.children.cycles-pp.rcu_do_batch
      0.00            +0.0        0.00            +0.0        0.01 ±282%  perf-profile.children.cycles-pp.timekeeping_update
      0.00            +0.0        0.00            +0.0        0.01 ±282%  perf-profile.children.cycles-pp.cpuidle_select
      0.00            +0.0        0.00            +0.0        0.01 ±282%  perf-profile.children.cycles-pp.path_lookupat
      0.00            +0.0        0.00            +0.0        0.01 ±282%  perf-profile.children.cycles-pp.release_pages
      0.00            +0.0        0.00            +0.0        0.01 ±282%  perf-profile.children.cycles-pp.filename_lookup
      0.00            +0.0        0.00            +0.0        0.03 ±282%  perf-profile.children.cycles-pp.unlink_anon_vmas
      0.00            +0.0        0.00            +0.0        0.03 ±282%  perf-profile.children.cycles-pp.free_pgtables
      0.01 ±282%      +0.0        0.01 ±224%      -0.0        0.00        perf-profile.children.cycles-pp.arch_cpu_idle_exit
      0.12 ± 15%      +0.0        0.12 ± 20%      +0.0        0.12 ± 15%  perf-profile.children.cycles-pp.cpumask_next_and
      0.02 ±197%      +0.0        0.02 ±146%      +0.0        0.04 ± 74%  perf-profile.children.cycles-pp.profile_tick
      0.02 ±142%      +0.0        0.02 ±145%      +0.0        0.02 ±143%  perf-profile.children.cycles-pp.mmap_region
      0.06 ± 95%      +0.0        0.06 ± 75%      -0.0        0.02 ±142%  perf-profile.children.cycles-pp.tick_check_broadcast_expired
      0.12 ± 24%      +0.0        0.12 ± 24%      -0.0        0.09 ± 36%  perf-profile.children.cycles-pp.tick_nohz_tick_stopped
      0.06 ± 45%      +0.0        0.06 ± 65%      +0.0        0.07 ± 42%  perf-profile.children.cycles-pp.irqentry_exit
      0.04 ± 91%      +0.0        0.04 ±106%      +0.0        0.06 ± 42%  perf-profile.children.cycles-pp.filemap_map_pages
      0.77 ± 13%      +0.0        0.77 ± 15%      +0.1        0.88 ±  7%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.32 ± 12%      +0.0        0.32 ± 16%      +0.0        0.36 ± 16%  perf-profile.children.cycles-pp.arch_scale_freq_tick
      0.09 ± 63%      +0.0        0.10 ± 44%      -0.1        0.03 ±114%  perf-profile.children.cycles-pp.irqtime_account_process_tick
      0.14 ± 66%      +0.0        0.14 ± 49%      -0.1        0.03 ±112%  perf-profile.children.cycles-pp.native_apic_mem_write
      0.01 ±187%      +0.0        0.02 ±177%      -0.0        0.01 ±282%  perf-profile.children.cycles-pp.account_process_tick
      0.02 ±114%      +0.0        0.03 ±122%      -0.0        0.00        perf-profile.children.cycles-pp.rcu_needs_cpu
      0.01 ±282%      +0.0        0.01 ±224%      +0.0        0.01 ±282%  perf-profile.children.cycles-pp.sched_clock_idle_wakeup_event
      0.01 ±282%      +0.0        0.01 ±224%      +0.0        0.02 ±145%  perf-profile.children.cycles-pp.walk_component
      0.05 ± 92%      +0.0        0.05 ± 73%      -0.0        0.01 ±282%  perf-profile.children.cycles-pp.restore_regs_and_return_to_kernel
      0.00            +0.0        0.00 ±331%      +0.0        0.00        perf-profile.children.cycles-pp.cpu_latency_qos_limit
      0.00            +0.0        0.00 ±331%      +0.0        0.00        perf-profile.children.cycles-pp.rcu_segcblist_ready_cbs
      0.00            +0.0        0.00 ±331%      +0.0        0.00        perf-profile.children.cycles-pp.__libc_read
      0.00            +0.0        0.00 ±331%      +0.0        0.03 ±282%  perf-profile.children.cycles-pp.do_anonymous_page
      0.02 ±143%      +0.0        0.02 ±145%      -0.0        0.01 ±187%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.01 ±282%      +0.0        0.01 ±223%      +0.0        0.02 ±111%  perf-profile.children.cycles-pp.dup_mm
      0.17 ± 40%      +0.0        0.18 ± 18%      -0.1        0.10 ± 17%  perf-profile.children.cycles-pp.rb_next
      0.00            +0.0        0.01 ±331%      +0.0        0.00        perf-profile.children.cycles-pp.__alloc_pages
      0.28 ± 27%      +0.0        0.29 ± 18%      -0.0        0.24 ± 13%  perf-profile.children.cycles-pp.update_irq_load_avg
      0.00            +0.0        0.01 ±331%      +0.0        0.00        perf-profile.children.cycles-pp.cpuidle_get_cpu_driver
      0.03 ± 90%      +0.0        0.04 ± 74%      -0.0        0.01 ±282%  perf-profile.children.cycles-pp.hrtimer_forward
      0.45 ± 29%      +0.0        0.46 ± 39%      -0.1        0.34 ± 19%  perf-profile.children.cycles-pp.rcu_sched_clock_irq
      0.05 ± 77%      +0.0        0.06 ± 49%      +0.0        0.07 ± 63%  perf-profile.children.cycles-pp.__x64_sys_write
      0.04 ± 92%      +0.0        0.05 ± 95%      -0.0        0.01 ±282%  perf-profile.children.cycles-pp.pm_qos_read_value
      0.03 ±114%      +0.0        0.04 ± 86%      +0.0        0.04 ± 72%  perf-profile.children.cycles-pp.kill_fasync
      0.01 ±187%      +0.0        0.02 ±142%      -0.0        0.00        perf-profile.children.cycles-pp.rcu_nocb_flush_deferred_wakeup
      0.00            +0.0        0.01 ±223%      +0.0        0.00        perf-profile.children.cycles-pp.io_serial_out
      0.00            +0.0        0.01 ±223%      +0.0        0.00        perf-profile.children.cycles-pp.error_return
      0.02 ±143%      +0.0        0.03 ±121%      -0.0        0.00        perf-profile.children.cycles-pp.leave_mm
      0.01 ±187%      +0.0        0.02 ±142%      +0.0        0.02 ±143%  perf-profile.children.cycles-pp.sched_clock
      0.00            +0.0        0.01 ±223%      +0.0        0.02 ±141%  perf-profile.children.cycles-pp.dup_mmap
      0.11 ± 56%      +0.0        0.12 ± 47%      -0.1        0.04 ± 72%  perf-profile.children.cycles-pp.rb_insert_color
      0.26 ± 17%      +0.0        0.27 ± 12%      +0.0        0.29 ± 12%  perf-profile.children.cycles-pp.rcu_all_qs
      0.01 ±188%      +0.0        0.02 ±119%      -0.0        0.00        perf-profile.children.cycles-pp.rcu_is_cpu_rrupt_from_idle
      0.06 ± 41%      +0.0        0.07 ± 41%      +0.0        0.08 ± 25%  perf-profile.children.cycles-pp.__libc_fork
      0.11 ± 26%      +0.0        0.13 ± 28%      -0.0        0.09 ± 45%  perf-profile.children.cycles-pp.rcu_dynticks_eqs_exit
      0.27 ± 26%      +0.0        0.28 ± 23%      +0.1        0.34 ± 18%  perf-profile.children.cycles-pp.make_kuid
      0.35 ± 22%      +0.0        0.37 ± 19%      -0.1        0.29 ±  8%  perf-profile.children.cycles-pp.run_rebalance_domains
      0.58 ± 21%      +0.0        0.60 ± 14%      +0.0        0.60 ± 13%  perf-profile.children.cycles-pp.__might_fault
      0.30 ± 32%      +0.0        0.32 ± 35%      -0.1        0.16 ± 15%  perf-profile.children.cycles-pp.enqueue_hrtimer
      0.26 ± 16%      +0.0        0.27 ± 15%      +0.0        0.26 ± 18%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.27 ± 34%      +0.0        0.29 ± 36%      -0.1        0.14 ± 18%  perf-profile.children.cycles-pp.timerqueue_add
      0.16 ± 27%      +0.0        0.18 ± 23%      +0.1        0.21 ± 18%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare
      0.43 ± 17%      +0.0        0.45 ± 14%      -0.0        0.39 ± 13%  perf-profile.children.cycles-pp.update_rq_clock
      0.03 ±113%      +0.0        0.04 ± 74%      +0.0        0.04 ± 92%  perf-profile.children.cycles-pp.rcu_nmi_exit
      0.33 ± 23%      +0.0        0.35 ± 19%      -0.0        0.28 ± 10%  perf-profile.children.cycles-pp.update_blocked_averages
      0.03 ±120%      +0.0        0.05 ± 87%      -0.0        0.00        perf-profile.children.cycles-pp.check_tsc_unstable
      0.03 ±114%      +0.0        0.05 ± 72%      -0.0        0.00        perf-profile.children.cycles-pp.rcu_irq_enter
      0.13 ± 19%      +0.0        0.15 ± 18%      +0.0        0.16 ± 19%  perf-profile.children.cycles-pp.call_cpuidle
      0.42 ± 29%      +0.0        0.44 ± 23%      -0.1        0.31 ± 16%  perf-profile.children.cycles-pp.hrtimer_update_next_event
      0.05 ± 89%      +0.0        0.07 ± 80%      -0.0        0.01 ±187%  perf-profile.children.cycles-pp.cpuidle_not_available
      0.02 ±145%      +0.0        0.04 ± 88%      -0.0        0.00        perf-profile.children.cycles-pp.cpuidle_reflect
      0.02 ±114%      +0.0        0.04 ± 73%      -0.0        0.00        perf-profile.children.cycles-pp.task_tick_idle
      0.13 ± 70%      +0.0        0.15 ± 66%      -0.1        0.03 ± 90%  perf-profile.children.cycles-pp.drm_fb_helper_damage_work
      0.05 ± 76%      +0.0        0.07 ± 39%      +0.0        0.05 ± 57%  perf-profile.children.cycles-pp.update_rt_rq_load_avg
      0.03 ±118%      +0.0        0.05 ± 47%      +0.0        0.04 ± 92%  perf-profile.children.cycles-pp.__wake_up_sync_key
      0.00            +0.0        0.02 ±206%      +0.0        0.02 ±141%  perf-profile.children.cycles-pp.poll_idle
      0.13 ± 70%      +0.0        0.15 ± 66%      -0.1        0.03 ± 90%  perf-profile.children.cycles-pp.memcpy_toio
      0.13 ± 71%      +0.0        0.15 ± 65%      -0.1        0.03 ± 91%  perf-profile.children.cycles-pp.irqentry_enter
      0.13 ± 55%      +0.0        0.15 ± 46%      -0.1        0.02 ±197%  perf-profile.children.cycles-pp.perf_event_task_tick
      0.20 ± 35%      +0.0        0.23 ± 27%      -0.1        0.15 ± 20%  perf-profile.children.cycles-pp.__hrtimer_get_next_event
      0.84 ± 23%      +0.0        0.86 ± 17%      +0.2        1.04 ± 11%  perf-profile.children.cycles-pp.__might_sleep
      0.74 ± 21%      +0.0        0.76 ± 21%      +0.2        0.93 ± 13%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      0.23 ± 31%      +0.0        0.26 ± 24%      +0.1        0.32 ± 24%  perf-profile.children.cycles-pp.ktime_get_coarse_real_ts64
      0.09 ± 23%      +0.0        0.12 ± 37%      -0.0        0.08 ± 35%  perf-profile.children.cycles-pp.rb_erase
      0.16 ± 73%      +0.0        0.18 ± 48%      -0.1        0.05 ± 55%  perf-profile.children.cycles-pp.tick_check_oneshot_broadcast_this_cpu
      0.06 ±102%      +0.0        0.09 ± 66%      -0.0        0.01 ±187%  perf-profile.children.cycles-pp.rcu_nmi_enter
      0.01 ±188%      +0.0        0.04 ± 91%      +0.0        0.05 ± 56%  perf-profile.children.cycles-pp.copy_process
      0.41 ± 23%      +0.0        0.44 ± 18%      +0.1        0.53 ± 13%  perf-profile.children.cycles-pp.map_id_range_down
      0.11 ± 79%      +0.0        0.14 ± 52%      -0.1        0.03 ± 91%  perf-profile.children.cycles-pp.delay_tsc
      0.46 ± 26%      +0.0        0.49 ± 17%      +0.2        0.65 ± 12%  perf-profile.children.cycles-pp.aa_file_perm
      0.13 ± 29%      +0.0        0.16 ± 44%      +0.0        0.16 ± 22%  perf-profile.children.cycles-pp.iov_iter_init
      0.03 ±113%      +0.0        0.06 ± 51%      +0.0        0.06 ± 43%  perf-profile.children.cycles-pp.__do_sys_clone
      0.03 ±113%      +0.0        0.06 ± 51%      +0.0        0.06 ± 43%  perf-profile.children.cycles-pp.kernel_clone
      0.15 ± 61%      +0.0        0.18 ± 46%      -0.1        0.06 ± 46%  perf-profile.children.cycles-pp.worker_thread
      0.01 ±282%      +0.0        0.04 ± 88%      -0.0        0.01 ±282%  perf-profile.children.cycles-pp.sched_idle_set_state
      0.02 ±114%      +0.0        0.06 ±206%      +0.0        0.03 ± 93%  perf-profile.children.cycles-pp.seq_read_iter
      0.00            +0.0        0.03 ±331%      +0.0        0.00        perf-profile.children.cycles-pp.node_read_vmstat
      0.28 ± 25%      +0.0        0.31 ± 19%      +0.1        0.35 ± 11%  perf-profile.children.cycles-pp.timestamp_truncate
      0.00            +0.0        0.03 ±331%      +0.0        0.00        perf-profile.children.cycles-pp.sysfs_kf_seq_show
      0.00            +0.0        0.03 ±331%      +0.0        0.00        perf-profile.children.cycles-pp.dev_attr_show
      0.14 ± 68%      +0.0        0.17 ± 46%      -0.1        0.06 ± 42%  perf-profile.children.cycles-pp.process_one_work
      0.00            +0.0        0.03 ±331%      +0.0        0.01 ±282%  perf-profile.children.cycles-pp.fold_vm_numa_events
      0.05 ± 76%      +0.0        0.08 ± 26%      +0.0        0.06 ± 58%  perf-profile.children.cycles-pp.tick_program_event
      0.30 ± 30%      +0.0        0.34 ± 33%      -0.1        0.19 ± 21%  perf-profile.children.cycles-pp.timerqueue_del
      0.17 ± 55%      +0.0        0.20 ± 40%      -0.1        0.10 ± 29%  perf-profile.children.cycles-pp.nr_iowait_cpu
      0.18 ± 76%      +0.0        0.22 ± 48%      -0.1        0.06 ± 40%  perf-profile.children.cycles-pp.update_ts_time_stats
      1.06 ± 13%      +0.0        1.10 ± 13%      +0.2        1.30 ±  6%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.02 ±112%      +0.0        0.06 ± 18%      +0.0        0.06 ± 45%  perf-profile.children.cycles-pp.__x64_sys_read
      0.38 ± 34%      +0.0        0.42 ± 35%      -0.2        0.23 ± 19%  perf-profile.children.cycles-pp.__remove_hrtimer
      0.52 ± 14%      +0.0        0.56 ± 14%      +0.1        0.62 ± 11%  perf-profile.children.cycles-pp.__cond_resched
      0.98 ± 25%      +0.0        1.02 ± 15%      -0.2        0.76 ± 34%  perf-profile.children.cycles-pp.irq_enter_rcu
      0.21 ± 27%      +0.0        0.25 ± 19%      +0.1        0.28 ± 13%  perf-profile.children.cycles-pp.__wake_up_common
      0.30 ± 18%      +0.0        0.35 ± 18%      +0.1        0.36 ± 10%  perf-profile.children.cycles-pp.syscall_enter_from_user_mode
      0.23 ± 24%      +0.0        0.28 ± 19%      +0.1        0.34 ± 15%  perf-profile.children.cycles-pp.make_kgid
      0.93 ± 26%      +0.0        0.98 ± 17%      -0.2        0.70 ± 38%  perf-profile.children.cycles-pp.tick_irq_enter
      0.83 ± 22%      +0.1        0.88 ± 15%      +0.2        1.01 ± 11%  perf-profile.children.cycles-pp.___might_sleep
      0.46 ± 65%      +0.1        0.52 ± 48%      -0.3        0.16 ± 23%  perf-profile.children.cycles-pp.io_serial_in
      1.12 ± 19%      +0.1        1.18 ± 17%      +0.3        1.43 ± 13%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      0.17 ± 69%      +0.1        0.24 ± 52%      -0.1        0.05 ± 73%  perf-profile.children.cycles-pp.run_posix_cpu_timers
      0.80 ± 22%      +0.1        0.87 ± 16%      +0.2        1.02 ± 12%  perf-profile.children.cycles-pp.current_time
      1.01 ± 19%      +0.1        1.08 ± 15%      +0.2        1.18 ±  8%  perf-profile.children.cycles-pp.mutex_unlock
      0.44 ± 21%      +0.1        0.51 ± 18%      +0.2        0.62 ± 20%  perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
     26.91 ± 18%      +0.1       26.99 ± 13%      +4.9       31.82 ±  8%  perf-profile.children.cycles-pp.do_syscall_64
      0.56 ± 66%      +0.1        0.65 ± 48%      -0.4        0.21 ± 14%  perf-profile.children.cycles-pp.serial8250_console_putchar
      0.58 ± 66%      +0.1        0.66 ± 47%      -0.4        0.21 ± 14%  perf-profile.children.cycles-pp.wait_for_xmitr
      0.62 ± 22%      +0.1        0.72 ± 19%      +0.2        0.83 ± 11%  perf-profile.children.cycles-pp.file_update_time
      0.58 ± 66%      +0.1        0.67 ± 47%      -0.4        0.21 ± 13%  perf-profile.children.cycles-pp.uart_console_write
      0.60 ± 65%      +0.1        0.69 ± 47%      -0.4        0.22 ± 13%  perf-profile.children.cycles-pp.serial8250_console_write
      0.62 ± 65%      +0.1        0.72 ± 46%      -0.4        0.23 ± 12%  perf-profile.children.cycles-pp.irq_work_run
      0.62 ± 65%      +0.1        0.72 ± 46%      -0.4        0.23 ± 12%  perf-profile.children.cycles-pp.printk
      0.62 ± 65%      +0.1        0.72 ± 46%      -0.4        0.23 ± 12%  perf-profile.children.cycles-pp.vprintk_emit
      0.62 ± 65%      +0.1        0.72 ± 46%      -0.4        0.23 ± 12%  perf-profile.children.cycles-pp.console_unlock
      0.62 ± 65%      +0.1        0.72 ± 46%      -0.4        0.23 ± 12%  perf-profile.children.cycles-pp.asm_sysvec_irq_work
      0.62 ± 65%      +0.1        0.72 ± 46%      -0.4        0.23 ± 12%  perf-profile.children.cycles-pp.sysvec_irq_work
      0.62 ± 65%      +0.1        0.72 ± 46%      -0.4        0.23 ± 12%  perf-profile.children.cycles-pp.__sysvec_irq_work
      0.62 ± 65%      +0.1        0.72 ± 46%      -0.4        0.23 ± 12%  perf-profile.children.cycles-pp.irq_work_single
      1.10 ± 19%      +0.1        1.20 ± 15%      +0.3        1.41 ± 12%  perf-profile.children.cycles-pp.atime_needs_update
      0.64 ± 65%      +0.1        0.74 ± 45%      -0.4        0.25 ± 12%  perf-profile.children.cycles-pp.irq_work_run_list
      1.19 ± 17%      +0.1        1.30 ± 14%      +0.3        1.45 ±  7%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      1.25 ± 19%      +0.1        1.39 ± 14%      +0.3        1.59 ± 11%  perf-profile.children.cycles-pp.touch_atime
      1.12 ± 21%      +0.1        1.26 ± 13%      +0.4        1.57 ± 14%  perf-profile.children.cycles-pp.common_file_perm
      2.06 ± 18%      +0.1        2.21 ± 15%      +0.5        2.60 ± 11%  perf-profile.children.cycles-pp.mutex_lock
      1.23 ± 16%      +0.2        1.39 ± 13%      +0.4        1.60 ±  6%  perf-profile.children.cycles-pp.__wake_up_common_lock
      1.67 ± 17%      +0.2        1.84 ± 13%      +0.4        2.04 ±  9%  perf-profile.children.cycles-pp.__fget_light
      1.76 ± 17%      +0.2        1.94 ± 16%      +0.5        2.21 ±  8%  perf-profile.children.cycles-pp.__entry_text_start
      1.94 ± 20%      +0.2        2.13 ± 14%      +0.5        2.43 ± 12%  perf-profile.children.cycles-pp.copyin
      1.95 ± 17%      +0.2        2.17 ± 14%      +0.5        2.41 ±  7%  perf-profile.children.cycles-pp.__fdget_pos
     27.76 ± 18%      +0.2       27.97 ± 13%      +5.1       32.91 ±  8%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
      2.05 ± 20%      +0.2        2.29 ± 14%      +0.4        2.43 ±  9%  perf-profile.children.cycles-pp.copyout
      3.22 ± 20%      +0.2        3.47 ± 12%      +0.8        4.02 ± 11%  perf-profile.children.cycles-pp.copy_page_from_iter
      8.26 ± 19%      +0.3        8.52 ± 14%      +1.7        9.92 ±  8%  perf-profile.children.cycles-pp.pipe_read
      3.27 ± 19%      +0.3        3.58 ± 12%      +0.6        3.84 ± 10%  perf-profile.children.cycles-pp.copy_page_to_iter
     10.87 ± 18%      +0.3       11.19 ± 13%      +2.4       13.31 ±  8%  perf-profile.children.cycles-pp.vfs_write
      2.39 ± 18%      +0.3        2.71 ± 15%      +0.6        3.01 ±  8%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      8.75 ± 19%      +0.4        9.11 ± 13%      +1.8       10.51 ±  8%  perf-profile.children.cycles-pp.new_sync_read
      3.80 ± 19%      +0.4        4.22 ± 14%      +0.8        4.61 ± 10%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
     12.12 ± 18%      +0.5       12.61 ± 13%      +2.7       14.84 ±  8%  perf-profile.children.cycles-pp.ksys_write
     40.83 ± 12%      +0.7       41.57 ±  9%      -5.8       35.07 ± 10%  perf-profile.children.cycles-pp.intel_idle
      8.46 ± 18%      +0.8        9.22 ± 13%      +2.4       10.82 ±  8%  perf-profile.children.cycles-pp.pipe_write
      8.99 ± 18%      +0.8        9.78 ± 13%      +2.5       11.46 ±  8%  perf-profile.children.cycles-pp.new_sync_write
      0.51 ± 32%      +0.9        1.42 ±205%      +0.0        0.54 ± 13%  perf-profile.children.cycles-pp.start_kernel
      1.41 ± 22%      -1.4        0.00            -1.4        0.00        perf-profile.self.cycles-pp.fsnotify
      2.41 ± 52%      -0.3        2.07 ± 70%      +0.8        3.26 ± 34%  perf-profile.self.cycles-pp.ktime_get
      0.95 ± 60%      -0.3        0.66 ± 90%      +0.5        1.47 ± 22%  perf-profile.self.cycles-pp.timekeeping_max_deferment
      0.57 ± 23%      -0.3        0.30 ± 22%      +0.2        0.73 ± 17%  perf-profile.self.cycles-pp.anon_pipe_buf_release
      2.15 ± 11%      -0.2        1.96 ± 22%      +0.0        2.15 ±  6%  perf-profile.self.cycles-pp.menu_select
      0.49 ± 35%      -0.1        0.34 ± 57%      +0.1        0.61 ± 23%  perf-profile.self.cycles-pp.calc_global_load_tick
      0.56 ± 40%      -0.1        0.42 ± 40%      +0.1        0.64 ± 19%  perf-profile.self.cycles-pp.tick_nohz_next_event
      0.98 ± 22%      -0.1        0.88 ± 20%      +0.1        1.13 ±  6%  perf-profile.self.cycles-pp.lapic_next_deadline
      0.71 ± 12%      -0.1        0.61 ± 18%      +0.1        0.76 ± 14%  perf-profile.self.cycles-pp.read_tsc
      0.60 ± 22%      -0.1        0.51 ± 11%      +0.1        0.70 ±  7%  perf-profile.self.cycles-pp.vfs_read
      0.55 ± 33%      -0.1        0.46 ± 33%      +0.0        0.57 ± 32%  perf-profile.self.cycles-pp.update_process_times
      2.44 ±  8%      -0.1        2.37 ± 11%      +0.0        2.48 ±  9%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.89 ± 28%      -0.1        0.82 ± 30%      +0.2        1.05 ± 11%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.25 ± 39%      -0.1        0.20 ± 31%      +0.1        0.31 ± 25%  perf-profile.self.cycles-pp.tsc_verify_tsc_adjust
      0.40 ± 22%      -0.1        0.35 ± 20%      +0.2        0.59 ± 11%  perf-profile.self.cycles-pp.security_file_permission
      1.08 ± 17%      -0.0        1.04 ± 22%      +0.2        1.25 ±  6%  perf-profile.self.cycles-pp.pipe_read
      0.57 ± 18%      -0.0        0.54 ± 21%      +0.1        0.69 ± 14%  perf-profile.self.cycles-pp.vfs_write
      0.31 ± 12%      -0.0        0.28 ± 13%      -0.0        0.31 ± 14%  perf-profile.self.cycles-pp.update_sd_lb_stats
      0.46 ± 45%      -0.0        0.43 ± 49%      +0.1        0.59 ± 41%  perf-profile.self.cycles-pp.ktime_get_update_offsets_now
      0.09 ± 27%      -0.0        0.06 ± 41%      -0.0        0.04 ± 74%  perf-profile.self.cycles-pp.cpuidle_governor_latency_req
      0.36 ± 29%      -0.0        0.34 ± 21%      +0.1        0.47 ± 16%  perf-profile.self.cycles-pp.do_syscall_64
      0.26 ± 19%      -0.0        0.24 ± 29%      -0.1        0.21 ±  9%  perf-profile.self.cycles-pp.irqtime_account_irq
      0.10 ± 54%      -0.0        0.08 ± 54%      -0.1        0.04 ± 74%  perf-profile.self.cycles-pp.irq_work_needs_cpu
      0.09 ± 25%      -0.0        0.06 ± 52%      -0.0        0.07 ± 42%  perf-profile.self.cycles-pp.tick_nohz_get_sleep_length
      0.36 ± 13%      -0.0        0.34 ± 17%      +0.0        0.37 ± 15%  perf-profile.self.cycles-pp._raw_spin_lock
      0.41 ± 28%      -0.0        0.38 ± 41%      -0.1        0.31 ± 16%  perf-profile.self.cycles-pp.rcu_sched_clock_irq
      0.27 ± 27%      -0.0        0.24 ± 18%      +0.0        0.31 ± 17%  perf-profile.self.cycles-pp.write
      0.13 ± 32%      -0.0        0.11 ± 49%      +0.0        0.14 ± 36%  perf-profile.self.cycles-pp.__softirqentry_text_start
      0.08 ± 25%      -0.0        0.06 ± 48%      -0.0        0.07 ± 44%  perf-profile.self.cycles-pp.tick_sched_timer
      0.27 ± 48%      -0.0        0.25 ± 18%      +0.1        0.34 ± 19%  perf-profile.self.cycles-pp._raw_spin_trylock
      0.12 ± 24%      -0.0        0.11 ± 37%      +0.0        0.13 ± 25%  perf-profile.self.cycles-pp.trigger_load_balance
      0.11 ± 22%      -0.0        0.10 ± 37%      -0.0        0.10 ± 31%  perf-profile.self.cycles-pp.rcu_idle_exit
      0.15 ± 31%      -0.0        0.13 ± 44%      -0.0        0.11 ± 23%  perf-profile.self.cycles-pp.rcu_eqs_exit
      0.10 ± 21%      -0.0        0.08 ± 38%      +0.0        0.10 ± 23%  perf-profile.self.cycles-pp.idle_cpu
      0.03 ± 96%      -0.0        0.02 ±142%      +0.0        0.06 ± 58%  perf-profile.self.cycles-pp.rcu_read_unlock_strict
      0.01 ±187%      -0.0        0.00            -0.0        0.00        perf-profile.self.cycles-pp.__intel_pmu_disable_all
      0.05 ± 58%      -0.0        0.03 ±108%      -0.0        0.01 ±187%  perf-profile.self.cycles-pp.update_dl_rq_load_avg
      0.15 ± 20%      -0.0        0.14 ± 21%      -0.0        0.12 ± 17%  perf-profile.self.cycles-pp.error_entry
      0.15 ± 21%      -0.0        0.14 ± 29%      -0.0        0.12 ± 28%  perf-profile.self.cycles-pp.__sysvec_apic_timer_interrupt
      0.09 ± 46%      -0.0        0.08 ± 40%      +0.0        0.09 ± 47%  perf-profile.self.cycles-pp.rcu_dynticks_eqs_enter
      0.13 ± 24%      -0.0        0.12 ± 22%      -0.0        0.11 ± 20%  perf-profile.self.cycles-pp.rebalance_domains
      0.01 ±187%      -0.0        0.00            -0.0        0.00        perf-profile.self.cycles-pp.hrtimer_run_queues
      0.05 ± 96%      -0.0        0.04 ± 90%      -0.0        0.01 ±187%  perf-profile.self.cycles-pp.do_nocb_deferred_wakeup
      0.14 ± 28%      -0.0        0.13 ± 28%      +0.0        0.14 ± 19%  perf-profile.self.cycles-pp.__might_fault
      0.08 ± 21%      -0.0        0.07 ± 18%      -0.0        0.06 ± 45%  perf-profile.self.cycles-pp.hrtimer_next_event_without
      0.32 ± 18%      -0.0        0.31 ± 21%      +0.0        0.36 ±  8%  perf-profile.self.cycles-pp.__intel_pmu_enable_all
      0.13 ± 21%      -0.0        0.12 ± 26%      -0.0        0.11 ± 20%  perf-profile.self.cycles-pp.load_balance
      0.20 ± 30%      -0.0        0.19 ± 35%      -0.0        0.15 ± 28%  perf-profile.self.cycles-pp.perf_mux_hrtimer_handler
      0.08 ± 62%      -0.0        0.07 ± 59%      -0.0        0.04 ± 94%  perf-profile.self.cycles-pp.x86_pmu_disable
      0.11 ± 29%      -0.0        0.10 ± 55%      -0.1        0.04 ± 72%  perf-profile.self.cycles-pp.timerqueue_del
      0.63 ± 16%      -0.0        0.62 ± 19%      +0.1        0.73 ±  8%  perf-profile.self.cycles-pp.native_sched_clock
      0.01 ±187%      -0.0        0.00 ±331%      +0.0        0.02 ±112%  perf-profile.self.cycles-pp.can_stop_idle_tick
      0.27 ± 11%      -0.0        0.27 ± 28%      -0.1        0.20 ± 14%  perf-profile.self.cycles-pp.__hrtimer_run_queues
      0.09 ± 50%      -0.0        0.09 ± 20%      +0.0        0.11 ± 25%  perf-profile.self.cycles-pp.copyout
      0.09 ± 88%      -0.0        0.08 ± 78%      -0.1        0.02 ±112%  perf-profile.self.cycles-pp.irq_work_tick
      0.11 ± 21%      -0.0        0.10 ± 23%      -0.0        0.10 ± 12%  perf-profile.self.cycles-pp._find_next_bit
      0.02 ±112%      -0.0        0.02 ±144%      -0.0        0.02 ±187%  perf-profile.self.cycles-pp.note_gp_changes
      0.01 ±282%      -0.0        0.00            -0.0        0.00        perf-profile.self.cycles-pp._dl_addr
      0.01 ±282%      -0.0        0.00            +0.0        0.01 ±282%  perf-profile.self.cycles-pp.tick_nohz_idle_retain_tick
      0.01 ±282%      -0.0        0.00            +0.0        0.03 ±119%  perf-profile.self.cycles-pp.cpuidle_enter
      0.06 ± 67%      -0.0        0.06 ± 63%      -0.0        0.02 ±190%  perf-profile.self.cycles-pp.rcu_eqs_enter
      0.24 ± 16%      -0.0        0.23 ± 18%      +0.1        0.29 ± 16%  perf-profile.self.cycles-pp.hrtimer_interrupt
      0.10 ± 22%      -0.0        0.09 ± 49%      -0.0        0.08 ± 24%  perf-profile.self.cycles-pp.update_blocked_averages
      0.01 ±188%      -0.0        0.01 ±223%      +0.0        0.01 ±188%  perf-profile.self.cycles-pp.find_busiest_group
      0.01 ±282%      -0.0        0.00 ±331%      -0.0        0.00        perf-profile.self.cycles-pp.ksoftirqd_running
      0.07 ± 53%      -0.0        0.07 ± 65%      -0.0        0.06 ± 42%  perf-profile.self.cycles-pp.get_cpu_device
      0.15 ± 31%      -0.0        0.15 ± 37%      -0.1        0.08 ± 18%  perf-profile.self.cycles-pp.sysvec_apic_timer_interrupt
      0.15 ± 30%      -0.0        0.15 ± 24%      +0.0        0.20 ± 18%  perf-profile.self.cycles-pp.syscall_exit_to_user_mode_prepare
      0.01 ±282%      -0.0        0.00 ±331%      -0.0        0.00        perf-profile.self.cycles-pp.timer_clear_idle
      0.01 ±187%      -0.0        0.01 ±227%      +0.0        0.02 ±144%  perf-profile.self.cycles-pp.update_group_capacity
      0.01 ±282%      -0.0        0.01 ±331%      +0.0        0.01 ±187%  perf-profile.self.cycles-pp.next_uptodate_page
      0.01 ±190%      -0.0        0.01 ±173%      -0.0        0.00        perf-profile.self.cycles-pp.intel_pmu_disable_all
      0.12 ± 45%      -0.0        0.12 ± 39%      -0.0        0.09 ± 21%  perf-profile.self.cycles-pp.get_next_timer_interrupt
      0.01 ±282%      -0.0        0.01 ±331%      -0.0        0.00        perf-profile.self.cycles-pp.tick_nohz_idle_got_tick
      0.00            +0.0        0.00            +0.0        0.01 ±187%  perf-profile.self.cycles-pp.__wake_up_sync_key
      0.02 ±145%      +0.0        0.02 ±119%      +0.0        0.04 ± 72%  perf-profile.self.cycles-pp.make_kuid
      0.02 ±197%      +0.0        0.02 ±146%      +0.0        0.04 ± 74%  perf-profile.self.cycles-pp.profile_tick
      0.07 ± 69%      +0.0        0.08 ± 26%      -0.1        0.02 ±150%  perf-profile.self.cycles-pp.hrtimer_update_next_event
      0.04 ± 92%      +0.0        0.04 ± 85%      -0.0        0.00        perf-profile.self.cycles-pp.restore_regs_and_return_to_kernel
      0.06 ± 96%      +0.0        0.06 ± 76%      -0.0        0.01 ±188%  perf-profile.self.cycles-pp.tick_check_broadcast_expired
      0.18 ± 22%      +0.0        0.18 ± 20%      +0.0        0.21 ± 15%  perf-profile.self.cycles-pp.rw_verify_area
      0.01 ±190%      +0.0        0.02 ±179%      +0.0        0.03 ± 91%  perf-profile.self.cycles-pp.__x64_sys_write
      0.09 ± 62%      +0.0        0.10 ± 46%      -0.1        0.03 ±114%  perf-profile.self.cycles-pp.irqtime_account_process_tick
      0.10 ± 29%      +0.0        0.10 ± 23%      -0.0        0.08 ± 22%  perf-profile.self.cycles-pp.irq_exit_rcu
      0.32 ± 12%      +0.0        0.32 ± 16%      +0.0        0.36 ± 16%  perf-profile.self.cycles-pp.arch_scale_freq_tick
      0.17 ± 30%      +0.0        0.17 ± 41%      -0.1        0.09 ± 23%  perf-profile.self.cycles-pp.timerqueue_add
      0.31 ± 25%      +0.0        0.32 ± 17%      +0.1        0.37 ± 16%  perf-profile.self.cycles-pp.current_time
      0.01 ±188%      +0.0        0.02 ±175%      -0.0        0.01 ±282%  perf-profile.self.cycles-pp.account_process_tick
      0.10 ± 19%      +0.0        0.11 ± 36%      -0.0        0.10 ± 32%  perf-profile.self.cycles-pp.sched_clock_cpu
      0.00            +0.0        0.00 ±331%      +0.0        0.00        perf-profile.self.cycles-pp.hrtimer_get_next_event
      0.00            +0.0        0.00 ±331%      +0.0        0.00        perf-profile.self.cycles-pp.__note_gp_changes
      0.00            +0.0        0.00 ±331%      +0.0        0.00        perf-profile.self.cycles-pp.run_rebalance_domains
      0.00            +0.0        0.00 ±331%      +0.0        0.00        perf-profile.self.cycles-pp.enqueue_hrtimer
      0.00            +0.0        0.00 ±331%      +0.0        0.00        perf-profile.self.cycles-pp.rcu_segcblist_ready_cbs
      0.00            +0.0        0.00 ±331%      +0.0        0.01 ±282%  perf-profile.self.cycles-pp.fold_vm_numa_events
      0.00            +0.0        0.00 ±331%      +0.0        0.03 ±114%  perf-profile.self.cycles-pp.timekeeping_advance
      0.16 ± 31%      +0.0        0.16 ± 33%      -0.1        0.08 ± 41%  perf-profile.self.cycles-pp.scheduler_tick
      0.02 ±144%      +0.0        0.02 ±144%      -0.0        0.00        perf-profile.self.cycles-pp.rcu_needs_cpu
      0.02 ±142%      +0.0        0.02 ±145%      -0.0        0.01 ±187%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.18 ± 12%      +0.0        0.18 ± 22%      +0.0        0.21 ± 11%  perf-profile.self.cycles-pp.rcu_all_qs
      0.01 ±282%      +0.0        0.01 ±230%      -0.0        0.00        perf-profile.self.cycles-pp.perf_ctx_unlock
      0.02 ±112%      +0.0        0.03 ±103%      -0.0        0.01 ±282%  perf-profile.self.cycles-pp.hrtimer_forward
      0.00            +0.0        0.01 ±331%      +0.0        0.00        perf-profile.self.cycles-pp.cpuidle_get_cpu_driver
      0.28 ± 27%      +0.0        0.28 ± 18%      -0.0        0.24 ± 13%  perf-profile.self.cycles-pp.update_irq_load_avg
      0.01 ±187%      +0.0        0.02 ±223%      -0.0        0.00        perf-profile.self.cycles-pp.rcu_core
      0.08 ± 24%      +0.0        0.08 ± 27%      -0.0        0.06 ± 58%  perf-profile.self.cycles-pp.apparmor_file_permission
      0.08 ± 41%      +0.0        0.09 ± 23%      -0.0        0.07 ± 54%  perf-profile.self.cycles-pp.tick_nohz_tick_stopped
      0.04 ± 92%      +0.0        0.05 ± 95%      -0.0        0.01 ±282%  perf-profile.self.cycles-pp.pm_qos_read_value
      0.43 ± 22%      +0.0        0.43 ± 14%      +0.1        0.52 ± 13%  perf-profile.self.cycles-pp.new_sync_read
      0.03 ±114%      +0.0        0.04 ± 86%      +0.0        0.04 ± 72%  perf-profile.self.cycles-pp.kill_fasync
      0.13 ± 76%      +0.0        0.14 ± 50%      -0.1        0.03 ±113%  perf-profile.self.cycles-pp.native_apic_mem_write
      0.08 ± 41%      +0.0        0.09 ± 36%      +0.0        0.12 ± 20%  perf-profile.self.cycles-pp.copyin
      0.03 ±115%      +0.0        0.04 ± 73%      -0.0        0.01 ±187%  perf-profile.self.cycles-pp.tick_sched_handle
      0.00            +0.0        0.01 ±223%      +0.0        0.00        perf-profile.self.cycles-pp.io_serial_out
      0.00            +0.0        0.01 ±223%      +0.0        0.00        perf-profile.self.cycles-pp.error_return
      0.18 ± 30%      +0.0        0.18 ± 34%      +0.0        0.19 ± 25%  perf-profile.self.cycles-pp.asm_sysvec_apic_timer_interrupt
      0.01 ±282%      +0.0        0.01 ±174%      -0.0        0.00        perf-profile.self.cycles-pp.rcu_nocb_flush_deferred_wakeup
      0.02 ±143%      +0.0        0.03 ±101%      +0.0        0.06 ± 56%  perf-profile.self.cycles-pp.irq_enter_rcu
      0.02 ±142%      +0.0        0.03 ±122%      -0.0        0.00        perf-profile.self.cycles-pp.leave_mm
      0.05 ± 95%      +0.0        0.06 ± 80%      -0.1        0.00        perf-profile.self.cycles-pp.irqentry_enter
      0.02 ±143%      +0.0        0.03 ±101%      -0.0        0.00        perf-profile.self.cycles-pp.tick_nohz_irq_exit
      0.10 ± 56%      +0.0        0.11 ± 47%      -0.1        0.03 ± 91%  perf-profile.self.cycles-pp.rb_insert_color
      0.01 ±188%      +0.0        0.02 ±119%      -0.0        0.00        perf-profile.self.cycles-pp.rcu_irq_enter
      0.24 ± 18%      +0.0        0.25 ± 16%      -0.0        0.24 ± 17%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.07 ± 87%      +0.0        0.08 ± 72%      -0.0        0.03 ±118%  perf-profile.self.cycles-pp.__remove_hrtimer
      0.13 ± 31%      +0.0        0.15 ± 31%      -0.0        0.09 ± 20%  perf-profile.self.cycles-pp.__hrtimer_get_next_event
      0.12 ± 18%      +0.0        0.14 ± 20%      +0.0        0.15 ± 20%  perf-profile.self.cycles-pp.call_cpuidle
      0.45 ± 21%      +0.0        0.46 ± 18%      +0.1        0.53 ± 12%  perf-profile.self.cycles-pp.new_sync_write
      0.09 ± 25%      +0.0        0.10 ± 24%      -0.0        0.09 ± 47%  perf-profile.self.cycles-pp.clockevents_program_event
      0.02 ±142%      +0.0        0.03 ±105%      +0.0        0.05 ± 56%  perf-profile.self.cycles-pp.make_kgid
      0.05 ± 96%      +0.0        0.06 ± 57%      -0.0        0.01 ±187%  perf-profile.self.cycles-pp.menu_reflect
      0.26 ± 18%      +0.0        0.27 ± 17%      +0.1        0.32 ± 17%  perf-profile.self.cycles-pp.syscall_exit_to_user_mode
      0.35 ± 23%      +0.0        0.36 ± 20%      -0.1        0.27 ± 14%  perf-profile.self.cycles-pp.__hrtimer_next_event_base
      0.05 ± 89%      +0.0        0.06 ± 83%      -0.0        0.01 ±190%  perf-profile.self.cycles-pp.cpuidle_not_available
      0.00            +0.0        0.01 ±174%      +0.0        0.00        perf-profile.self.cycles-pp.rcu_is_cpu_rrupt_from_idle
      0.03 ±124%      +0.0        0.05 ± 77%      -0.0        0.00        perf-profile.self.cycles-pp.rcu_nmi_enter
      0.16 ± 38%      +0.0        0.17 ± 20%      -0.1        0.10 ± 21%  perf-profile.self.cycles-pp.rb_next
      0.03 ±117%      +0.0        0.05 ± 87%      -0.0        0.00        perf-profile.self.cycles-pp.check_tsc_unstable
      0.10 ± 24%      +0.0        0.12 ± 25%      -0.0        0.08 ± 45%  perf-profile.self.cycles-pp.rcu_dynticks_eqs_exit
      0.02 ±114%      +0.0        0.04 ± 73%      -0.0        0.00        perf-profile.self.cycles-pp.task_tick_idle
      0.07 ± 85%      +0.0        0.09 ± 46%      -0.1        0.02 ±142%  perf-profile.self.cycles-pp.update_ts_time_stats
      0.00            +0.0        0.02 ±195%      +0.0        0.02 ±141%  perf-profile.self.cycles-pp.poll_idle
      0.02 ±148%      +0.0        0.04 ± 89%      -0.0        0.00        perf-profile.self.cycles-pp.cpuidle_reflect
      0.12 ± 32%      +0.0        0.14 ± 21%      +0.0        0.14 ± 25%  perf-profile.self.cycles-pp.iov_iter_init
      0.00            +0.0        0.02 ±143%      +0.0        0.03 ±156%  perf-profile.self.cycles-pp.main
      0.42 ± 16%      +0.0        0.44 ± 16%      -0.0        0.39 ±  9%  perf-profile.self.cycles-pp.do_idle
      0.05 ± 76%      +0.0        0.07 ± 39%      +0.0        0.05 ± 57%  perf-profile.self.cycles-pp.update_rt_rq_load_avg
      0.00            +0.0        0.02 ±145%      +0.0        0.01 ±282%  perf-profile.self.cycles-pp.__x64_sys_read
      0.08 ± 33%      +0.0        0.10 ± 24%      -0.0        0.06 ± 56%  perf-profile.self.cycles-pp.update_rq_clock
      0.01 ±282%      +0.0        0.03 ±103%      -0.0        0.01 ±282%  perf-profile.self.cycles-pp.sched_idle_set_state
      0.13 ± 70%      +0.0        0.15 ± 66%      -0.1        0.03 ± 90%  perf-profile.self.cycles-pp.memcpy_toio
      0.21 ± 29%      +0.0        0.23 ± 24%      +0.1        0.30 ± 24%  perf-profile.self.cycles-pp.ktime_get_coarse_real_ts64
      0.69 ± 22%      +0.0        0.71 ± 21%      +0.2        0.88 ± 14%  perf-profile.self.cycles-pp.exit_to_user_mode_prepare
      0.09 ± 28%      +0.0        0.11 ± 47%      -0.0        0.07 ± 47%  perf-profile.self.cycles-pp.rb_erase
      0.01 ±188%      +0.0        0.04 ± 86%      +0.0        0.04 ± 91%  perf-profile.self.cycles-pp.rcu_nmi_exit
      0.13 ± 57%      +0.0        0.15 ± 46%      -0.1        0.02 ±197%  perf-profile.self.cycles-pp.perf_event_task_tick
      0.15 ± 74%      +0.0        0.18 ± 48%      -0.1        0.04 ± 72%  perf-profile.self.cycles-pp.tick_check_oneshot_broadcast_this_cpu
      0.71 ± 25%      +0.0        0.74 ± 16%      +0.2        0.87 ± 13%  perf-profile.self.cycles-pp.__might_sleep
      0.11 ± 79%      +0.0        0.14 ± 52%      -0.1        0.03 ± 91%  perf-profile.self.cycles-pp.delay_tsc
      0.07 ± 62%      +0.0        0.10 ± 19%      -0.0        0.06 ± 39%  perf-profile.self.cycles-pp.tick_irq_enter
      0.16 ± 56%      +0.0        0.19 ± 41%      -0.1        0.09 ± 30%  perf-profile.self.cycles-pp.nr_iowait_cpu
      0.62 ± 22%      +0.0        0.65 ± 14%      +0.1        0.76 ± 12%  perf-profile.self.cycles-pp.copy_page_from_iter
      0.24 ± 20%      +0.0        0.27 ± 24%      +0.1        0.32 ± 12%  perf-profile.self.cycles-pp.read
      0.22 ± 20%      +0.0        0.26 ± 14%      +0.0        0.27 ± 21%  perf-profile.self.cycles-pp.atime_needs_update
      0.67 ± 17%      +0.0        0.70 ± 13%      +0.2        0.82 ± 14%  perf-profile.self.cycles-pp.copy_page_to_iter
      0.05 ± 75%      +0.0        0.08 ± 24%      +0.0        0.06 ± 56%  perf-profile.self.cycles-pp.tick_program_event
      0.38 ± 23%      +0.0        0.42 ± 18%      +0.1        0.49 ± 13%  perf-profile.self.cycles-pp.map_id_range_down
      0.38 ± 27%      +0.0        0.42 ± 20%      +0.1        0.52 ± 12%  perf-profile.self.cycles-pp.aa_file_perm
      0.32 ± 25%      +0.0        0.36 ± 21%      +0.1        0.40 ± 12%  perf-profile.self.cycles-pp.__fdget_pos
      0.25 ± 26%      +0.0        0.29 ± 18%      +0.1        0.32 ± 12%  perf-profile.self.cycles-pp.timestamp_truncate
      0.24 ± 18%      +0.0        0.28 ± 21%      +0.1        0.34 ± 16%  perf-profile.self.cycles-pp.file_update_time
      0.82 ± 21%      +0.0        0.86 ± 15%      +0.2        0.99 ± 12%  perf-profile.self.cycles-pp.___might_sleep
      0.24 ± 25%      +0.0        0.28 ± 24%      +0.1        0.31 ± 16%  perf-profile.self.cycles-pp.__wake_up_common_lock
      1.03 ± 13%      +0.0        1.07 ± 13%      +0.2        1.27 ±  5%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.25 ± 23%      +0.0        0.29 ± 17%      +0.1        0.30 ± 21%  perf-profile.self.cycles-pp.__cond_resched
      0.19 ± 30%      +0.0        0.24 ± 18%      +0.1        0.26 ± 15%  perf-profile.self.cycles-pp.__wake_up_common
      0.30 ± 25%      +0.0        0.34 ± 16%      +0.1        0.39 ± 21%  perf-profile.self.cycles-pp.ksys_read
      1.43 ± 21%      +0.0        1.48 ± 15%      +0.4        1.84 ±  9%  perf-profile.self.cycles-pp.pipe_write
      0.29 ± 19%      +0.0        0.34 ± 17%      +0.1        0.35 ± 11%  perf-profile.self.cycles-pp.syscall_enter_from_user_mode
      0.14 ± 21%      +0.1        0.19 ± 18%      +0.0        0.18 ± 20%  perf-profile.self.cycles-pp.touch_atime
      0.46 ± 65%      +0.1        0.52 ± 48%      -0.3        0.16 ± 23%  perf-profile.self.cycles-pp.io_serial_in
      0.17 ± 69%      +0.1        0.24 ± 52%      -0.1        0.05 ± 73%  perf-profile.self.cycles-pp.run_posix_cpu_timers
      0.44 ± 21%      +0.1        0.51 ± 18%      +0.2        0.62 ± 20%  perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
      0.95 ± 19%      +0.1        1.03 ± 16%      +0.2        1.14 ±  8%  perf-profile.self.cycles-pp.mutex_unlock
      0.29 ± 17%      +0.1        0.37 ± 18%      +0.1        0.38 ± 17%  perf-profile.self.cycles-pp.ksys_write
      0.67 ± 21%      +0.1        0.77 ± 14%      +0.3        0.97 ± 16%  perf-profile.self.cycles-pp.common_file_perm
      0.86 ± 19%      +0.1        0.98 ± 14%      +0.2        1.08 ± 12%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      1.14 ± 18%      +0.1        1.26 ± 13%      +0.3        1.41 ±  6%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      1.13 ± 20%      +0.1        1.26 ± 14%      +0.3        1.44 ± 15%  perf-profile.self.cycles-pp.mutex_lock
      1.61 ± 17%      +0.2        1.77 ± 13%      +0.4        1.97 ±  8%  perf-profile.self.cycles-pp.__fget_light
      1.37 ± 19%      +0.2        1.54 ± 15%      +0.3        1.72 ± 11%  perf-profile.self.cycles-pp.__entry_text_start
      2.35 ± 17%      +0.3        2.67 ± 15%      +0.6        2.96 ±  9%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      3.79 ± 20%      +0.4        4.21 ± 14%      +0.8        4.60 ± 10%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
     40.82 ± 12%      +0.7       41.56 ±  9%      -5.8       35.07 ± 10%  perf-profile.self.cycles-pp.intel_idle
    872.33 ±  2%      -2.7%     848.42            -2.6%     849.22 ±  2%  softirqs.BLOCK
      5.22 ±170%    +251.1%      18.33 ± 93%    +114.9%      11.22 ±116%  softirqs.CPU0.BLOCK
      1.00            +0.0%       1.00            +0.0%       1.00        softirqs.CPU0.HI
    497.67 ±256%     -89.2%      53.92 ±302%     -99.1%       4.44 ± 98%  softirqs.CPU0.NET_RX
      9.33 ±170%     -26.8%       6.83 ±213%     +41.7%      13.22 ±137%  softirqs.CPU0.NET_TX
     21651 ± 19%     +13.7%      24626 ± 46%      +7.3%      23227 ± 20%  softirqs.CPU0.RCU
     54004 ±  5%      +2.1%      55129 ±  2%      +2.9%      55586 ±  3%  softirqs.CPU0.SCHED
    113.33            +0.7%     114.17 ±  3%      -0.3%     113.00        softirqs.CPU0.TASKLET
      1472 ±  3%     +10.2%       1623 ± 14%      +0.8%       1484 ±  3%  softirqs.CPU0.TIMER
      4.33 ±200%    +413.5%      22.25 ±125%    +338.5%      19.00 ±106%  softirqs.CPU1.BLOCK
      1538 ±274%     -99.9%       1.83 ±283%     -99.6%       5.56 ±150%  softirqs.CPU1.NET_RX
      4.33 ±274%     -94.2%       0.25 ±238%     -92.3%       0.33 ±282%  softirqs.CPU1.NET_TX
     21072 ± 21%     +10.8%      23348 ± 52%      +4.3%      21976 ± 19%  softirqs.CPU1.RCU
     51548 ±  4%      -5.5%      48735 ± 21%      +2.0%      52598 ± 13%  softirqs.CPU1.SCHED
      0.00       +1.7e+101%       0.17 ±223% +1.1e+101%       0.11 ±282%  softirqs.CPU1.TASKLET
    538.44 ±119%     -71.1%     155.83 ±162%     -35.4%     347.78 ±165%  softirqs.CPU1.TIMER
     26.89 ±131%     +17.1%      31.50 ±240%     -27.7%      19.44 ±123%  softirqs.CPU10.BLOCK
      1.00            +0.0%       1.00            +0.0%       1.00        softirqs.CPU10.HI
    108.33 ±157%     -90.5%      10.33 ±213%     -29.2%      76.67 ±276%  softirqs.CPU10.NET_RX
      0.11 ±282%     -25.0%       0.08 ±331%    +100.0%       0.22 ±187%  softirqs.CPU10.NET_TX
     20571 ± 23%      +6.2%      21838 ± 41%      -6.1%      19313 ± 23%  softirqs.CPU10.RCU
     50168 ±  2%      -4.6%      47875 ± 17%      -2.1%      49115 ± 11%  softirqs.CPU10.SCHED
    646.78            +0.1%     647.50            +0.1%     647.11        softirqs.CPU10.TASKLET
    266.67 ±159%    +758.7%       2289 ±318%    +715.0%       2173 ±269%  softirqs.CPU10.TIMER
      1.00 ±221%   +1383.3%      14.83 ±217%   +1777.8%      18.78 ±172%  softirqs.CPU11.BLOCK
     33.11 ±203%      -9.1%      30.08 ±318%     -77.2%       7.56 ±207%  softirqs.CPU11.NET_RX
      0.00       +1.7e+101%       0.17 ±331% +1.1e+101%       0.11 ±282%  softirqs.CPU11.NET_TX
     20971 ± 23%     +12.6%      23603 ± 47%      +4.7%      21964 ± 19%  softirqs.CPU11.RCU
     50429            +2.4%      51637 ±  3%      -0.0%      50420 ±  3%  softirqs.CPU11.SCHED
      2.33 ± 28%     -17.9%       1.92 ± 14%     +85.7%       4.33 ±152%  softirqs.CPU11.TASKLET
    178.44 ±137%      -5.6%     168.50 ±177%     -39.4%     108.22 ± 75%  softirqs.CPU11.TIMER
      3.89 ±141%      +7.1%       4.17 ±276%     -91.4%       0.33 ±282%  softirqs.CPU12.BLOCK
      4.00 ±189%   +6756.2%     274.25 ±330%  +1.8e+05%       7060 ±270%  softirqs.CPU12.NET_RX
      0.44 ±111%     -43.8%       0.25 ±173%     -25.0%       0.33 ±199%  softirqs.CPU12.NET_TX
     19443 ± 28%     +15.3%      22422 ± 53%      +6.5%      20701 ± 23%  softirqs.CPU12.RCU
     49929 ±  3%      +0.5%      50185 ± 12%      +1.8%      50838 ±  3%  softirqs.CPU12.SCHED
      3.78 ±118%     -40.4%       2.25 ± 26%      +5.9%       4.00 ±141%  softirqs.CPU12.TASKLET
    889.89 ±262%     -83.3%     148.17 ±125%     -89.6%      92.89 ±108%  softirqs.CPU12.TIMER
     17.44 ±241%      -4.0%      16.75 ±225%     -93.0%       1.22 ±255%  softirqs.CPU13.BLOCK
      3504 ±277%     -99.3%      24.42 ±296%     -99.1%      32.00 ±243%  softirqs.CPU13.NET_RX
      0.56 ±123%     -10.0%       0.50 ±129%    +100.0%       1.11 ± 66%  softirqs.CPU13.NET_TX
     20901 ± 22%     +10.4%      23076 ± 50%      -1.5%      20582 ± 20%  softirqs.CPU13.RCU
     45582 ± 16%     +12.5%      51261 ±  4%     +12.1%      51099        softirqs.CPU13.SCHED
      3.78 ±144%     -36.0%       2.42 ± 61%     -38.2%       2.33 ± 63%  softirqs.CPU13.TASKLET
     79.33 ± 56%    +471.4%     453.33 ±137%    +185.9%     226.78 ±151%  softirqs.CPU13.TIMER
     14.67 ±275%     -14.8%      12.50 ±166%     -93.9%       0.89 ±282%  softirqs.CPU14.BLOCK
     81.33 ±193%   +2217.8%       1885 ±329%     -78.1%      17.78 ±266%  softirqs.CPU14.NET_RX
      0.56 ± 89%     -10.0%       0.50 ±129%     +40.0%       0.78 ± 80%  softirqs.CPU14.NET_TX
     20560 ± 23%     +16.1%      23879 ± 52%      +8.7%      22343 ± 18%  softirqs.CPU14.RCU
     50616            -5.1%      48032 ± 17%      +1.1%      51162        softirqs.CPU14.SCHED
      3.89 ±156%     +26.4%       4.92 ±133%     -42.9%       2.22 ± 28%  softirqs.CPU14.TASKLET
    647.44 ±185%     -72.8%     176.08 ±146%     -62.6%     241.89 ±166%  softirqs.CPU14.TIMER
     16.67 ±222%     -21.5%      13.08 ±266%     -17.3%      13.78 ±277%  softirqs.CPU15.BLOCK
     17.56 ±272%     +62.3%      28.50 ±216%    +340.5%      77.33 ±212%  softirqs.CPU15.NET_RX
      0.33 ±141%      +0.0%       0.33 ±141%     +66.7%       0.56 ±172%  softirqs.CPU15.NET_TX
     19950 ± 27%     +15.0%      22933 ± 50%      +8.3%      21600 ± 23%  softirqs.CPU15.RCU
     50579 ±  2%      -0.1%      50535 ±  7%      +1.0%      51103        softirqs.CPU15.SCHED
      1.56 ± 53%     +55.4%       2.42 ± 85%      +0.0%       1.56 ± 53%  softirqs.CPU15.TASKLET
      1753 ±267%     -92.0%     140.42 ±164%     -92.1%     139.22 ±192%  softirqs.CPU15.TIMER
     60.11 ±175%      +8.8%      65.42 ±168%     -66.9%      19.89 ±180%  softirqs.CPU16.BLOCK
      2.89 ±258%     +18.3%       3.42 ±140%   +3023.1%      90.22 ±243%  softirqs.CPU16.NET_RX
      0.44 ±215%      -6.2%       0.42 ±118%      +0.0%       0.44 ±154%  softirqs.CPU16.NET_TX
     18033 ± 23%     +11.8%      20167 ± 50%      +3.4%      18642 ± 19%  softirqs.CPU16.RCU
     48483 ± 15%      +4.0%      50422 ± 10%      -4.3%      46395 ± 20%  softirqs.CPU16.SCHED
      1.22 ± 92%     +63.6%       2.00 ± 73%      +9.1%       1.33 ± 93%  softirqs.CPU16.TASKLET
    587.89 ±228%     +78.7%       1050 ±245%     -85.1%      87.67 ±119%  softirqs.CPU16.TIMER
     46.89 ±184%     +65.8%      77.75 ± 94%     +16.4%      54.56 ±150%  softirqs.CPU17.BLOCK
     10.89 ±162%    +641.6%      80.75 ±325%     +54.1%      16.78 ±158%  softirqs.CPU17.NET_RX
      0.89 ±134%     -71.9%       0.25 ±173%     +25.0%       1.11 ±107%  softirqs.CPU17.NET_TX
     17551 ± 22%     +17.2%      20564 ± 53%      +3.3%      18130 ± 23%  softirqs.CPU17.RCU
     47778 ± 14%      +4.4%      49870 ± 15%      +7.5%      51369 ± 21%  softirqs.CPU17.SCHED
      0.56 ±149%    +125.0%       1.25 ±198%     +80.0%       1.00 ±188%  softirqs.CPU17.TASKLET
    373.33 ±213%     -66.0%     126.75 ±194%     -87.9%      45.33 ± 58%  softirqs.CPU17.TIMER
     71.56 ±112%      +9.0%      78.00 ±113%     -49.1%      36.44 ±164%  softirqs.CPU18.BLOCK
     12.22 ±262%     +51.4%      18.50 ±293%    +220.0%      39.11 ±270%  softirqs.CPU18.NET_RX
      0.78 ±169%    +339.3%       3.42 ±314%     -42.9%       0.44 ±154%  softirqs.CPU18.NET_TX
     17352 ± 20%     +10.6%      19187 ± 51%      +0.8%      17488 ± 22%  softirqs.CPU18.RCU
     47274 ± 19%      +6.9%      50542 ±  6%     +15.1%      54431 ± 28%  softirqs.CPU18.SCHED
      0.89 ±171%    -100.0%       0.00          -100.0%       0.00        softirqs.CPU18.TASKLET
    301.00 ±198%     -25.9%     223.08 ±194%     +77.5%     534.33 ±230%  softirqs.CPU18.TIMER
     29.22 ±224%     +30.9%      38.25 ±220%     +27.8%      37.33 ±178%  softirqs.CPU19.BLOCK
     38.22 ±272%   +9118.2%       3523 ±331%     -87.5%       4.78 ±166%  softirqs.CPU19.NET_RX
      0.56 ±191%     +50.0%       0.83 ±128%     -60.0%       0.22 ±187%  softirqs.CPU19.NET_TX
     17525 ± 23%     +12.2%      19668 ± 51%      +1.3%      17748 ± 24%  softirqs.CPU19.RCU
     49855 ±  3%      -0.3%      49684 ± 28%      +2.7%      51197 ±  2%  softirqs.CPU19.SCHED
      4.89 ±261%     -96.6%       0.17 ±331%     -90.9%       0.44 ±187%  softirqs.CPU19.TASKLET
      1106 ±259%     -84.7%     168.75 ±228%     -89.0%     121.89 ±102%  softirqs.CPU19.TIMER
      1.56 ±260%     +28.6%       2.00 ±206%    +535.7%       9.89 ±140%  softirqs.CPU2.BLOCK
      3.56 ±145%   +1243.0%      47.75 ±307%   +2353.1%      87.22 ±259%  softirqs.CPU2.NET_RX
      0.22 ±282%     -62.5%       0.08 ±331%    +200.0%       0.67 ±122%  softirqs.CPU2.NET_TX
     21418 ± 19%      +8.5%      23245 ± 54%      +0.3%      21483 ± 19%  softirqs.CPU2.RCU
     48241 ± 10%      +4.4%      50354 ± 12%      +7.0%      51620        softirqs.CPU2.SCHED
      0.33 ±200%   +1325.0%       4.75 ±225%    +600.0%       2.33 ±240%  softirqs.CPU2.TASKLET
    411.11 ±104%     +14.9%     472.33 ±265%     -71.9%     115.33 ± 54%  softirqs.CPU2.TIMER
      5.33 ±158%     +20.3%       6.42 ±176%     -68.8%       1.67 ±174%  softirqs.CPU20.BLOCK
      1.78 ±173%   +6068.8%     109.67 ±234%   +1018.8%      19.89 ±277%  softirqs.CPU20.NET_RX
      0.11 ±282%    +125.0%       0.25 ±173%    -100.0%       0.00        softirqs.CPU20.NET_TX
     18789 ± 23%     +11.4%      20928 ± 55%      -1.4%      18518 ± 21%  softirqs.CPU20.RCU
     51234 ± 18%      +0.2%      51352 ±  3%      -7.0%      47661 ± 15%  softirqs.CPU20.SCHED
      0.00       +1.7e+101%       0.17 ±331%   +1e+102%       1.00 ±221%  softirqs.CPU20.TASKLET
    105.67 ±110%     +79.0%     189.17 ±228%     +99.2%     210.44 ±152%  softirqs.CPU20.TIMER
      4.56 ±227%     +59.1%       7.25 ±208%     -97.6%       0.11 ±282%  softirqs.CPU21.BLOCK
      1.44 ± 92%    +176.9%       4.00 ±257%     +53.8%       2.22 ±251%  softirqs.CPU21.NET_RX
      0.44 ±154%     -43.8%       0.25 ±173%     -75.0%       0.11 ±282%  softirqs.CPU21.NET_TX
     18074 ± 23%     +15.6%      20893 ± 54%      +5.8%      19130 ± 21%  softirqs.CPU21.RCU
     49770 ±  6%      +3.6%      51564 ±  3%      +0.7%      50132 ±  5%  softirqs.CPU21.SCHED
      2.22 ±282%     -88.8%       0.25 ±238%    -100.0%       0.00        softirqs.CPU21.TASKLET
    504.89 ±235%     -52.3%     241.00 ±178%     -72.5%     138.67 ±195%  softirqs.CPU21.TIMER
      2.22 ±267%      +5.0%       2.33 ±254%    +120.0%       4.89 ± 98%  softirqs.CPU22.BLOCK
      0.67 ±122%   +8087.5%      54.58 ±297%    +166.7%       1.78 ±190%  softirqs.CPU22.NET_RX
      0.33 ±200%     +50.0%       0.50 ±129%     -66.7%       0.11 ±282%  softirqs.CPU22.NET_TX
     18024 ± 21%     +14.1%      20557 ± 54%      +3.0%      18569 ± 18%  softirqs.CPU22.RCU
     50204 ±  3%      -1.1%      49631 ± 13%      +1.9%      51172        softirqs.CPU22.SCHED
      2.11 ±219%     -88.2%       0.25 ±238%     -84.2%       0.33 ±200%  softirqs.CPU22.TASKLET
     65.33 ± 82%    +606.8%     461.75 ±180%    +338.3%     286.33 ±192%  softirqs.CPU22.TIMER
      5.33 ±172%     -43.8%       3.00 ±203%     -22.9%       4.11 ±147%  softirqs.CPU23.BLOCK
     34.33 ±223%      -5.6%      32.42 ±272%    +802.3%     309.78 ±222%  softirqs.CPU23.NET_RX
      0.44 ±154%     -25.0%       0.33 ±254%     -50.0%       0.22 ±187%  softirqs.CPU23.NET_TX
     18366 ± 23%     +10.5%      20296 ± 48%      +1.2%      18578 ± 21%  softirqs.CPU23.RCU
     48464 ± 10%      +7.6%      52163 ± 14%      +6.4%      51576 ±  2%  softirqs.CPU23.SCHED
      0.22 ±282%    +912.5%       2.25 ±331%     -50.0%       0.11 ±282%  softirqs.CPU23.TASKLET
    111.11 ± 65%     +61.0%     178.92 ±173%      +7.4%     119.33 ±114%  softirqs.CPU23.TIMER
      0.56 ±282%     +65.0%       0.92 ±300%   +1500.0%       8.89 ±278%  softirqs.CPU24.BLOCK
     28.44 ±205%    +365.5%     132.42 ±223%    +383.2%     137.44 ±220%  softirqs.CPU24.NET_RX
      4.44 ±282%    +192.5%      13.00 ±140%    +195.0%      13.11 ±141%  softirqs.CPU24.NET_TX
     18437 ± 21%     +18.9%      21923 ± 56%      -7.2%      17109 ± 22%  softirqs.CPU24.RCU
     51590            -0.7%      51247 ±  4%      -1.6%      50774 ±  2%  softirqs.CPU24.SCHED
      2.00 ±265%    -100.0%       0.00           -16.7%       1.67 ±262%  softirqs.CPU24.TASKLET
    797.56 ±168%     -39.9%     479.08 ±122%     -35.0%     518.67 ±142%  softirqs.CPU24.TIMER
      0.22 ±282%   +5787.5%      13.08 ±219%   +5450.0%      12.33 ±277%  softirqs.CPU25.BLOCK
    807.56 ±282%     -99.1%       7.17 ±327%     -98.6%      11.56 ±168%  softirqs.CPU25.NET_RX
      0.00          -100.0%       0.00       +3.3e+101%       0.33 ±200%  softirqs.CPU25.NET_TX
     17665 ± 16%     +17.4%      20734 ± 58%      -6.3%      16545 ± 21%  softirqs.CPU25.RCU
     51798 ±  3%      +0.4%      51981 ±  2%      +3.4%      53542 ± 15%  softirqs.CPU25.SCHED
      3.11 ±187%    -100.0%       0.00          -100.0%       0.00        softirqs.CPU25.TASKLET
    387.78 ±129%     +79.4%     695.83 ±123%     +73.5%     672.78 ±118%  softirqs.CPU25.TIMER
     17.44 ±251%     -78.5%       3.75 ±307%     -72.6%       4.78 ±240%  softirqs.CPU26.BLOCK
      0.00       +1.4e+102%       1.42 ±234%    -100.0%       0.00        softirqs.CPU26.NET_RX
      0.00       +8.3e+100%       0.08 ±331%    -100.0%       0.00        softirqs.CPU26.NET_TX
     18487 ± 17%     +21.5%      22467 ± 55%     -14.5%      15800 ± 26%  softirqs.CPU26.RCU
     50992 ±  2%      +0.3%      51166 ±  6%      -7.0%      47409 ± 14%  softirqs.CPU26.SCHED
      2.00            +4.2%       2.08 ± 13%      +5.6%       2.11 ± 14%  softirqs.CPU26.TASKLET
     72.00 ± 91%    +792.9%     642.92 ±105%   +2374.5%       1781 ±272%  softirqs.CPU26.TIMER
      3.78 ±282%     -16.2%       3.17 ±322%     -82.4%       0.67 ±234%  softirqs.CPU27.BLOCK
     31.56 ±281%     -81.5%       5.83 ±326%     -88.0%       3.78 ±264%  softirqs.CPU27.NET_RX
      0.00          -100.0%       0.00       +2.2e+101%       0.22 ±282%  softirqs.CPU27.NET_TX
     17296 ± 17%     +16.1%      20084 ± 61%      -6.3%      16205 ± 32%  softirqs.CPU27.RCU
     51298 ±  2%      -6.6%      47917 ± 15%     -12.2%      45035 ± 20%  softirqs.CPU27.SCHED
      1.67 ± 39%     +10.0%       1.83 ± 30%     +93.3%       3.22 ±108%  softirqs.CPU27.TASKLET
    442.78 ±239%     +36.3%     603.58 ±164%    +141.7%       1070 ±257%  softirqs.CPU27.TIMER
      7.56 ±282%     -96.7%       0.25 ±173%     -86.8%       1.00 ±282%  softirqs.CPU28.BLOCK
    120.22 ±258%     -82.2%      21.42 ±233%     -31.2%      82.67 ±282%  softirqs.CPU28.NET_RX
     18663 ± 17%     +17.7%      21964 ± 58%     -10.3%      16741 ± 22%  softirqs.CPU28.RCU
     51875 ± 11%      -0.0%      51867 ±  6%      -2.5%      50572 ±  3%  softirqs.CPU28.SCHED
      2.11 ± 78%     -17.1%       1.75 ± 34%     -15.8%       1.78 ± 23%  softirqs.CPU28.TASKLET
    228.11 ±227%     +61.8%     369.17 ±162%     -37.7%     142.22 ±227%  softirqs.CPU28.TIMER
      0.89 ±208%   +3106.2%      28.50 ±229%     -25.0%       0.67 ±234%  softirqs.CPU29.BLOCK
      5.00 ±282%     -53.3%       2.33 ±247%    +722.2%      41.11 ±258%  softirqs.CPU29.NET_RX
      0.11 ±282%     -25.0%       0.08 ±331%    +100.0%       0.22 ±282%  softirqs.CPU29.NET_TX
     18572 ± 16%     +19.9%      22267 ± 59%      -9.4%      16819 ± 23%  softirqs.CPU29.RCU
     51629 ±  2%      -0.1%      51595 ±  2%      -0.0%      51605 ±  6%  softirqs.CPU29.SCHED
      2.11 ± 14%      -5.3%       2.00            +5.3%       2.22 ± 28%  softirqs.CPU29.TASKLET
    244.56 ±213%     -53.0%     114.92 ±109%     -74.9%      61.33 ±113%  softirqs.CPU29.TIMER
      0.33 ±200%   +2650.0%       9.17 ±131%   +1833.3%       6.44 ±206%  softirqs.CPU3.BLOCK
     19.44 ±228%     -78.1%       4.25 ±210%    +761.1%     167.44 ±276%  softirqs.CPU3.NET_RX
      0.11 ±282%   +3050.0%       3.50 ±314%    +100.0%       0.22 ±187%  softirqs.CPU3.NET_TX
     21276 ± 24%      +9.8%      23363 ± 44%      +3.4%      21996 ± 21%  softirqs.CPU3.RCU
     49871 ± 18%      +4.9%      52335 ±  2%     +10.2%      54960 ± 16%  softirqs.CPU3.SCHED
      0.33 ±199%     +50.0%       0.50 ±129%    -100.0%       0.00        softirqs.CPU3.TASKLET
    600.00 ±199%     -30.9%     414.50 ±141%     -27.4%     435.67 ±126%  softirqs.CPU3.TIMER
     12.33 ±274%     -68.2%       3.92 ±286%    +172.1%      33.56 ±158%  softirqs.CPU30.BLOCK
      0.11 ±282%   +8675.0%       9.75 ±182%   +8400.0%       9.44 ±282%  softirqs.CPU30.NET_RX
      0.00       +2.5e+101%       0.25 ±238%    -100.0%       0.00        softirqs.CPU30.NET_TX
     17107 ± 17%     +17.4%      20084 ± 55%      -4.5%      16336 ± 25%  softirqs.CPU30.RCU
     51445 ±  2%      +2.7%      52822 ±  9%      -3.8%      49475 ±  2%  softirqs.CPU30.SCHED
      1.78 ± 35%      -6.2%       1.67 ± 44%      +6.2%       1.89 ± 16%  softirqs.CPU30.TASKLET
    228.67 ±223%     -13.1%     198.75 ±277%     -63.0%      84.56 ± 88%  softirqs.CPU30.TIMER
      7.78 ±185%     -28.2%       5.58 ±218%     -91.4%       0.67 ±158%  softirqs.CPU31.BLOCK
      7008 ±282%     -97.4%     185.58 ±200%     -99.9%       6.33 ±271%  softirqs.CPU31.NET_RX
      0.00          -100.0%       0.00       +1.1e+101%       0.11 ±282%  softirqs.CPU31.NET_TX
     18292 ± 18%     +18.8%      21733 ± 59%      -7.6%      16905 ± 24%  softirqs.CPU31.RCU
     51179 ±  2%      -6.3%      47945 ± 25%      -1.2%      50579 ±  5%  softirqs.CPU31.SCHED
      3.67 ±128%     -47.7%       1.92 ± 14%     -48.5%       1.89 ± 16%  softirqs.CPU31.TASKLET
    885.33 ±253%     -78.6%     189.58 ±235%     -95.8%      37.33 ± 84%  softirqs.CPU31.TIMER
     12.33 ±277%     -96.6%       0.42 ±267%     -23.4%       9.44 ±188%  softirqs.CPU32.BLOCK
     66.44 ±240%    +118.4%     145.08 ±127%     -99.8%       0.11 ±282%  softirqs.CPU32.NET_RX
     23106 ± 22%     +19.7%      27663 ± 61%     -19.2%      18679 ± 23%  softirqs.CPU32.RCU
     52990 ± 11%      -3.9%      50906 ±  6%      -1.3%      52284 ± 10%  softirqs.CPU32.SCHED
      2.67 ± 70%     -25.0%       2.00           -25.0%       2.00        softirqs.CPU32.TASKLET
    688.44 ±244%     -95.8%      28.83 ± 98%     -89.8%      70.33 ±105%  softirqs.CPU32.TIMER
      0.44 ±187%     +50.0%       0.67 ±215%    +875.0%       4.33 ±259%  softirqs.CPU33.BLOCK
    112.11 ±191%     -95.8%       4.75 ±273%    -100.0%       0.00        softirqs.CPU33.NET_RX
     22133 ± 19%     +19.2%      26379 ± 58%     -16.0%      18594 ± 22%  softirqs.CPU33.RCU
     50935 ±  2%      -0.2%      50829 ±  4%      -2.7%      49547 ±  2%  softirqs.CPU33.SCHED
      1.56 ± 53%     +28.6%       2.00           +28.6%       2.00        softirqs.CPU33.TASKLET
     44.11 ±151%     -46.7%      23.50 ± 47%     +21.7%      53.67 ±109%  softirqs.CPU33.TIMER
      0.67 ±282%    +487.5%       3.92 ±223%     +50.0%       1.00 ±200%  softirqs.CPU34.BLOCK
      7.33 ±250%    +633.0%      53.75 ±331%     -93.9%       0.44 ±187%  softirqs.CPU34.NET_RX
      0.11 ±282%    -100.0%       0.00          -100.0%       0.00        softirqs.CPU34.NET_TX
     21628 ± 16%     +26.4%      27330 ± 57%      -9.2%      19641 ± 21%  softirqs.CPU34.RCU
     51610 ±  2%      -0.1%      51580 ±  2%      -9.0%      46955 ± 13%  softirqs.CPU34.SCHED
      6.44 ±239%     -92.2%       0.50 ±173%     -25.9%       4.78 ±268%  softirqs.CPU34.TASKLET
      2043 ±146%     -91.9%     166.25 ±242%     -94.6%     109.89 ±151%  softirqs.CPU34.TIMER
     10.56 ±184%     +57.9%      16.67 ±250%      -3.2%      10.22 ±187%  softirqs.CPU35.BLOCK
      1.00 ±249%    +191.7%       2.92 ±226%  +19222.2%     193.22 ±282%  softirqs.CPU35.NET_RX
      0.11 ±282%    -100.0%       0.00          -100.0%       0.00        softirqs.CPU35.NET_TX
     22703 ± 20%     +20.3%      27309 ± 58%     -13.7%      19603 ± 23%  softirqs.CPU35.RCU
     51171 ±  2%      -3.7%      49279 ±  9%      -2.7%      49780        softirqs.CPU35.SCHED
      0.22 ±282%     -25.0%       0.17 ±331%    -100.0%       0.00        softirqs.CPU35.TASKLET
     41.33 ±156%     +78.0%      73.58 ±188%     +58.6%      65.56 ±142%  softirqs.CPU35.TIMER
      7.56 ±282%     +82.0%      13.75 ±236%     -54.4%       3.44 ±244%  softirqs.CPU36.BLOCK
    103.67 ±195%     -54.7%      47.00 ±331%    +797.0%     929.89 ±272%  softirqs.CPU36.NET_RX
      0.00          -100.0%       0.00       +2.2e+101%       0.22 ±282%  softirqs.CPU36.NET_TX
     20295 ± 19%     +14.5%      23244 ± 57%     -10.5%      18174 ± 24%  softirqs.CPU36.RCU
     51071 ±  3%      +0.8%      51499 ±  2%      -4.5%      48757 ±  3%  softirqs.CPU36.SCHED
      0.00          -100.0%       0.00       +5.6e+101%       0.56 ±282%  softirqs.CPU36.TASKLET
    642.78 ±112%     -25.3%     480.42 ±159%      -1.6%     632.44 ±228%  softirqs.CPU36.TIMER
     12.22 ±277%     -64.5%       4.33 ±190%     +58.2%      19.33 ±281%  softirqs.CPU37.BLOCK
      0.22 ±187%  +86375.0%     192.17 ±314%   +6550.0%      14.78 ±196%  softirqs.CPU37.NET_RX
      0.00       +8.3e+100%       0.08 ±331% +1.1e+101%       0.11 ±282%  softirqs.CPU37.NET_TX
     22730 ± 19%      +9.7%      24937 ± 61%     -17.9%      18653 ± 26%  softirqs.CPU37.RCU
     50830 ±  2%      -4.9%      48343 ± 23%      -9.4%      46045 ± 16%  softirqs.CPU37.SCHED
      0.00       +5.8e+101%       0.58 ±331%    -100.0%       0.00        softirqs.CPU37.TASKLET
    202.33 ±201%    +468.5%       1150 ±317%     -13.6%     174.78 ±188%  softirqs.CPU37.TIMER
      0.67 ±282%   +1350.0%       9.67 ±196%   +4083.3%      27.89 ±132%  softirqs.CPU38.BLOCK
    151.33 ±271%    +127.3%     344.00 ±327%     -68.9%      47.11 ±282%  softirqs.CPU38.NET_RX
      0.11 ±282%     -25.0%       0.08 ±331%    -100.0%       0.00        softirqs.CPU38.NET_TX
     23513 ± 22%     +19.8%      28159 ± 60%     -18.0%      19278 ± 23%  softirqs.CPU38.RCU
     49645 ±  8%      +4.1%      51680 ±  2%      +0.5%      49891        softirqs.CPU38.SCHED
      0.44 ±282%     -81.2%       0.08 ±331%     -25.0%       0.33 ±282%  softirqs.CPU38.TASKLET
     46.78 ±130%     +21.7%      56.92 ±138%     +63.7%      76.56 ±122%  softirqs.CPU38.TIMER
      0.00         +1e+102%       1.00 ±187% +3.3e+101%       0.33 ±200%  softirqs.CPU39.BLOCK
    172.56 ±272%     -87.3%      21.92 ±322%     -36.4%     109.78 ±282%  softirqs.CPU39.NET_RX
      0.00       +1.7e+101%       0.17 ±331% +1.1e+101%       0.11 ±282%  softirqs.CPU39.NET_TX
     21125 ± 18%     +19.1%      25169 ± 58%      -7.9%      19447 ± 22%  softirqs.CPU39.RCU
     48596 ± 15%      +6.0%      51506 ±  2%      +1.3%      49216 ±  3%  softirqs.CPU39.SCHED
     22.78 ± 53%    +140.4%      54.75 ±109%    +182.0%      64.22 ±187%  softirqs.CPU39.TIMER
      0.11 ±282%   +5525.0%       6.25 ±204%   +2200.0%       2.56 ±212%  softirqs.CPU4.BLOCK
     19.56 ±181%    +423.7%     102.42 ±301%     -98.3%       0.33 ±141%  softirqs.CPU4.NET_RX
      0.22 ±187%     +12.5%       0.25 ±238%      +0.0%       0.22 ±187%  softirqs.CPU4.NET_TX
     20875 ± 23%     +11.6%      23302 ± 43%      +7.0%      22337 ± 18%  softirqs.CPU4.RCU
     45490 ± 23%     +10.7%      50378 ± 13%      +3.9%      47260 ± 26%  softirqs.CPU4.SCHED
      4.56 ±237%     -81.7%       0.83 ±206%     -92.7%       0.33 ±200%  softirqs.CPU4.TASKLET
    116.00 ±170%   +1145.8%       1445 ±301%      +1.8%     118.11 ± 82%  softirqs.CPU4.TIMER
      0.44 ±282%   +1906.2%       8.92 ±331%   +1425.0%       6.78 ±233%  softirqs.CPU40.BLOCK
     60.78 ±281%     +40.7%      85.50 ±331%     +14.6%      69.67 ±282%  softirqs.CPU40.NET_RX
     23462 ± 22%     +17.0%      27444 ± 56%     -18.7%      19072 ± 26%  softirqs.CPU40.RCU
     48447 ± 15%      +6.3%      51489 ±  2%      +2.8%      49806        softirqs.CPU40.SCHED
      0.89 ±245%    -100.0%       0.00           +75.0%       1.56 ±282%  softirqs.CPU40.TASKLET
     52.89 ±187%    +156.2%     135.50 ±138%    +309.7%     216.67 ±130%  softirqs.CPU40.TIMER
      5.78 ±282%     -94.2%       0.33 ±187%     -78.8%       1.22 ±230%  softirqs.CPU41.BLOCK
     28.78 ±245%     -97.1%       0.83 ±331%    +149.8%      71.89 ±261%  softirqs.CPU41.NET_RX
      0.00          -100.0%       0.00       +1.1e+101%       0.11 ±282%  softirqs.CPU41.NET_TX
     23735 ± 22%     +15.0%      27296 ± 64%     -17.5%      19571 ± 22%  softirqs.CPU41.RCU
     51396            -3.2%      49734 ± 13%      -2.7%      50020        softirqs.CPU41.SCHED
      0.11 ±282%    +200.0%       0.33 ±331%    -100.0%       0.00        softirqs.CPU41.TASKLET
    206.11 ±213%    +383.0%     995.50 ±295%     +48.8%     306.67 ±217%  softirqs.CPU41.TIMER
      0.44 ±215%   +2018.8%       9.42 ±231%   +2175.0%      10.11 ±268%  softirqs.CPU42.BLOCK
      8.11 ±282%    +379.8%      38.92 ±317%     -78.1%       1.78 ±282%  softirqs.CPU42.NET_RX
     18961 ± 21%      +6.5%      20187 ± 50%      -2.2%      18547 ± 24%  softirqs.CPU42.RCU
     48638 ± 15%      +5.5%      51325 ±  2%      +1.4%      49334 ±  2%  softirqs.CPU42.SCHED
      0.11 ±282%    -100.0%       0.00         +4500.0%       5.11 ±282%  softirqs.CPU42.TASKLET
    215.44 ±223%     -40.5%     128.17 ±148%    +125.9%     486.67 ±260%  softirqs.CPU42.TIMER
      0.11 ±282%    -100.0%       0.00        +16300.0%      18.22 ±198%  softirqs.CPU43.BLOCK
      0.00       +4.9e+102%       4.92 ±331% +7.3e+102%       7.33 ±207%  softirqs.CPU43.NET_RX
      0.00       +1.7e+101%       0.17 ±331% +3.3e+101%       0.33 ±199%  softirqs.CPU43.NET_TX
     19958 ± 18%     +16.6%      23280 ± 55%      -3.5%      19264 ± 20%  softirqs.CPU43.RCU
     51940 ±  5%      -1.1%      51365 ±  2%      -9.4%      47042 ± 16%  softirqs.CPU43.SCHED
     75.11 ±205%     -55.2%      33.67 ± 72%    +117.9%     163.67 ±150%  softirqs.CPU43.TIMER
      9.89 ±196%     -11.5%       8.75 ±307%     -41.6%       5.78 ±258%  softirqs.CPU44.BLOCK
     16.67 ±282%    +113.5%      35.58 ±324%    +200.7%      50.11 ±280%  softirqs.CPU44.NET_RX
      0.00       +8.3e+100%       0.08 ±331%    -100.0%       0.00        softirqs.CPU44.NET_TX
     23808 ± 22%     +17.9%      28063 ± 59%     -23.1%      18306 ± 21%  softirqs.CPU44.RCU
     51398 ±  2%      +2.1%      52483 ±  3%      -5.5%      48556 ±  9%  softirqs.CPU44.SCHED
      0.22 ±282%    -100.0%       0.00          -100.0%       0.00        softirqs.CPU44.TASKLET
    103.67 ±130%     +63.6%     169.58 ±233%   +1330.4%       1482 ±268%  softirqs.CPU44.TIMER
     23.89 ±119%     -94.8%       1.25 ±251%     -50.2%      11.89 ±259%  softirqs.CPU45.BLOCK
      0.00         +1e+104%     104.17 ±256% +9.6e+102%       9.56 ±279%  softirqs.CPU45.NET_RX
     23605 ± 20%     +16.3%      27443 ± 56%     -18.7%      19201 ± 24%  softirqs.CPU45.RCU
     51184 ±  2%      -2.7%      49820 ± 13%      -4.5%      48873 ±  4%  softirqs.CPU45.SCHED
      0.00       +2.2e+102%       2.25 ±232%    -100.0%       0.00        softirqs.CPU45.TASKLET
    205.11 ±242%     -55.5%      91.33 ±159%     +52.9%     313.56 ±264%  softirqs.CPU45.TIMER
     27.67 ±180%     -90.7%       2.58 ±309%     -99.6%       0.11 ±282%  softirqs.CPU46.BLOCK
     54.11 ±253%     +26.0%      68.17 ±243%     -98.4%       0.89 ±282%  softirqs.CPU46.NET_RX
      0.00       +8.3e+100%       0.08 ±331%    -100.0%       0.00        softirqs.CPU46.NET_TX
     23097 ± 24%     +20.3%      27785 ± 59%     -15.3%      19569 ± 23%  softirqs.CPU46.RCU
     51665            -0.1%      51620 ±  2%      -6.6%      48260 ± 11%  softirqs.CPU46.SCHED
      0.00          -100.0%       0.00       +1.3e+102%       1.33 ±234%  softirqs.CPU46.TASKLET
    392.11 ±177%     -38.8%     239.83 ±229%      -4.1%     375.89 ±165%  softirqs.CPU46.TIMER
      0.44 ±187%   +2412.5%      11.17 ±250%   +1800.0%       8.44 ±254%  softirqs.CPU47.BLOCK
      9.22 ±184%     +32.8%      12.25 ±199%     -88.0%       1.11 ±252%  softirqs.CPU47.NET_RX
     13.00 ±141%     -50.6%       6.42 ±223%     -66.7%       4.33 ±274%  softirqs.CPU47.NET_TX
     21654 ± 23%     +19.9%      25967 ± 54%     -10.9%      19286 ± 25%  softirqs.CPU47.RCU
     46124 ± 15%      -8.0%      42423 ± 17%      -8.1%      42378 ± 24%  softirqs.CPU47.SCHED
      1.44 ±282%    -100.0%       0.00          -100.0%       0.00        softirqs.CPU47.TASKLET
    366.22 ±254%     -79.0%      77.08 ± 94%     -88.3%      42.78 ± 59%  softirqs.CPU47.TIMER
      2.67 ±282%     -65.6%       0.92 ±331%    +287.5%      10.33 ±144%  softirqs.CPU48.BLOCK
     71.33 ±271%     -99.2%       0.58 ±130%     -97.0%       2.11 ±157%  softirqs.CPU48.NET_RX
      0.22 ±187%    +125.0%       0.50 ±129%   +2050.0%       4.78 ±253%  softirqs.CPU48.NET_TX
     19990 ± 19%     +13.3%      22642 ± 50%      +1.5%      20283 ± 24%  softirqs.CPU48.RCU
     42280 ± 17%      -7.1%      39299 ± 31%      +2.2%      43231 ± 26%  softirqs.CPU48.SCHED
      0.22 ±282%    -100.0%       0.00         +2150.0%       5.00 ±275%  softirqs.CPU48.TASKLET
    331.44 ±156%    +139.1%     792.58 ±198%     -51.0%     162.44 ±128%  softirqs.CPU48.TIMER
      3.00 ±282%     +36.1%       4.08 ±192%     +11.1%       3.33 ±187%  softirqs.CPU49.BLOCK
     54.67 ±273%  +15359.6%       8451 ±256%     -95.5%       2.44 ±114%  softirqs.CPU49.NET_RX
      0.67 ±100%     -87.5%       0.08 ±331%     -66.7%       0.22 ±187%  softirqs.CPU49.NET_TX
     19479 ± 21%     +15.4%      22474 ± 54%      +5.4%      20534 ± 25%  softirqs.CPU49.RCU
     42715 ± 19%     +17.0%      49974 ±  9%     +16.5%      49775 ±  6%  softirqs.CPU49.SCHED
      0.44 ±154%     -81.2%       0.08 ±331%     -75.0%       0.11 ±282%  softirqs.CPU49.TASKLET
    143.78 ± 57%     +63.2%     234.58 ±105%    +661.7%       1095 ±107%  softirqs.CPU49.TIMER
      7.33 ±119%    -100.0%       0.00           -87.9%       0.89 ±282%  softirqs.CPU5.BLOCK
      1343 ±282%     -97.6%      32.25 ±318%     -99.8%       3.33 ±272%  softirqs.CPU5.NET_RX
      0.22 ±282%    -100.0%       0.00           -50.0%       0.11 ±282%  softirqs.CPU5.NET_TX
     21258 ± 23%     +14.4%      24314 ± 54%      +3.4%      21979 ± 21%  softirqs.CPU5.RCU
     51415 ±  5%      +1.0%      51909            -1.5%      50665 ±  6%  softirqs.CPU5.SCHED
      0.11 ±282%   +3725.0%       4.25 ±303%    +400.0%       0.56 ±149%  softirqs.CPU5.TASKLET
    109.44 ± 89%    +203.3%     331.92 ±167%    +139.1%     261.67 ±201%  softirqs.CPU5.TIMER
      0.11 ±282%  +16550.0%      18.50 ±188%      +0.0%       0.11 ±282%  softirqs.CPU50.BLOCK
     10.44 ±256%   +1192.6%     135.00 ±323%     +47.9%      15.44 ±273%  softirqs.CPU50.NET_RX
      0.22 ±187%     +50.0%       0.33 ±141%     +50.0%       0.33 ±141%  softirqs.CPU50.NET_TX
     20329 ± 20%     +13.3%      23040 ± 54%      +1.2%      20571 ± 23%  softirqs.CPU50.RCU
     46651 ± 10%      +7.2%      50007 ±  7%      +7.3%      50046 ±  2%  softirqs.CPU50.SCHED
      0.22 ±187%    +125.0%       0.50 ±152%    +100.0%       0.44 ±154%  softirqs.CPU50.TASKLET
    132.78 ± 71%    +253.1%     468.83 ±135%    +131.5%     307.44 ±192%  softirqs.CPU50.TIMER
     38.11 ±178%     -87.5%       4.75 ±209%     -97.7%       0.89 ±282%  softirqs.CPU51.BLOCK
     58.11 ±273%    +154.4%     147.83 ±241%     +39.2%      80.89 ±229%  softirqs.CPU51.NET_RX
      0.22 ±187%     +50.0%       0.33 ±141%    +150.0%       0.56 ±123%  softirqs.CPU51.NET_TX
     20816 ± 22%     +10.3%      22963 ± 46%      +0.3%      20872 ± 26%  softirqs.CPU51.RCU
     49448 ±  2%      +3.9%      51390 ±  3%      -8.4%      45296 ± 20%  softirqs.CPU51.SCHED
      1.44 ±237%    +107.7%       3.00 ±209%     -53.8%       0.67 ±158%  softirqs.CPU51.TASKLET
    105.56 ± 68%    +183.5%     299.25 ±154%    +676.8%     820.00 ±255%  softirqs.CPU51.TIMER
      1.22 ±203%    +350.0%       5.50 ±215%    +218.2%       3.89 ±219%  softirqs.CPU52.BLOCK
      1.56 ±125%   +4367.9%      69.50 ±260%    +114.3%       3.33 ± 94%  softirqs.CPU52.NET_RX
      0.44 ±154%     -43.8%       0.25 ±238%     -25.0%       0.33 ±141%  softirqs.CPU52.NET_TX
     21237 ± 21%     +12.0%      23783 ± 50%      +3.4%      21951 ± 23%  softirqs.CPU52.RCU
     47366 ± 14%      +9.0%      51649 ±  2%      +2.5%      48552 ± 14%  softirqs.CPU52.SCHED
      0.00          -100.0%       0.00       +2.1e+102%       2.11 ±234%  softirqs.CPU52.TASKLET
     81.44 ±106%     +45.8%     118.75 ± 59%    +133.3%     190.00 ±120%  softirqs.CPU52.TIMER
      9.33 ±187%     -84.8%       1.42 ±310%     -84.5%       1.44 ±282%  softirqs.CPU53.BLOCK
      5.00 ±163%     -41.7%       2.92 ±175%     -51.1%       2.44 ±169%  softirqs.CPU53.NET_RX
      0.44 ±111%      -6.2%       0.42 ±118%     +50.0%       0.67 ±100%  softirqs.CPU53.NET_TX
     20792 ± 22%     +14.4%      23780 ± 51%      +3.3%      21471 ± 23%  softirqs.CPU53.RCU
     46769 ± 13%      +6.2%      49665 ± 13%      +6.9%      50017 ±  5%  softirqs.CPU53.SCHED
      0.22 ±187%    +987.5%       2.42 ±331%     -50.0%       0.11 ±282%  softirqs.CPU53.TASKLET
    117.11 ± 85%    +247.6%     407.08 ±146%     +21.0%     141.67 ±122%  softirqs.CPU53.TIMER
      0.00         +8e+102%       8.00 ±197% +1.3e+103%      12.78 ±174%  softirqs.CPU54.BLOCK
      6.56 ±209%    +632.2%      48.00 ±322%     -54.2%       3.00 ±167%  softirqs.CPU54.NET_RX
      4.89 ±268%     -94.9%       0.25 ±173%     -93.2%       0.33 ±141%  softirqs.CPU54.NET_TX
     20183 ± 21%     +13.5%      22898 ± 50%      +9.4%      22080 ± 20%  softirqs.CPU54.RCU
     46601 ± 15%      +2.4%      47742 ± 19%      +8.4%      50502 ±  4%  softirqs.CPU54.SCHED
      0.44 ±187%     -62.5%       0.17 ±223%     -50.0%       0.22 ±282%  softirqs.CPU54.TASKLET
    146.44 ±101%    +138.3%     349.00 ±155%      -1.4%     144.44 ± 89%  softirqs.CPU54.TIMER
      1.11 ±226%    +342.5%       4.92 ±235%    +900.0%      11.11 ±279%  softirqs.CPU55.BLOCK
      0.33 ±141%   +2150.0%       7.50 ±168%  +20033.3%      67.11 ±280%  softirqs.CPU55.NET_RX
      0.11 ±282%    +425.0%       0.58 ±163%      +0.0%       0.11 ±282%  softirqs.CPU55.NET_TX
     20608 ± 20%     +15.0%      23702 ± 50%      +4.7%      21585 ± 22%  softirqs.CPU55.RCU
     48731 ±  3%      +3.3%      50331 ±  5%     -11.3%      43227 ± 28%  softirqs.CPU55.SCHED
      0.00       +2.2e+102%       2.25 ±228% +2.2e+101%       0.22 ±282%  softirqs.CPU55.TASKLET
    320.67 ±183%     -70.1%      95.83 ±100%     -16.6%     267.56 ±137%  softirqs.CPU55.TIMER
      2.89 ±172%     -13.5%       2.50 ±182%    +150.0%       7.22 ±277%  softirqs.CPU56.BLOCK
     21.56 ±266%     -88.0%       2.58 ±176%  +33677.3%       7280 ±282%  softirqs.CPU56.NET_RX
      0.33 ±200%     -50.0%       0.17 ±223%     +33.3%       0.44 ±111%  softirqs.CPU56.NET_TX
     21164 ± 22%     +10.4%      23370 ± 48%      +0.8%      21339 ± 20%  softirqs.CPU56.RCU
     48675 ±  5%      +0.2%      48780 ± 20%      +4.2%      50711 ±  3%  softirqs.CPU56.SCHED
      0.22 ±282%     -25.0%       0.17 ±331%     -50.0%       0.11 ±282%  softirqs.CPU56.TASKLET
    214.33 ±105%     +33.4%     285.83 ±174%     -61.7%      82.00 ± 64%  softirqs.CPU56.TIMER
      1.11 ±172%     +35.0%       1.50 ±195%      +0.0%       1.11 ±177%  softirqs.CPU57.BLOCK
    149.56 ±217%     -99.1%       1.33 ±184%     -95.8%       6.22 ±260%  softirqs.CPU57.NET_RX
      0.56 ±123%     -25.0%       0.42 ±153%     -20.0%       0.44 ±111%  softirqs.CPU57.NET_TX
     20817 ± 22%     +14.3%      23799 ± 46%      +4.7%      21804 ± 21%  softirqs.CPU57.RCU
     49647 ±  3%      -2.1%      48601 ± 19%      -0.4%      49442 ±  6%  softirqs.CPU57.SCHED
      0.44 ±187%    +143.8%       1.08 ±233%    -100.0%       0.00        softirqs.CPU57.TASKLET
    190.44 ±212%     +82.6%     347.75 ± 93%     -42.3%     109.89 ± 76%  softirqs.CPU57.TIMER
      6.00 ±150%    +312.5%      24.75 ±212%    +177.8%      16.67 ±136%  softirqs.CPU58.BLOCK
      1.33 ±180%  +13093.8%     175.92 ±312%   +4366.7%      59.56 ±233%  softirqs.CPU58.NET_RX
      0.33 ±141%      +0.0%       0.33 ±141%     -66.7%       0.11 ±282%  softirqs.CPU58.NET_TX
     20906 ± 20%     +11.7%      23343 ± 48%      -5.2%      19815 ± 24%  softirqs.CPU58.RCU
     46451 ± 12%      +6.2%      49321 ± 12%      +8.9%      50595 ±  2%  softirqs.CPU58.SCHED
      0.00       +2.9e+102%       2.92 ±331% +5.6e+101%       0.56 ±191%  softirqs.CPU58.TASKLET
     66.11 ± 48%    +123.1%     147.50 ±122%   +1255.0%     895.78 ±249%  softirqs.CPU58.TIMER
      0.33 ±141%   +4175.0%      14.25 ±284%   +1000.0%       3.67 ±273%  softirqs.CPU59.BLOCK
      0.67 ±187%   +3250.0%      22.33 ±262%   +2116.7%      14.78 ±178%  softirqs.CPU59.NET_RX
      0.22 ±187%    +237.5%       0.75 ±155%      +0.0%       0.22 ±187%  softirqs.CPU59.NET_TX
     20703 ± 21%     +14.0%      23604 ± 46%      +7.1%      22180 ± 22%  softirqs.CPU59.RCU
     46586 ± 13%      +9.1%      50845 ± 10%      +8.0%      50294 ±  4%  softirqs.CPU59.SCHED
      0.22 ±282%     +12.5%       0.25 ±331%    +300.0%       0.89 ±282%  softirqs.CPU59.TASKLET
    224.89 ±172%     -27.4%     163.17 ±107%     -35.2%     145.67 ± 49%  softirqs.CPU59.TIMER
      7.44 ±214%     -89.9%       0.75 ±293%     -83.6%       1.22 ±210%  softirqs.CPU6.BLOCK
      5556 ±282%     -99.2%      42.50 ±316%     -98.4%      91.00 ±165%  softirqs.CPU6.NET_RX
      0.11 ±282%     +50.0%       0.17 ±223%    +400.0%       0.56 ±191%  softirqs.CPU6.NET_TX
     20684 ± 24%     +15.8%      23947 ± 53%      +4.1%      21539 ± 19%  softirqs.CPU6.RCU
     50588 ±  2%      -1.9%      49627 ± 12%      -4.2%      48466 ± 13%  softirqs.CPU6.SCHED
      2.11 ±282%     +18.4%       2.50 ±223%     -84.2%       0.33 ±199%  softirqs.CPU6.TASKLET
    226.22 ±169%     +72.6%     390.50 ±199%     -77.9%      49.89 ± 49%  softirqs.CPU6.TIMER
      3.44 ±252%     -32.3%       2.33 ±143%      +3.2%       3.56 ±221%  softirqs.CPU60.BLOCK
    141.11 ±280%     -63.4%      51.58 ±288%     +13.5%     160.22 ±142%  softirqs.CPU60.NET_RX
      0.67 ± 70%     -50.0%       0.33 ±141%     -66.7%       0.22 ±187%  softirqs.CPU60.NET_TX
     21003 ± 23%      +5.1%      22068 ± 51%      +1.3%      21276 ± 24%  softirqs.CPU60.RCU
     48251 ±  4%      +4.1%      50244 ±  5%      +5.1%      50702 ±  3%  softirqs.CPU60.SCHED
      0.22 ±282%     +87.5%       0.42 ±182%    -100.0%       0.00        softirqs.CPU60.TASKLET
    201.44 ± 95%    +121.0%     445.25 ±226%      -6.7%     188.00 ± 59%  softirqs.CPU60.TIMER
      3.33 ±140%     +37.5%       4.58 ±215%    +153.3%       8.44 ±160%  softirqs.CPU61.BLOCK
    101.78 ±277%     -26.5%      74.83 ±319%     -96.2%       3.89 ±135%  softirqs.CPU61.NET_RX
      0.56 ±149%     -10.0%       0.50 ±129%      +0.0%       0.56 ± 89%  softirqs.CPU61.NET_TX
     21236 ± 21%     +10.2%      23410 ± 49%      +0.8%      21415 ± 23%  softirqs.CPU61.RCU
     47194 ± 11%      +8.3%      51129 ±  3%      +7.4%      50685 ±  2%  softirqs.CPU61.SCHED
      3.00 ±248%     -16.7%       2.50 ±214%     -59.3%       1.22 ± 92%  softirqs.CPU61.TASKLET
    115.22 ±117%    +167.9%     308.67 ±182%     +81.3%     208.89 ± 67%  softirqs.CPU61.TIMER
     31.00 ±153%     -83.6%       5.08 ±235%     -59.1%      12.67 ±282%  softirqs.CPU62.BLOCK
    179.00 ±213%    +155.4%     457.08 ±195%     -96.1%       7.00 ±228%  softirqs.CPU62.NET_RX
      0.00       +5.8e+101%       0.58 ±147% +4.4e+101%       0.44 ±154%  softirqs.CPU62.NET_TX
     20841 ± 19%     +12.4%      23426 ± 49%      +5.7%      22027 ± 20%  softirqs.CPU62.RCU
     49538 ±  3%      -0.1%      49489 ± 13%      +3.8%      51441 ±  2%  softirqs.CPU62.SCHED
      1.89 ±247%     -69.1%       0.58 ±130%      +0.0%       1.89 ±212%  softirqs.CPU62.TASKLET
    103.78 ± 56%    +345.6%     462.42 ±194%      +3.7%     107.67 ± 70%  softirqs.CPU62.TIMER
      4.78 ±138%     -73.8%       1.25 ±308%     -97.7%       0.11 ±282%  softirqs.CPU63.BLOCK
     90.11 ±264%     +54.0%     138.75 ±239%   +1030.1%       1018 ±282%  softirqs.CPU63.NET_RX
      0.33 ±200%     -25.0%       0.25 ±173%     -66.7%       0.11 ±282%  softirqs.CPU63.NET_TX
     20934 ± 21%     +11.4%      23326 ± 47%      +3.2%      21602 ± 23%  softirqs.CPU63.RCU
     49439 ±  4%      +2.5%      50673 ±  4%      +4.0%      51398 ±  2%  softirqs.CPU63.SCHED
      1.78 ±229%    +125.0%       4.00 ±265%    -100.0%       0.00        softirqs.CPU63.TASKLET
    109.22 ± 58%    +158.6%     282.50 ±171%    +269.9%     404.00 ±146%  softirqs.CPU63.TIMER
      2.22 ±183%    +380.0%      10.67 ±309%    -100.0%       0.00        softirqs.CPU64.BLOCK
     27.11 ±275%    +183.4%      76.83 ±313%     -87.3%       3.44 ±161%  softirqs.CPU64.NET_RX
      0.22 ±187%     +50.0%       0.33 ±331%     -50.0%       0.11 ±282%  softirqs.CPU64.NET_TX
     18361 ± 21%     +13.9%      20906 ± 54%      +0.8%      18513 ± 21%  softirqs.CPU64.RCU
     50247 ±  2%      +2.7%      51620 ±  2%      +1.7%      51079 ±  2%  softirqs.CPU64.SCHED
      0.22 ±282%    +612.5%       1.58 ±242%      +0.0%       0.22 ±282%  softirqs.CPU64.TASKLET
    386.89 ±141%     -67.9%     124.00 ±127%     -65.9%     131.89 ±105%  softirqs.CPU64.TIMER
      0.22 ±187%   +2225.0%       5.17 ±256%    -100.0%       0.00        softirqs.CPU65.BLOCK
     10.11 ±198%    +339.3%      44.42 ±284%     -25.3%       7.56 ±204%  softirqs.CPU65.NET_RX
      0.11 ±282%     +50.0%       0.17 ±223%    +100.0%       0.22 ±187%  softirqs.CPU65.NET_TX
     17654 ± 18%     +16.4%      20540 ± 55%      +2.5%      18102 ± 24%  softirqs.CPU65.RCU
     49228 ±  3%      +3.7%      51069 ±  4%      -0.8%      48848 ± 16%  softirqs.CPU65.SCHED
      0.22 ±282%    +387.5%       1.08 ±278%    +750.0%       1.89 ±282%  softirqs.CPU65.TASKLET
    200.00 ± 83%     -25.3%     149.33 ± 87%     +18.8%     237.56 ±151%  softirqs.CPU65.TIMER
      5.11 ±269%     +53.3%       7.83 ±157%    +402.2%      25.67 ±180%  softirqs.CPU66.BLOCK
     14.78 ±266%    +171.2%      40.08 ±327%     -12.0%      13.00 ±264%  softirqs.CPU66.NET_RX
      0.22 ±282%     +50.0%       0.33 ±187%      +0.0%       0.22 ±187%  softirqs.CPU66.NET_TX
     16719 ± 22%     +12.2%      18755 ± 53%      +6.7%      17832 ± 24%  softirqs.CPU66.RCU
     46999 ± 14%      +5.3%      49508 ±  8%      +4.7%      49199 ±  9%  softirqs.CPU66.SCHED
      0.00          -100.0%       0.00       +3.3e+101%       0.33 ±282%  softirqs.CPU66.TASKLET
      1302 ±180%     -50.6%     643.58 ±241%     -85.8%     185.11 ± 89%  softirqs.CPU66.TIMER
      6.11 ±217%     +28.2%       7.83 ±170%     +20.0%       7.33 ±216%  softirqs.CPU67.BLOCK
      9.44 ±143%   +1015.3%     105.33 ±306%     -97.6%       0.22 ±282%  softirqs.CPU67.NET_RX
      0.89 ±134%     -71.9%       0.25 ±173%    -100.0%       0.00        softirqs.CPU67.NET_TX
     17631 ± 20%     +11.2%      19609 ± 52%      -0.3%      17576 ± 25%  softirqs.CPU67.RCU
     45003 ± 20%     +14.4%      51465 ±  4%      +7.9%      48537 ± 15%  softirqs.CPU67.SCHED
      2.22 ±162%     -43.8%       1.25 ±331%     -35.0%       1.44 ±282%  softirqs.CPU67.TASKLET
    705.78 ±132%     -79.2%     146.67 ± 89%     -52.8%     333.22 ±149%  softirqs.CPU67.TIMER
      4.78 ±275%     +20.3%       5.75 ±261%    -100.0%       0.00        softirqs.CPU68.BLOCK
     66.56 ±257%     -92.7%       4.83 ±258%     +19.5%      79.56 ±275%  softirqs.CPU68.NET_RX
      4.44 ±274%     -94.4%       0.25 ±238%     -90.0%       0.44 ±154%  softirqs.CPU68.NET_TX
     18043 ± 22%     +15.3%      20800 ± 56%      +2.7%      18530 ± 21%  softirqs.CPU68.RCU
     47181 ± 13%     +10.4%      52110 ±  2%      +7.7%      50821 ±  4%  softirqs.CPU68.SCHED
      2.33 ±282%     -78.6%       0.50 ±173%    -100.0%       0.00        softirqs.CPU68.TASKLET
     88.56 ± 56%    +768.4%     769.00 ±177%    +227.9%     290.33 ±170%  softirqs.CPU68.TIMER
      0.00       +6.7e+101%       0.67 ±331%   +1e+102%       1.00 ±249%  softirqs.CPU69.BLOCK
      1.11 ±166%    +380.0%       5.33 ±177%   +6210.0%      70.11 ±275%  softirqs.CPU69.NET_RX
      0.22 ±282%     +87.5%       0.42 ±182%     +50.0%       0.33 ±199%  softirqs.CPU69.NET_TX
     18237 ± 20%     +12.2%      20456 ± 58%      +1.0%      18425 ± 21%  softirqs.CPU69.RCU
     48624 ±  6%      +3.3%      50227 ±  6%      -1.4%      47924 ± 11%  softirqs.CPU69.SCHED
      0.00       +2.5e+101%       0.25 ±238%    -100.0%       0.00        softirqs.CPU69.TASKLET
    334.78 ±137%     -60.8%     131.33 ± 46%     -72.4%      92.56 ± 70%  softirqs.CPU69.TIMER
      7.56 ±157%     -92.3%       0.58 ±283%     -29.4%       5.33 ±244%  softirqs.CPU7.BLOCK
      0.89 ±245%    +659.4%       6.75 ±280%    +437.5%       4.78 ±151%  softirqs.CPU7.NET_RX
      0.00       +8.3e+100%       0.08 ±331% +3.3e+101%       0.33 ±282%  softirqs.CPU7.NET_TX
     20425 ± 20%     +14.5%      23387 ± 52%      +1.3%      20696 ± 22%  softirqs.CPU7.RCU
     49348 ±  4%      -0.1%      49313 ± 13%      +0.5%      49578 ±  9%  softirqs.CPU7.SCHED
      0.56 ±172%     -40.0%       0.33 ±223%     -80.0%       0.11 ±282%  softirqs.CPU7.TASKLET
    172.56 ±124%    +111.2%     364.42 ±183%     +86.9%     322.44 ±218%  softirqs.CPU7.TIMER
      1.78 ±263%     -20.3%       1.42 ±224%      +0.0%       1.78 ±245%  softirqs.CPU70.BLOCK
      3.00 ±191%  +23094.4%     695.83 ±330%   +3229.6%      99.89 ±177%  softirqs.CPU70.NET_RX
      0.22 ±282%     +12.5%       0.25 ±173%      +0.0%       0.22 ±282%  softirqs.CPU70.NET_TX
     17653 ± 20%     +15.5%      20391 ± 56%      -3.3%      17071 ± 30%  softirqs.CPU70.RCU
     49553 ±  3%      -8.3%      45424 ± 21%      +3.0%      51034        softirqs.CPU70.SCHED
      2.33 ±239%     -85.7%       0.33 ±187%     -76.2%       0.56 ±191%  softirqs.CPU70.TASKLET
    421.78 ±245%    +164.7%       1116 ±206%    +430.9%       2239 ±185%  softirqs.CPU70.TIMER
      7.78 ±191%     -81.8%       1.42 ±271%     -78.6%       1.67 ±241%  softirqs.CPU71.BLOCK
      2.67 ±231%  +71793.8%       1917 ±320%   +3870.8%     105.89 ±281%  softirqs.CPU71.NET_RX
      0.22 ±187%     -62.5%       0.08 ±331%     -50.0%       0.11 ±282%  softirqs.CPU71.NET_TX
     18238 ± 19%     +12.2%      20454 ± 55%      +1.7%      18548 ± 20%  softirqs.CPU71.RCU
     48137 ±  7%      +2.1%      49163 ± 12%      +6.4%      51236 ±  2%  softirqs.CPU71.SCHED
      0.00       +5.7e+102%       5.67 ±248% +1.1e+101%       0.11 ±282%  softirqs.CPU71.TASKLET
    501.22 ±163%     -20.6%     398.00 ±171%     -62.0%     190.44 ±125%  softirqs.CPU71.TIMER
      6.00 ±156%    +158.3%      15.50 ±225%     -40.7%       3.56 ±245%  softirqs.CPU72.BLOCK
      6.78 ±198%     -47.1%       3.58 ±265%  +53473.8%       3631 ±281%  softirqs.CPU72.NET_RX
      0.00       +3.6e+102%       3.58 ±298% +4.4e+102%       4.44 ±282%  softirqs.CPU72.NET_TX
     18811 ± 19%     +13.1%      21281 ± 56%      -9.5%      17018 ± 24%  softirqs.CPU72.RCU
     51911 ±  2%      +0.2%      51997            -4.3%      49701 ±  3%  softirqs.CPU72.SCHED
      0.11 ±282%    +575.0%       0.75 ±238%    -100.0%       0.00        softirqs.CPU72.TASKLET
    163.22 ±205%      +8.4%     177.00 ± 81%     -42.9%      93.22 ± 93%  softirqs.CPU72.TIMER
     15.56 ±278%     -97.3%       0.42 ±228%     +22.1%      19.00 ±176%  softirqs.CPU73.BLOCK
     94.22 ±282%     -61.0%      36.75 ±329%     -70.2%      28.11 ±252%  softirqs.CPU73.NET_RX
      0.00          -100.0%       0.00       +1.1e+101%       0.11 ±282%  softirqs.CPU73.NET_TX
     17225 ± 17%     +18.7%      20450 ± 56%      -4.8%      16401 ± 24%  softirqs.CPU73.RCU
     50971 ±  4%      +1.3%      51651 ±  3%      -5.5%      48158 ±  5%  softirqs.CPU73.SCHED
      1.44 ±282%    -100.0%       0.00            +0.0%       1.44 ±219%  softirqs.CPU73.TASKLET
    284.67 ±180%     -32.5%     192.08 ±154%     -55.0%     128.11 ± 84%  softirqs.CPU73.TIMER
     24.00 ±282%     -98.3%       0.42 ±153%     -96.8%       0.78 ±240%  softirqs.CPU74.BLOCK
     60.89 ±282%     -88.9%       6.75 ±242%     -27.0%      44.44 ±279%  softirqs.CPU74.NET_RX
      0.00          -100.0%       0.00       +1.1e+101%       0.11 ±282%  softirqs.CPU74.NET_TX
     17655 ± 14%     +18.6%      20931 ± 60%      -6.8%      16461 ± 23%  softirqs.CPU74.RCU
     51131 ±  3%      +1.6%      51924 ±  3%      -2.6%      49785 ±  6%  softirqs.CPU74.SCHED
      0.67 ±282%     -87.5%       0.08 ±331%     -83.3%       0.11 ±282%  softirqs.CPU74.TASKLET
    989.67 ±256%     +14.5%       1133 ±244%     -73.7%     259.89 ±144%  softirqs.CPU74.TIMER
      2.89 ±187%      -4.8%       2.75 ±290%    +819.2%      26.56 ±148%  softirqs.CPU75.BLOCK
      0.00       +3.3e+101%       0.33 ±331% +7.1e+103%      71.33 ±243%  softirqs.CPU75.NET_RX
     17115 ± 16%     +18.2%      20225 ± 57%      -3.4%      16532 ± 22%  softirqs.CPU75.RCU
     51071 ±  3%      -1.3%      50404 ±  6%      -3.6%      49217 ±  2%  softirqs.CPU75.SCHED
      0.00       +8.3e+100%       0.08 ±331%   +5e+102%       5.00 ±184%  softirqs.CPU75.TASKLET
    248.56 ±234%     -73.6%      65.58 ± 85%     -30.4%     172.89 ±208%  softirqs.CPU75.TIMER
      5.56 ±270%     +42.5%       7.92 ±252%     -64.0%       2.00 ±141%  softirqs.CPU76.BLOCK
      5.44 ±282%    +686.7%      42.83 ±330%    +740.8%      45.78 ±282%  softirqs.CPU76.NET_RX
      0.22 ±282%    -100.0%       0.00          -100.0%       0.00        softirqs.CPU76.NET_TX
     18363 ± 19%     +16.4%      21372 ± 58%      -8.1%      16880 ± 23%  softirqs.CPU76.RCU
     51183 ±  3%      +0.6%      51507 ±  4%      -3.7%      49313 ±  2%  softirqs.CPU76.SCHED
      0.11 ±282%   +4775.0%       5.42 ±331%    -100.0%       0.00        softirqs.CPU76.TASKLET
     84.33 ±141%    +383.6%     407.83 ±277%     -54.2%      38.67 ± 81%  softirqs.CPU76.TIMER
     16.00 ±212%     -90.6%       1.50 ±331%     -73.6%       4.22 ±274%  softirqs.CPU77.BLOCK
      1.89 ±282%    +822.1%      17.42 ±308%  +4.2e+05%       7941 ±250%  softirqs.CPU77.NET_RX
      0.00       +8.3e+100%       0.08 ±331%    -100.0%       0.00        softirqs.CPU77.NET_TX
     18240 ± 19%     +20.5%      21987 ± 57%     -10.3%      16362 ± 27%  softirqs.CPU77.RCU
     49828 ±  9%      +0.6%      50104 ± 11%      -0.1%      49756 ±  3%  softirqs.CPU77.SCHED
      0.00       +8.3e+100%       0.08 ±331%    -100.0%       0.00        softirqs.CPU77.TASKLET
    843.33 ±245%     -84.6%     130.00 ±183%     +51.1%       1274 ±191%  softirqs.CPU77.TIMER
      8.56 ±258%     -24.0%       6.50 ±327%    +258.4%      30.67 ±160%  softirqs.CPU78.BLOCK
      0.11 ±282%   +1550.0%       1.83 ±331%    +500.0%       0.67 ±282%  softirqs.CPU78.NET_RX
      0.00       +1.7e+101%       0.17 ±331%    -100.0%       0.00        softirqs.CPU78.NET_TX
     17327 ± 18%     +15.8%      20059 ± 55%      -5.2%      16433 ± 24%  softirqs.CPU78.RCU
     50752 ±  4%      +1.6%      51546 ±  3%      -5.1%      48160 ±  3%  softirqs.CPU78.SCHED
      0.11 ±282%     -25.0%       0.08 ±331%    -100.0%       0.00        softirqs.CPU78.TASKLET
     80.00 ±130%     -41.5%      46.83 ± 95%      +3.3%      82.67 ± 90%  softirqs.CPU78.TIMER
      0.78 ±240%     -46.4%       0.42 ±331%    -100.0%       0.00        softirqs.CPU79.BLOCK
    321.33 ±149%     -63.1%     118.50 ±310%    -100.0%       0.00        softirqs.CPU79.NET_RX
     18585 ± 18%     +15.3%      21429 ± 57%      -9.5%      16824 ± 23%  softirqs.CPU79.RCU
     50722 ±  3%      +2.0%      51726 ±  2%      -3.7%      48848 ±  2%  softirqs.CPU79.SCHED
      1.56 ±196%     -89.3%       0.17 ±331%    -100.0%       0.00        softirqs.CPU79.TASKLET
     66.22 ±122%     -45.3%      36.25 ± 92%     -32.7%      44.56 ±106%  softirqs.CPU79.TIMER
      5.78 ±195%     -36.5%       3.67 ±229%    +407.7%      29.33 ±214%  softirqs.CPU8.BLOCK
      1.44 ±166%     -82.7%       0.25 ±173%   +3592.3%      53.33 ±253%  softirqs.CPU8.NET_RX
      0.33 ±199%    -100.0%       0.00          -100.0%       0.00        softirqs.CPU8.NET_TX
     21261 ± 19%      +9.2%      23211 ± 48%      +2.2%      21734 ± 22%  softirqs.CPU8.RCU
     43950 ± 20%     +16.7%      51281 ± 15%     +14.8%      50469 ± 22%  softirqs.CPU8.SCHED
      0.11 ±282%     -25.0%       0.08 ±331%    -100.0%       0.00        softirqs.CPU8.TASKLET
    199.33 ±180%     -72.3%      55.25 ± 78%    +173.0%     544.22 ±174%  softirqs.CPU8.TIMER
      0.78 ±282%    +103.6%       1.58 ±295%   +1685.7%      13.89 ±241%  softirqs.CPU80.BLOCK
     17.33 ±282%      -1.4%      17.08 ±329%    +301.9%      69.67 ±270%  softirqs.CPU80.NET_RX
     23593 ± 22%     +15.0%      27127 ± 58%     -20.4%      18791 ± 22%  softirqs.CPU80.RCU
     51390 ±  2%      -5.5%      48538 ± 15%      -2.7%      49995        softirqs.CPU80.SCHED
      0.00          -100.0%       0.00          -100.0%       0.00        softirqs.CPU80.TASKLET
    107.00 ±132%     -52.8%      50.50 ±116%     -38.6%      65.67 ±113%  softirqs.CPU80.TIMER
     18.33 ±183%     -96.8%       0.58 ±247%     -70.9%       5.33 ±282%  softirqs.CPU81.BLOCK
      0.11 ±282%   +1175.0%       1.42 ±245%  +54100.0%      60.22 ±261%  softirqs.CPU81.NET_RX
      0.00          -100.0%       0.00          -100.0%       0.00        softirqs.CPU81.NET_TX
     21889 ± 20%     +17.1%      25637 ± 56%     -14.9%      18619 ± 21%  softirqs.CPU81.RCU
     50839 ±  4%      +0.2%      50926 ±  5%      -6.1%      47717 ±  4%  softirqs.CPU81.SCHED
      0.11 ±282%     -25.0%       0.08 ±331%    -100.0%       0.00        softirqs.CPU81.TASKLET
     99.00 ± 87%     -49.6%      49.92 ±114%     -39.2%      60.22 ± 75%  softirqs.CPU81.TIMER
      9.89 ±193%     -99.2%       0.08 ±331%     -77.5%       2.22 ±223%  softirqs.CPU82.BLOCK
      0.11 ±282%   +1250.0%       1.50 ±312%  +51200.0%      57.00 ±282%  softirqs.CPU82.NET_RX
     22805 ± 20%     +16.6%      26581 ± 55%     -16.3%      19076 ± 22%  softirqs.CPU82.RCU
     51203 ±  3%      +1.5%      51948 ±  3%      -4.2%      49036 ±  3%  softirqs.CPU82.SCHED
      0.11 ±282%   +2750.0%       3.17 ±331%    -100.0%       0.00        softirqs.CPU82.TASKLET
     79.67 ± 94%     -53.0%      37.42 ±142%     -23.7%      60.78 ± 97%  softirqs.CPU82.TIMER
     22.56 ±195%     -61.9%       8.58 ±182%     -87.7%       2.78 ±282%  softirqs.CPU83.BLOCK
     73.44 ±200%     -83.3%      12.25 ±321%     -39.2%      44.67 ±203%  softirqs.CPU83.NET_RX
     22459 ± 20%     +17.4%      26362 ± 57%     -13.4%      19459 ± 22%  softirqs.CPU83.RCU
     49530 ±  8%      +4.6%      51808 ±  2%      -1.6%      48732 ±  5%  softirqs.CPU83.SCHED
     28.44 ± 35%     +87.2%      53.25 ±135%    +105.5%      58.44 ±109%  softirqs.CPU83.TIMER
      1.89 ±166%    +147.1%       4.67 ±300%    -100.0%       0.00        softirqs.CPU84.BLOCK
      0.00          -100.0%       0.00       +1.9e+103%      18.67 ±187%  softirqs.CPU84.NET_RX
     20524 ± 20%     +12.4%      23060 ± 53%      -5.6%      19380 ± 24%  softirqs.CPU84.RCU
     50684 ±  4%      -4.9%      48189 ± 14%      -9.4%      45896 ± 14%  softirqs.CPU84.SCHED
    210.56 ±225%     -78.3%      45.75 ±123%     -62.1%      79.78 ± 92%  softirqs.CPU84.TIMER
      4.00 ±227%     +20.8%       4.83 ±257%    +180.6%      11.22 ±173%  softirqs.CPU85.BLOCK
     18.89 ±187%    +449.7%     103.83 ±222%     -95.3%       0.89 ±282%  softirqs.CPU85.NET_RX
      0.00          -100.0%       0.00       +1.1e+101%       0.11 ±282%  softirqs.CPU85.NET_TX
     22675 ± 22%     +12.8%      25586 ± 56%     -17.4%      18722 ± 24%  softirqs.CPU85.RCU
     50500 ±  4%      +2.5%      51753 ±  3%      -4.6%      48159 ±  3%  softirqs.CPU85.SCHED
      0.00          -100.0%       0.00       +5.2e+102%       5.22 ±276%  softirqs.CPU85.TASKLET
    192.44 ±212%     -55.7%      85.33 ±162%     -56.6%      83.44 ±130%  softirqs.CPU85.TIMER
     28.78 ±205%     -95.1%       1.42 ±183%     -20.5%      22.89 ±154%  softirqs.CPU86.BLOCK
      0.00       +9.4e+103%      93.58 ±239%    -100.0%       0.00        softirqs.CPU86.NET_RX
     23468 ± 23%     +17.7%      27632 ± 59%     -18.7%      19079 ± 22%  softirqs.CPU86.RCU
     51158 ±  3%      +1.5%      51945 ±  2%      -3.2%      49525 ±  2%  softirqs.CPU86.SCHED
      1.67 ±282%     -95.0%       0.08 ±331%     -93.3%       0.11 ±282%  softirqs.CPU86.TASKLET
    112.44 ±145%     -48.3%      58.17 ±122%     -30.9%      77.67 ± 81%  softirqs.CPU86.TIMER
      5.11 ±179%     +67.9%       8.58 ±293%    +154.3%      13.00 ±259%  softirqs.CPU87.BLOCK
     19.22 ±282%     -92.2%       1.50 ±312%     -80.9%       3.67 ±282%  softirqs.CPU87.NET_RX
      0.00          -100.0%       0.00       +3.3e+101%       0.33 ±282%  softirqs.CPU87.NET_TX
     20720 ± 19%     +17.6%      24371 ± 55%      -8.6%      18935 ± 20%  softirqs.CPU87.RCU
     50865 ±  3%      +1.5%      51645 ±  3%      -3.1%      49272 ±  2%  softirqs.CPU87.SCHED
      0.00          -100.0%       0.00       +3.3e+102%       3.33 ±252%  softirqs.CPU87.TASKLET
    181.33 ±161%     -49.3%      92.00 ± 79%     +25.9%     228.22 ±216%  softirqs.CPU87.TIMER
      7.78 ±225%      -6.8%       7.25 ±278%     -67.1%       2.56 ±282%  softirqs.CPU88.BLOCK
    287.44 ±282%     -81.0%      54.58 ±297%     -90.9%      26.22 ±282%  softirqs.CPU88.NET_RX
      0.22 ±282%    -100.0%       0.00          -100.0%       0.00        softirqs.CPU88.NET_TX
     23299 ± 20%     +14.6%      26695 ± 54%     -18.0%      19106 ± 24%  softirqs.CPU88.RCU
     51078 ±  4%      +1.6%      51904 ±  2%      -4.8%      48622 ±  9%  softirqs.CPU88.SCHED
      0.00          -100.0%       0.00       +2.1e+102%       2.11 ±282%  softirqs.CPU88.TASKLET
     60.56 ±120%    +252.2%     213.25 ±226%    +292.1%     237.44 ±220%  softirqs.CPU88.TIMER
      5.33 ±243%     -25.0%       4.00 ±288%     -85.4%       0.78 ±240%  softirqs.CPU89.BLOCK
      6.67 ±247%    +271.2%      24.75 ±331%     -96.7%       0.22 ±187%  softirqs.CPU89.NET_RX
      0.00          -100.0%       0.00          -100.0%       0.00        softirqs.CPU89.NET_TX
     23476 ± 24%     +17.6%      27608 ± 57%     -16.3%      19639 ± 24%  softirqs.CPU89.RCU
     49036 ± 14%      +5.6%      51782 ±  3%      +0.5%      49280 ±  2%  softirqs.CPU89.SCHED
      0.00          -100.0%       0.00       +4.4e+101%       0.44 ±282%  softirqs.CPU89.TASKLET
    128.11 ±109%     -45.8%      69.42 ± 93%     -41.8%      74.56 ± 93%  softirqs.CPU89.TIMER
      8.44 ±188%    -100.0%       0.00           -68.4%       2.67 ±200%  softirqs.CPU9.BLOCK
     72.44 ±230%    +685.1%     568.75 ±226%     -99.4%       0.44 ±111%  softirqs.CPU9.NET_RX
      0.33 ±200%     -50.0%       0.17 ±223%     -66.7%       0.11 ±282%  softirqs.CPU9.NET_TX
     20699 ± 24%     +15.2%      23836 ± 41%      +2.1%      21140 ± 21%  softirqs.CPU9.RCU
     50649 ±  2%      +3.6%      52473            -2.9%      49177 ±  8%  softirqs.CPU9.SCHED
      2.89 ±216%     -39.4%       1.75 ±298%    -100.0%       0.00        softirqs.CPU9.TASKLET
     92.22 ± 78%   +1024.8%       1037 ±242%     -23.3%      70.78 ± 75%  softirqs.CPU9.TIMER
      8.11 ±261%     +54.1%      12.50 ±319%     -95.9%       0.33 ±200%  softirqs.CPU90.BLOCK
    359.11 ±282%   +1414.5%       5438 ±331%    -100.0%       0.00        softirqs.CPU90.NET_RX
     19034 ± 22%      +6.5%      20270 ± 49%      -1.0%      18843 ± 22%  softirqs.CPU90.RCU
     50884 ±  4%      +0.9%      51339 ±  2%      -5.3%      48199 ±  3%  softirqs.CPU90.SCHED
      0.00       +8.3e+100%       0.08 ±331% +1.1e+101%       0.11 ±282%  softirqs.CPU90.TASKLET
     95.67 ±125%     -13.3%      82.92 ± 80%     -44.4%      53.22 ±117%  softirqs.CPU90.TIMER
      0.56 ±282%   +1640.0%       9.67 ±214%   +5420.0%      30.67 ±226%  softirqs.CPU91.BLOCK
      6891 ±282%    -100.0%       0.08 ±331%    -100.0%       0.00        softirqs.CPU91.NET_RX
      0.33 ±199%    -100.0%       0.00          -100.0%       0.00        softirqs.CPU91.NET_TX
     19717 ± 20%     +15.3%      22737 ± 52%      -5.3%      18663 ± 21%  softirqs.CPU91.RCU
     51094 ±  3%      +1.3%      51781 ±  2%      -2.9%      49634 ±  4%  softirqs.CPU91.SCHED
      0.00       +8.3e+100%       0.08 ±331% +1.1e+101%       0.11 ±282%  softirqs.CPU91.TASKLET
    131.78 ±134%     +45.1%     191.25 ±222%     +60.6%     211.67 ±217%  softirqs.CPU91.TIMER
      0.22 ±282%   +2712.5%       6.25 ±229%   +3600.0%       8.22 ±261%  softirqs.CPU92.BLOCK
      0.33 ±282%     -25.0%       0.25 ±331%   +6000.0%      20.33 ±272%  softirqs.CPU92.NET_RX
     23495 ± 23%     +16.3%      27316 ± 57%     -16.7%      19581 ± 25%  softirqs.CPU92.RCU
     51247 ±  2%      -2.5%      49961 ± 12%      -5.7%      48323 ±  8%  softirqs.CPU92.SCHED
     74.78 ±122%     -41.9%      43.42 ± 75%     -38.2%      46.22 ±134%  softirqs.CPU92.TIMER
     16.89 ±280%     -97.5%       0.42 ±331%     -80.3%       3.33 ±251%  softirqs.CPU93.BLOCK
      1368 ±206%     -83.4%     227.08 ±327%    -100.0%       0.00        softirqs.CPU93.NET_RX
     22947 ± 24%     +16.2%      26665 ± 57%     -17.3%      18973 ± 23%  softirqs.CPU93.RCU
     50793 ±  3%      +1.9%      51751 ±  2%      -4.0%      48766 ±  3%  softirqs.CPU93.SCHED
      0.00       +1.3e+102%       1.33 ±331% +1.6e+102%       1.56 ±260%  softirqs.CPU93.TASKLET
    360.44 ±227%     -77.2%      82.08 ±193%     -90.4%      34.67 ± 83%  softirqs.CPU93.TIMER
      0.78 ±169%     -25.0%       0.58 ±203%    +200.0%       2.33 ±198%  softirqs.CPU94.BLOCK
     23193 ± 23%     +16.4%      26990 ± 57%     -16.6%      19335 ± 23%  softirqs.CPU94.RCU
     51088 ±  3%      +1.8%      52010 ±  2%      -4.0%      49067 ±  2%  softirqs.CPU94.SCHED
      0.00       +1.7e+101%       0.17 ±223% +4.8e+102%       4.78 ±254%  softirqs.CPU94.TASKLET
     76.78 ±122%    +233.1%     255.75 ±202%     +20.3%      92.33 ±145%  softirqs.CPU94.TIMER
      6.89 ±244%     +40.3%       9.67 ±192%     -35.5%       4.44 ±195%  softirqs.CPU95.BLOCK
      0.22 ±187%   +1437.5%       3.42 ±305%      +0.0%       0.22 ±187%  softirqs.CPU95.NET_TX
     22570 ± 19%     +16.1%      26201 ± 52%     -13.0%      19642 ± 24%  softirqs.CPU95.RCU
     41358 ± 28%     -15.6%      34889 ± 27%      -2.6%      40276 ± 25%  softirqs.CPU95.SCHED
      3.11 ±196%    -100.0%       0.00          -100.0%       0.00        softirqs.CPU95.TASKLET
     71.89 ± 61%    +450.6%     395.83 ±142%    +120.7%     158.67 ± 86%  softirqs.CPU95.TIMER
      2.00            +0.0%       2.00            +0.0%       2.00        softirqs.HI
     32269 ± 81%     -15.2%      27360 ±102%      -3.4%      31156 ± 92%  softirqs.NET_RX
     59.44 ±  2%      -2.6%      57.92 ±  2%      +3.6%      61.56 ±  8%  softirqs.NET_TX
   1949585 ± 18%     +14.8%    2237679 ± 53%      -5.0%    1852982 ± 21%  softirqs.RCU
   4756610            +1.6%    4831530 ±  2%      -0.3%    4741884        softirqs.SCHED
    860.44            -0.0%     860.42            -0.2%     858.44        softirqs.TASKLET
     30885 ±  9%      +4.3%      32212 ± 27%      -2.9%      29995 ± 13%  softirqs.TIMER
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.0:IO-APIC.2-edge.timer
     48.11 ±282%     -32.4%      32.50 ±264%     -92.4%       3.67 ±254%  interrupts.100:PCI-MSI.31981633-edge.i40e-eth0-TxRx-64
     16.33 ±192%    +428.1%      86.25 ±293%     -12.2%      14.33 ±208%  interrupts.101:PCI-MSI.31981634-edge.i40e-eth0-TxRx-65
     28.11 ±282%    +161.2%      73.42 ±330%     -32.4%      19.00 ±262%  interrupts.102:PCI-MSI.31981635-edge.i40e-eth0-TxRx-66
     18.67 ±152%    +954.9%     196.92 ±307%     -97.6%       0.44 ±187%  interrupts.103:PCI-MSI.31981636-edge.i40e-eth0-TxRx-67
     74.67 ±251%     -90.4%       7.17 ±331%    +108.2%     155.44 ±282%  interrupts.104:PCI-MSI.31981637-edge.i40e-eth0-TxRx-68
      0.00       +8.6e+102%       8.58 ±229%   +1e+104%     100.11 ±282%  interrupts.105:PCI-MSI.31981638-edge.i40e-eth0-TxRx-69
      4.33 ±282%  +32007.7%       1391 ±330%   +2859.0%     128.22 ±198%  interrupts.106:PCI-MSI.31981639-edge.i40e-eth0-TxRx-70
      3.78 ±264%  +92041.9%       3480 ±319%   +5494.1%     211.33 ±282%  interrupts.107:PCI-MSI.31981640-edge.i40e-eth0-TxRx-71
     13.44 ±200%     -54.1%       6.17 ±307%  +53891.7%       7258 ±282%  interrupts.108:PCI-MSI.31981641-edge.i40e-eth0-TxRx-72
    189.00 ±282%     -61.3%      73.08 ±331%     -70.5%      55.67 ±250%  interrupts.109:PCI-MSI.31981642-edge.i40e-eth0-TxRx-73
    103.56 ±282%     -87.7%      12.75 ±243%     -14.3%      88.78 ±279%  interrupts.110:PCI-MSI.31981643-edge.i40e-eth0-TxRx-74
      0.44 ±187%     -25.0%       0.33 ±223%  +29050.0%     129.56 ±242%  interrupts.111:PCI-MSI.31981644-edge.i40e-eth0-TxRx-75
      8.78 ±274%    +861.7%      84.42 ±331%    +946.8%      91.89 ±282%  interrupts.112:PCI-MSI.31981645-edge.i40e-eth0-TxRx-76
      3.33 ±282%    +955.0%      35.17 ±306%  +4.7e+05%      15833 ±251%  interrupts.113:PCI-MSI.31981646-edge.i40e-eth0-TxRx-77
      0.00       +3.8e+102%       3.83 ±331% +1.7e+102%       1.67 ±262%  interrupts.114:PCI-MSI.31981647-edge.i40e-eth0-TxRx-78
    573.11 ±161%     -71.8%     161.67 ±302%    -100.0%       0.00        interrupts.115:PCI-MSI.31981648-edge.i40e-eth0-TxRx-79
     35.11 ±280%      -1.3%      34.67 ±325%    +295.9%     139.00 ±271%  interrupts.116:PCI-MSI.31981649-edge.i40e-eth0-TxRx-80
      0.44 ±187%    +593.8%       3.08 ±238%  +26925.0%     120.11 ±262%  interrupts.117:PCI-MSI.31981650-edge.i40e-eth0-TxRx-81
      0.22 ±282%   +1100.0%       2.67 ±309%  +50000.0%     111.33 ±281%  interrupts.118:PCI-MSI.31981651-edge.i40e-eth0-TxRx-82
    144.56 ±200%     -83.0%      24.58 ±321%     -37.8%      89.89 ±202%  interrupts.119:PCI-MSI.31981652-edge.i40e-eth0-TxRx-83
      0.22 ±282%    +200.0%       0.67 ±141%  +15750.0%      35.22 ±184%  interrupts.120:PCI-MSI.31981653-edge.i40e-eth0-TxRx-84
     34.78 ±187%    +455.4%     193.17 ±218%     -99.4%       0.22 ±282%  interrupts.121:PCI-MSI.31981654-edge.i40e-eth0-TxRx-85
      0.00       +1.8e+104%     176.08 ±243% +1.1e+101%       0.11 ±282%  interrupts.122:PCI-MSI.31981655-edge.i40e-eth0-TxRx-86
     38.56 ±282%     -93.5%       2.50 ±331%     -79.8%       7.78 ±269%  interrupts.123:PCI-MSI.31981656-edge.i40e-eth0-TxRx-87
    564.11 ±282%     -80.6%     109.17 ±298%     -90.7%      52.56 ±282%  interrupts.124:PCI-MSI.31981657-edge.i40e-eth0-TxRx-88
     11.44 ±240%    +334.7%      49.75 ±330%     -95.1%       0.56 ±191%  interrupts.125:PCI-MSI.31981658-edge.i40e-eth0-TxRx-89
    538.78 ±282%   +1919.0%      10877 ±331%    -100.0%       0.22 ±187%  interrupts.126:PCI-MSI.31981659-edge.i40e-eth0-TxRx-90
     13770 ±282%    -100.0%       0.08 ±331%    -100.0%       0.00        interrupts.127:PCI-MSI.31981660-edge.i40e-eth0-TxRx-91
      0.89 ±215%     -62.5%       0.33 ±223%   +4400.0%      40.00 ±279%  interrupts.128:PCI-MSI.31981661-edge.i40e-eth0-TxRx-92
      2715 ±206%     -83.3%     454.17 ±327%    -100.0%       0.00        interrupts.129:PCI-MSI.31981662-edge.i40e-eth0-TxRx-93
    382.00            +0.1%     382.33            +0.2%     382.89        interrupts.293:PCI-MSI.327680-edge.xhci_hcd
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.295:PCI-MSI.65536-edge.ioat-msix
      0.00       +3.2e+103%      31.83 ±223%    -100.0%       0.00        interrupts.296:PCI-MSI.288768-edge.ahci[0000:00:11.5]
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.297:PCI-MSI.67584-edge.ioat-msix
     21.22 ±282%    -100.0%       0.00          -100.0%       0.00        interrupts.298:PCI-MSI.288768-edge.ahci[0000:00:11.5]
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.298:PCI-MSI.67584-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.298:PCI-MSI.69632-edge.ioat-msix
      0.00       +1.6e+103%      15.92 ±331%    -100.0%       0.00        interrupts.299:PCI-MSI.288768-edge.ahci[0000:00:11.5]
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.299:PCI-MSI.69632-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.299:PCI-MSI.71680-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.29:PCI-MSI.48791552-edge.PCIe.PME,pciehp
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.300:PCI-MSI.71680-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.300:PCI-MSI.73728-edge.ioat-msix
      0.00          -100.0%       0.00       +2.1e+103%      21.22 ±282%  interrupts.301:PCI-MSI.288768-edge.ahci[0000:00:11.5]
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.301:PCI-MSI.73728-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.301:PCI-MSI.75776-edge.ioat-msix
      0.00       +1.6e+103%      15.92 ±331%    -100.0%       0.00        interrupts.302:PCI-MSI.288768-edge.ahci[0000:00:11.5]
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.302:PCI-MSI.75776-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.302:PCI-MSI.77824-edge.ioat-msix
     21.33 ±282%    -100.0%       0.00            -0.5%      21.22 ±282%  interrupts.303:PCI-MSI.288768-edge.ahci[0000:00:11.5]
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.303:PCI-MSI.77824-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.303:PCI-MSI.79872-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.304:PCI-MSI.79872-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.305:PCI-MSI.67174400-edge.ioat-msix
     63.67 ±141%     -25.0%      47.75 ±173%     +33.3%      84.89 ±111%  interrupts.306:PCI-MSI.288768-edge.ahci[0000:00:11.5]
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.306:PCI-MSI.67174400-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.307:PCI-MSI.376832-edge.ahci[0000:00:17.0]
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.307:PCI-MSI.67176448-edge.ioat-msix
     21.22 ±282%     +50.0%      31.83 ±223%    -100.0%       0.00        interrupts.308:PCI-MSI.288768-edge.ahci[0000:00:11.5]
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.308:PCI-MSI.67176448-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.308:PCI-MSI.67178496-edge.ioat-msix
     21.22 ±282%     -25.0%      15.92 ±331%    -100.0%       0.00        interrupts.309:PCI-MSI.288768-edge.ahci[0000:00:11.5]
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.309:PCI-MSI.67176448-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.309:PCI-MSI.67178496-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.309:PCI-MSI.67180544-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.30:PCI-MSI.48807936-edge.PCIe.PME,pciehp
      0.00       +1.6e+103%      15.92 ±331% +2.1e+103%      21.22 ±282%  interrupts.310:PCI-MSI.288768-edge.ahci[0000:00:11.5]
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.310:PCI-MSI.67178496-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.310:PCI-MSI.67180544-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.310:PCI-MSI.67182592-edge.ioat-msix
      0.00          -100.0%       0.00       +2.1e+103%      21.22 ±282%  interrupts.311:PCI-MSI.288768-edge.ahci[0000:00:11.5]
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.311:PCI-MSI.67180544-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.311:PCI-MSI.67182592-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.311:PCI-MSI.67184640-edge.ioat-msix
     21.22 ±282%    -100.0%       0.00          -100.0%       0.00        interrupts.312:PCI-MSI.288768-edge.ahci[0000:00:11.5]
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.312:PCI-MSI.67182592-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.312:PCI-MSI.67184640-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.312:PCI-MSI.67186688-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.313:PCI-MSI.67184640-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.313:PCI-MSI.67186688-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.313:PCI-MSI.67188736-edge.ioat-msix
     21.22 ±282%     -25.0%      15.92 ±331%      +0.5%      21.33 ±282%  interrupts.314:PCI-MSI.288768-edge.ahci[0000:00:11.5]
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.314:PCI-MSI.67186688-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.314:PCI-MSI.67188736-edge.ioat-msix
     43.89 ± 89%    -100.0%       0.00           -81.5%       8.11 ±282%  interrupts.315:PCI-MSI.376832-edge.ahci[0000:00:17.0]
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.315:PCI-MSI.67188736-edge.ioat-msix
    969.11 ±263%     -89.3%     103.42 ±316%     -99.8%       2.22 ± 89%  interrupts.36:PCI-MSI.31981569-edge.i40e-eth0-TxRx-0
      3070 ±274%     -99.9%       3.33 ±314%     -99.7%       9.89 ±162%  interrupts.37:PCI-MSI.31981570-edge.i40e-eth0-TxRx-1
      3.89 ±240%   +2315.0%      93.92 ±311%   +4357.1%     173.33 ±261%  interrupts.38:PCI-MSI.31981571-edge.i40e-eth0-TxRx-2
     38.44 ±231%     -90.9%       3.50 ±331%    +181.8%     108.33 ±270%  interrupts.39:PCI-MSI.31981572-edge.i40e-eth0-TxRx-3
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.3:IO-APIC.3-edge
     34.78 ±189%    +482.5%     202.58 ±304%     -99.4%       0.22 ±282%  interrupts.40:PCI-MSI.31981573-edge.i40e-eth0-TxRx-4
      2667 ±282%     -97.6%      63.42 ±325%     -99.8%       5.78 ±282%  interrupts.41:PCI-MSI.31981574-edge.i40e-eth0-TxRx-5
     11109 ±282%     -99.2%      84.50 ±317%     -98.4%     176.56 ±165%  interrupts.42:PCI-MSI.31981575-edge.i40e-eth0-TxRx-6
      1.44 ±282%    +690.4%      11.42 ±331%    +507.7%       8.78 ±150%  interrupts.43:PCI-MSI.31981576-edge.i40e-eth0-TxRx-7
      2.00 ±265%     -83.3%       0.33 ±223%   +5216.7%     106.33 ±255%  interrupts.44:PCI-MSI.31981577-edge.i40e-eth0-TxRx-8
    143.78 ±232%    +416.9%     743.25 ±221%     -99.9%       0.11 ±282%  interrupts.45:PCI-MSI.31981578-edge.i40e-eth0-TxRx-9
    206.67 ±157%     -91.1%      18.33 ±224%     -28.0%     148.89 ±280%  interrupts.46:PCI-MSI.31981579-edge.i40e-eth0-TxRx-10
     64.78 ±206%     -10.2%      58.17 ±330%     -76.5%      15.22 ±201%  interrupts.47:PCI-MSI.31981580-edge.i40e-eth0-TxRx-11
      6.78 ±205%   +7985.2%     548.00 ±331%  +2.1e+05%      14116 ±271%  interrupts.48:PCI-MSI.31981581-edge.i40e-eth0-TxRx-12
      6939 ±277%     -99.5%      31.25 ±286%     -99.4%      42.67 ±239%  interrupts.49:PCI-MSI.31981582-edge.i40e-eth0-TxRx-13
     49.33 ±  6%      -0.3%      49.17 ±  6%      -1.1%      48.78 ±  5%  interrupts.4:IO-APIC.4-edge.ttyS0
    158.44 ±195%   +2058.4%       3419 ±329%     -84.0%      25.33 ±278%  interrupts.50:PCI-MSI.31981583-edge.i40e-eth0-TxRx-14
     34.11 ±282%     +60.5%      54.75 ±221%    +225.7%     111.11 ±212%  interrupts.51:PCI-MSI.31981584-edge.i40e-eth0-TxRx-15
      5.00 ±282%      -1.7%       4.92 ±224%   +3397.8%     174.89 ±250%  interrupts.52:PCI-MSI.31981585-edge.i40e-eth0-TxRx-16
     17.33 ±187%    +499.0%     103.83 ±326%     +41.0%      24.44 ±200%  interrupts.53:PCI-MSI.31981586-edge.i40e-eth0-TxRx-17
     21.11 ±282%     +64.6%      34.75 ±314%    +266.8%      77.44 ±274%  interrupts.54:PCI-MSI.31981587-edge.i40e-eth0-TxRx-18
     69.67 ±280%  +10011.4%       7044 ±331%     -88.7%       7.89 ±187%  interrupts.55:PCI-MSI.31981588-edge.i40e-eth0-TxRx-19
      1.56 ±282%  +11032.1%     173.17 ±226%   +2428.6%      39.33 ±281%  interrupts.56:PCI-MSI.31981589-edge.i40e-eth0-TxRx-20
      2.00 ±174%    +216.7%       6.33 ±286%     +83.3%       3.67 ±282%  interrupts.57:PCI-MSI.31981590-edge.i40e-eth0-TxRx-21
      0.00       +8.4e+103%      83.67 ±296% +2.8e+102%       2.78 ±258%  interrupts.58:PCI-MSI.31981591-edge.i40e-eth0-TxRx-22
     66.67 ±231%      -6.6%      62.25 ±285%    +767.5%     578.33 ±239%  interrupts.59:PCI-MSI.31981592-edge.i40e-eth0-TxRx-23
     54.78 ±213%    +314.1%     226.83 ±238%    +235.9%     184.00 ±242%  interrupts.60:PCI-MSI.31981593-edge.i40e-eth0-TxRx-24
      1582 ±282%     -99.2%      13.25 ±327%     -98.8%      19.67 ±186%  interrupts.61:PCI-MSI.31981594-edge.i40e-eth0-TxRx-25
      0.00       +2.8e+102%       2.83 ±245% +2.2e+101%       0.22 ±282%  interrupts.62:PCI-MSI.31981595-edge.i40e-eth0-TxRx-26
     63.22 ±281%     -83.5%      10.42 ±331%     -87.9%       7.67 ±264%  interrupts.63:PCI-MSI.31981596-edge.i40e-eth0-TxRx-27
    210.89 ±257%     -93.3%      14.17 ±304%     -21.5%     165.56 ±281%  interrupts.64:PCI-MSI.31981597-edge.i40e-eth0-TxRx-28
      9.00 ±267%     -63.9%       3.25 ±313%    +807.4%      81.67 ±261%  interrupts.65:PCI-MSI.31981598-edge.i40e-eth0-TxRx-29
      0.00       +1.8e+103%      18.33 ±174% +1.8e+103%      17.67 ±278%  interrupts.66:PCI-MSI.31981599-edge.i40e-eth0-TxRx-30
     14016 ±282%     -97.8%     306.33 ±202%     -99.9%      11.44 ±270%  interrupts.67:PCI-MSI.31981600-edge.i40e-eth0-TxRx-31
    102.11 ±233%    +177.2%     283.08 ±127%     -99.8%       0.22 ±282%  interrupts.68:PCI-MSI.31981601-edge.i40e-eth0-TxRx-32
    219.67 ±193%     -95.8%       9.17 ±262%    -100.0%       0.00        interrupts.69:PCI-MSI.31981602-edge.i40e-eth0-TxRx-33
     11.78 ±242%    +698.8%      94.08 ±331%     -92.5%       0.89 ±215%  interrupts.70:PCI-MSI.31981603-edge.i40e-eth0-TxRx-34
      2.11 ±251%    +136.8%       5.00 ±228%  +18221.1%     386.78 ±282%  interrupts.71:PCI-MSI.31981604-edge.i40e-eth0-TxRx-35
    207.78 ±195%     -61.7%      79.58 ±331%    +793.1%       1855 ±272%  interrupts.72:PCI-MSI.31981605-edge.i40e-eth0-TxRx-36
      0.22 ±282%  +1.7e+05%     378.75 ±315%  +12000.0%      26.89 ±194%  interrupts.73:PCI-MSI.31981606-edge.i40e-eth0-TxRx-37
    182.33 ±265%    +276.7%     686.92 ±327%     -48.2%      94.44 ±282%  interrupts.74:PCI-MSI.31981607-edge.i40e-eth0-TxRx-38
    344.67 ±273%     -89.5%      36.25 ±319%     -51.7%     166.56 ±282%  interrupts.75:PCI-MSI.31981608-edge.i40e-eth0-TxRx-39
    101.89 ±282%     +15.3%     117.50 ±331%     +27.0%     129.44 ±282%  interrupts.76:PCI-MSI.31981609-edge.i40e-eth0-TxRx-40
     37.89 ±223%     -94.3%       2.17 ±278%     +91.2%      72.44 ±244%  interrupts.77:PCI-MSI.31981610-edge.i40e-eth0-TxRx-41
     16.67 ±278%    +357.0%      76.17 ±319%     -77.3%       3.78 ±273%  interrupts.78:PCI-MSI.31981611-edge.i40e-eth0-TxRx-42
      0.00       +8.5e+102%       8.50 ±324% +1.2e+103%      12.22 ±210%  interrupts.79:PCI-MSI.31981612-edge.i40e-eth0-TxRx-43
     33.78 ±280%    +112.2%      71.67 ±323%    +195.4%      99.78 ±282%  interrupts.80:PCI-MSI.31981613-edge.i40e-eth0-TxRx-44
      0.00       +1.7e+104%     165.42 ±254% +1.7e+103%      17.44 ±282%  interrupts.81:PCI-MSI.31981614-edge.i40e-eth0-TxRx-45
    108.67 ±252%      +9.2%     118.67 ±234%     -98.2%       2.00 ±282%  interrupts.82:PCI-MSI.31981615-edge.i40e-eth0-TxRx-46
     11.33 ±245%     +80.1%      20.42 ±223%     -95.1%       0.56 ±226%  interrupts.83:PCI-MSI.31981616-edge.i40e-eth0-TxRx-47
    134.22 ±273%     -99.8%       0.33 ±223%     -99.5%       0.67 ±200%  interrupts.84:PCI-MSI.31981617-edge.i40e-eth0-TxRx-48
     56.56 ±277%  +20032.9%      11386 ±233%     -94.5%       3.11 ±162%  interrupts.85:PCI-MSI.31981618-edge.i40e-eth0-TxRx-49
     18.67 ±262%   +1340.2%     268.83 ±324%     +60.7%      30.00 ±282%  interrupts.86:PCI-MSI.31981619-edge.i40e-eth0-TxRx-50
    114.33 ±278%     +23.7%     141.42 ±195%     +37.4%     157.11 ±236%  interrupts.87:PCI-MSI.31981620-edge.i40e-eth0-TxRx-51
      1.44 ±193%   +8196.2%     119.83 ±295%    +284.6%       5.56 ±119%  interrupts.88:PCI-MSI.31981621-edge.i40e-eth0-TxRx-52
      5.22 ±282%     -18.6%       4.25 ±227%     -42.6%       3.00 ±282%  interrupts.89:PCI-MSI.31981622-edge.i40e-eth0-TxRx-53
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.8:IO-APIC.8-edge.rtc0
      9.89 ±234%    +852.2%      94.17 ±329%     -66.3%       3.33 ±282%  interrupts.90:PCI-MSI.31981623-edge.i40e-eth0-TxRx-54
      0.56 ±149%   +2450.0%      14.17 ±179%  +23960.0%     133.67 ±282%  interrupts.91:PCI-MSI.31981624-edge.i40e-eth0-TxRx-55
     33.78 ±282%     -92.6%       2.50 ±308%  +43009.5%      14561 ±282%  interrupts.92:PCI-MSI.31981625-edge.i40e-eth0-TxRx-56
    289.89 ±219%     -99.4%       1.67 ±297%     -96.7%       9.67 ±275%  interrupts.93:PCI-MSI.31981626-edge.i40e-eth0-TxRx-57
      0.00       +3.5e+104%     349.58 ±315% +1.2e+104%     115.44 ±238%  interrupts.94:PCI-MSI.31981627-edge.i40e-eth0-TxRx-58
      0.00       +4.1e+103%      40.75 ±288% +2.6e+103%      26.11 ±187%  interrupts.95:PCI-MSI.31981628-edge.i40e-eth0-TxRx-59
    281.22 ±281%     -64.3%     100.42 ±296%      +7.5%     302.22 ±141%  interrupts.96:PCI-MSI.31981629-edge.i40e-eth0-TxRx-60
    201.89 ±279%     -26.8%     147.75 ±323%     -97.2%       5.67 ±162%  interrupts.97:PCI-MSI.31981630-edge.i40e-eth0-TxRx-61
    349.11 ±217%    +161.4%     912.42 ±195%     -96.9%      10.89 ±276%  interrupts.98:PCI-MSI.31981631-edge.i40e-eth0-TxRx-62
    178.22 ±268%     +48.2%     264.17 ±242%   +1035.8%       2024 ±282%  interrupts.99:PCI-MSI.31981632-edge.i40e-eth0-TxRx-63
     95682 ± 20%      -2.6%      93152 ± 26%     +35.9%     130063 ± 14%  interrupts.CAL:Function_call_interrupts
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU0.0:IO-APIC.2-edge.timer
      0.22 ±187%     +12.5%       0.25 ±173%     +50.0%       0.33 ±141%  interrupts.CPU0.119:PCI-MSI.31981652-edge.i40e-eth0-TxRx-83
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU0.301:PCI-MSI.75776-edge.ioat-msix
    969.00 ±263%     -89.3%     103.25 ±316%     -99.8%       2.11 ± 95%  interrupts.CPU0.36:PCI-MSI.31981569-edge.i40e-eth0-TxRx-0
      0.11 ±282%    -100.0%       0.00          +100.0%       0.22 ±187%  interrupts.CPU0.71:PCI-MSI.31981604-edge.i40e-eth0-TxRx-35
      1946 ±166%     -55.4%     867.75 ±116%     +47.5%       2872 ±190%  interrupts.CPU0.CAL:Function_call_interrupts
    774652 ±  3%      -6.7%     722676 ± 18%      +1.2%     783713        interrupts.CPU0.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU0.MCP:Machine_check_polls
     90.56 ± 37%    +144.6%     221.50 ±189%     -14.0%      77.89 ± 12%  interrupts.CPU0.NMI:Non-maskable_interrupts
     90.56 ± 37%    +144.6%     221.50 ±189%     -14.0%      77.89 ± 12%  interrupts.CPU0.PMI:Performance_monitoring_interrupts
     13.00 ± 51%    +241.0%      44.33 ±172%      +5.1%      13.67 ± 73%  interrupts.CPU0.RES:Rescheduling_interrupts
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU0.RTR:APIC_ICR_read_retries
      1.89 ± 76%     +54.4%       2.92 ±116%     +88.2%       3.56 ± 44%  interrupts.CPU0.TLB:TLB_shootdowns
      0.11 ±282%    +200.0%       0.33 ±141%    +200.0%       0.33 ±141%  interrupts.CPU1.120:PCI-MSI.31981653-edge.i40e-eth0-TxRx-84
      3070 ±274%     -99.9%       3.25 ±322%     -99.7%       9.56 ±166%  interrupts.CPU1.37:PCI-MSI.31981570-edge.i40e-eth0-TxRx-1
      0.33 ±141%    -100.0%       0.00           -33.3%       0.22 ±187%  interrupts.CPU1.72:PCI-MSI.31981605-edge.i40e-eth0-TxRx-36
    547.78 ± 20%     +72.0%     942.33 ± 69%    +840.1%       5149 ±248%  interrupts.CPU1.CAL:Function_call_interrupts
      0.00       +8.3e+100%       0.08 ±331%    -100.0%       0.00        interrupts.CPU1.IWI:IRQ_work_interrupts
    774552 ±  3%      -6.7%     722476 ± 18%      +1.2%     783803        interrupts.CPU1.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU1.MCP:Machine_check_polls
     84.33 ± 26%    +611.5%     600.00 ±238%    +157.2%     216.89 ±188%  interrupts.CPU1.NMI:Non-maskable_interrupts
     84.33 ± 26%    +611.5%     600.00 ±238%    +157.2%     216.89 ±188%  interrupts.CPU1.PMI:Performance_monitoring_interrupts
     22.22 ± 49%      +1.3%      22.50 ± 62%     -39.0%      13.56 ± 72%  interrupts.CPU1.RES:Rescheduling_interrupts
      1.67 ± 74%      +0.0%       1.67 ±143%    +120.0%       3.67 ± 38%  interrupts.CPU1.TLB:TLB_shootdowns
      0.11 ±282%     +50.0%       0.17 ±223%    -100.0%       0.00        interrupts.CPU10.129:PCI-MSI.31981662-edge.i40e-eth0-TxRx-93
    382.00            +0.1%     382.33            +0.2%     382.89        interrupts.CPU10.293:PCI-MSI.327680-edge.xhci_hcd
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU10.298:PCI-MSI.69632-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU10.315:PCI-MSI.376832-edge.ahci[0000:00:17.0]
    206.44 ±157%     -91.1%      18.33 ±224%     -28.0%     148.67 ±280%  interrupts.CPU10.46:PCI-MSI.31981579-edge.i40e-eth0-TxRx-10
      0.00       +8.3e+100%       0.08 ±331%    -100.0%       0.00        interrupts.CPU10.81:PCI-MSI.31981614-edge.i40e-eth0-TxRx-45
    564.33 ±  9%      +0.3%     565.92 ±  8%    +190.5%       1639 ±162%  interrupts.CPU10.CAL:Function_call_interrupts
    774527 ±  3%      -6.7%     722599 ± 18%      +1.2%     783661        interrupts.CPU10.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU10.MCP:Machine_check_polls
     94.44 ± 30%      -1.2%      93.33 ± 36%     -17.9%      77.56 ± 10%  interrupts.CPU10.NMI:Non-maskable_interrupts
     94.44 ± 30%      -1.2%      93.33 ± 36%     -17.9%      77.56 ± 10%  interrupts.CPU10.PMI:Performance_monitoring_interrupts
     30.67 ± 80%   +2062.2%     663.08 ±318%    +147.1%      75.78 ±214%  interrupts.CPU10.RES:Rescheduling_interrupts
      1.44 ±118%     +38.5%       2.00 ±145%    +169.2%       3.89 ± 47%  interrupts.CPU10.TLB:TLB_shootdowns
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU11.298:PCI-MSI.67584-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU11.298:PCI-MSI.69632-edge.ioat-msix
      0.00       +1.6e+103%      15.92 ±331%    -100.0%       0.00        interrupts.CPU11.299:PCI-MSI.288768-edge.ahci[0000:00:11.5]
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU11.299:PCI-MSI.69632-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU11.299:PCI-MSI.71680-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU11.300:PCI-MSI.73728-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU11.303:PCI-MSI.79872-edge.ioat-msix
     64.56 ±207%     -10.2%      58.00 ±331%     -76.8%      15.00 ±205%  interrupts.CPU11.47:PCI-MSI.31981580-edge.i40e-eth0-TxRx-11
      0.33 ±141%     -50.0%       0.17 ±223%     -66.7%       0.11 ±282%  interrupts.CPU11.82:PCI-MSI.31981615-edge.i40e-eth0-TxRx-46
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU11.8:IO-APIC.8-edge.rtc0
    825.44 ± 88%     +45.0%       1196 ±109%     -33.7%     547.00 ±  2%  interrupts.CPU11.CAL:Function_call_interrupts
      0.00       +8.3e+100%       0.08 ±331% +1.1e+101%       0.11 ±282%  interrupts.CPU11.IWI:IRQ_work_interrupts
    774519 ±  3%      -6.7%     722533 ± 18%      +1.2%     783775        interrupts.CPU11.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU11.MCP:Machine_check_polls
    108.89 ± 14%    +354.2%     494.58 ±267%    +547.0%     704.56 ±249%  interrupts.CPU11.NMI:Non-maskable_interrupts
    108.89 ± 14%    +354.2%     494.58 ±267%    +547.0%     704.56 ±249%  interrupts.CPU11.PMI:Performance_monitoring_interrupts
     27.89 ±104%     -14.5%      23.83 ± 91%     -34.7%      18.22 ± 39%  interrupts.CPU11.RES:Rescheduling_interrupts
      2.00 ± 97%     +12.5%       2.25 ±134%    +111.1%       4.22 ± 39%  interrupts.CPU11.TLB:TLB_shootdowns
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU12.295:PCI-MSI.65536-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU12.299:PCI-MSI.69632-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU12.299:PCI-MSI.71680-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU12.300:PCI-MSI.71680-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU12.300:PCI-MSI.73728-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU12.301:PCI-MSI.75776-edge.ioat-msix
     21.22 ±282%    -100.0%       0.00          -100.0%       0.00        interrupts.CPU12.306:PCI-MSI.288768-edge.ahci[0000:00:11.5]
      6.67 ±208%   +8116.2%     547.75 ±331%  +2.1e+05%      14116 ±271%  interrupts.CPU12.48:PCI-MSI.31981581-edge.i40e-eth0-TxRx-12
      5.67 ±282%     +50.0%       8.50 ±223%     +90.2%      10.78 ±187%  interrupts.CPU12.4:IO-APIC.4-edge.ttyS0
      0.00       +8.3e+100%       0.08 ±331% +1.1e+101%       0.11 ±282%  interrupts.CPU12.83:PCI-MSI.31981616-edge.i40e-eth0-TxRx-47
    557.44 ±  5%      -3.0%     540.67 ±  5%    +238.2%       1885 ±177%  interrupts.CPU12.CAL:Function_call_interrupts
    774602 ±  3%      -6.7%     722477 ± 18%      +1.2%     783722        interrupts.CPU12.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU12.MCP:Machine_check_polls
     92.67 ± 28%    +115.1%     199.33 ±180%     -19.5%      74.56 ± 18%  interrupts.CPU12.NMI:Non-maskable_interrupts
     92.67 ± 28%    +115.1%     199.33 ±180%     -19.5%      74.56 ± 18%  interrupts.CPU12.PMI:Performance_monitoring_interrupts
    200.56 ±256%     -84.9%      30.33 ± 88%     -89.8%      20.44 ± 52%  interrupts.CPU12.RES:Rescheduling_interrupts
      2.44 ±109%      -8.0%       2.25 ±117%     +77.3%       4.33 ± 42%  interrupts.CPU12.TLB:TLB_shootdowns
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU13.295:PCI-MSI.65536-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU13.298:PCI-MSI.69632-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU13.300:PCI-MSI.71680-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU13.300:PCI-MSI.73728-edge.ioat-msix
      0.00          -100.0%       0.00       +2.1e+103%      21.22 ±282%  interrupts.CPU13.301:PCI-MSI.288768-edge.ahci[0000:00:11.5]
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU13.301:PCI-MSI.73728-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU13.301:PCI-MSI.75776-edge.ioat-msix
      0.00       +1.6e+103%      15.92 ±331%    -100.0%       0.00        interrupts.CPU13.302:PCI-MSI.288768-edge.ahci[0000:00:11.5]
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU13.302:PCI-MSI.77824-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU13.315:PCI-MSI.376832-edge.ahci[0000:00:17.0]
      0.11 ±282%     +50.0%       0.17 ±223%      +0.0%       0.11 ±282%  interrupts.CPU13.36:PCI-MSI.31981569-edge.i40e-eth0-TxRx-0
      6939 ±277%     -99.6%      31.08 ±288%     -99.4%      42.56 ±240%  interrupts.CPU13.49:PCI-MSI.31981582-edge.i40e-eth0-TxRx-13
     11.67 ±187%     -27.9%       8.42 ±224%      -7.6%      10.78 ±187%  interrupts.CPU13.4:IO-APIC.4-edge.ttyS0
      0.00       +1.7e+101%       0.17 ±223% +1.1e+101%       0.11 ±282%  interrupts.CPU13.84:PCI-MSI.31981617-edge.i40e-eth0-TxRx-48
    878.00 ± 57%     -37.8%     546.50 ±  2%     -15.8%     739.11 ± 74%  interrupts.CPU13.CAL:Function_call_interrupts
      0.11 ±282%    -100.0%       0.00          -100.0%       0.00        interrupts.CPU13.IWI:IRQ_work_interrupts
    774539 ±  3%      -6.7%     722427 ± 18%      +1.2%     783671        interrupts.CPU13.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU13.MCP:Machine_check_polls
    980.44 ±172%     -70.1%     293.25 ±217%     -92.5%      73.44 ± 18%  interrupts.CPU13.NMI:Non-maskable_interrupts
    980.44 ±172%     -70.1%     293.25 ±217%     -92.5%      73.44 ± 18%  interrupts.CPU13.PMI:Performance_monitoring_interrupts
     19.67 ± 68%    +147.0%      48.58 ±135%    +132.2%      45.67 ±125%  interrupts.CPU13.RES:Rescheduling_interrupts
      1.67 ±109%     +35.0%       2.25 ±140%    +166.7%       4.44 ± 30%  interrupts.CPU13.TLB:TLB_shootdowns
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU14.295:PCI-MSI.65536-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU14.300:PCI-MSI.73728-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU14.301:PCI-MSI.73728-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU14.301:PCI-MSI.75776-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU14.302:PCI-MSI.75776-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU14.302:PCI-MSI.77824-edge.ioat-msix
     21.33 ±282%    -100.0%       0.00          -100.0%       0.00        interrupts.CPU14.303:PCI-MSI.288768-edge.ahci[0000:00:11.5]
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU14.303:PCI-MSI.77824-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU14.303:PCI-MSI.79872-edge.ioat-msix
      0.00       +8.3e+100%       0.08 ±331% +3.3e+101%       0.33 ±141%  interrupts.CPU14.37:PCI-MSI.31981570-edge.i40e-eth0-TxRx-1
     10.78 ±187%     -21.9%       8.42 ±224%    -100.0%       0.00        interrupts.CPU14.4:IO-APIC.4-edge.ttyS0
    158.22 ±195%   +2061.5%       3419 ±329%     -84.1%      25.11 ±281%  interrupts.CPU14.50:PCI-MSI.31981583-edge.i40e-eth0-TxRx-14
      0.00       +1.7e+101%       0.17 ±223% +2.2e+101%       0.22 ±187%  interrupts.CPU14.85:PCI-MSI.31981618-edge.i40e-eth0-TxRx-49
    709.22 ± 61%     -25.7%     526.67 ± 10%     -21.3%     558.44 ±  6%  interrupts.CPU14.CAL:Function_call_interrupts
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU14.IWI:IRQ_work_interrupts
    774500 ±  3%      -6.7%     722695 ± 18%      +1.2%     783694        interrupts.CPU14.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU14.MCP:Machine_check_polls
    145.67 ± 94%      -0.5%     144.92 ±102%     -55.5%      64.78 ± 40%  interrupts.CPU14.NMI:Non-maskable_interrupts
    145.67 ± 94%      -0.5%     144.92 ±102%     -55.5%      64.78 ± 40%  interrupts.CPU14.PMI:Performance_monitoring_interrupts
     76.78 ±147%     -47.3%      40.50 ± 74%     -74.7%      19.44 ± 49%  interrupts.CPU14.RES:Rescheduling_interrupts
      1.44 ±113%     +32.7%       1.92 ±146%    +176.9%       4.00 ± 35%  interrupts.CPU14.TLB:TLB_shootdowns
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU15.301:PCI-MSI.75776-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU15.302:PCI-MSI.75776-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU15.302:PCI-MSI.77824-edge.ioat-msix
      0.00          -100.0%       0.00       +2.1e+103%      21.22 ±282%  interrupts.CPU15.303:PCI-MSI.288768-edge.ahci[0000:00:11.5]
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU15.303:PCI-MSI.77824-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU15.303:PCI-MSI.79872-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU15.304:PCI-MSI.79872-edge.ioat-msix
     21.22 ±282%     -25.0%      15.92 ±331%    -100.0%       0.00        interrupts.CPU15.309:PCI-MSI.288768-edge.ahci[0000:00:11.5]
      0.00          -100.0%       0.00       +4.4e+101%       0.44 ±111%  interrupts.CPU15.38:PCI-MSI.31981571-edge.i40e-eth0-TxRx-2
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU15.3:IO-APIC.3-edge
      0.00       +7.7e+102%       7.67 ±223%    -100.0%       0.00        interrupts.CPU15.4:IO-APIC.4-edge.ttyS0
     34.00 ±282%     +60.0%      54.42 ±223%    +226.5%     111.00 ±213%  interrupts.CPU15.51:PCI-MSI.31981584-edge.i40e-eth0-TxRx-15
      0.22 ±187%     -25.0%       0.17 ±223%     -50.0%       0.11 ±282%  interrupts.CPU15.86:PCI-MSI.31981619-edge.i40e-eth0-TxRx-50
      1272 ±158%     -19.3%       1026 ± 73%     -52.8%     600.56 ± 25%  interrupts.CPU15.CAL:Function_call_interrupts
      0.00       +8.3e+100%       0.08 ±331%    -100.0%       0.00        interrupts.CPU15.IWI:IRQ_work_interrupts
    774611 ±  3%      -6.7%     722447 ± 18%      +1.2%     783698        interrupts.CPU15.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU15.MCP:Machine_check_polls
     93.89 ± 29%    +499.7%     563.08 ±271%     -35.5%      60.56 ± 36%  interrupts.CPU15.NMI:Non-maskable_interrupts
     93.89 ± 29%    +499.7%     563.08 ±271%     -35.5%      60.56 ± 36%  interrupts.CPU15.PMI:Performance_monitoring_interrupts
    366.78 ±273%     -93.9%      22.25 ± 66%     -94.9%      18.78 ± 69%  interrupts.CPU15.RES:Rescheduling_interrupts
      2.00 ± 97%      +0.0%       2.00 ±148%    +116.7%       4.33 ± 40%  interrupts.CPU15.TLB:TLB_shootdowns
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU16.295:PCI-MSI.65536-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU16.297:PCI-MSI.67584-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU16.302:PCI-MSI.77824-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU16.303:PCI-MSI.77824-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU16.303:PCI-MSI.79872-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU16.304:PCI-MSI.79872-edge.ioat-msix
      0.00       +1.6e+103%      15.92 ±331% +4.2e+103%      42.44 ±187%  interrupts.CPU16.306:PCI-MSI.288768-edge.ahci[0000:00:11.5]
     21.22 ±282%    -100.0%       0.00          -100.0%       0.00        interrupts.CPU16.308:PCI-MSI.288768-edge.ahci[0000:00:11.5]
      0.00          -100.0%       0.00       +2.1e+103%      21.22 ±282%  interrupts.CPU16.310:PCI-MSI.288768-edge.ahci[0000:00:11.5]
      0.00       +1.6e+103%      15.92 ±331% +2.1e+103%      21.33 ±282%  interrupts.CPU16.314:PCI-MSI.288768-edge.ahci[0000:00:11.5]
      9.33 ±282%    -100.0%       0.00          -100.0%       0.00        interrupts.CPU16.315:PCI-MSI.376832-edge.ahci[0000:00:17.0]
      0.33 ±141%    -100.0%       0.00           -66.7%       0.11 ±282%  interrupts.CPU16.39:PCI-MSI.31981572-edge.i40e-eth0-TxRx-3
      0.00          -100.0%       0.00       +5.7e+102%       5.67 ±282%  interrupts.CPU16.4:IO-APIC.4-edge.ttyS0
      5.00 ±282%      -5.0%       4.75 ±224%   +3395.6%     174.78 ±251%  interrupts.CPU16.52:PCI-MSI.31981585-edge.i40e-eth0-TxRx-16
      0.22 ±187%     -62.5%       0.08 ±331%     -50.0%       0.11 ±282%  interrupts.CPU16.87:PCI-MSI.31981620-edge.i40e-eth0-TxRx-51
      1913 ±199%     -50.2%     952.92 ±117%     -43.9%       1072 ± 64%  interrupts.CPU16.CAL:Function_call_interrupts
    774609 ±  3%      -6.7%     722394 ± 18%      +1.2%     783588        interrupts.CPU16.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU16.MCP:Machine_check_polls
    104.00 ± 26%      -1.8%     102.08 ± 26%     -36.5%      66.00 ± 30%  interrupts.CPU16.NMI:Non-maskable_interrupts
    104.00 ± 26%      -1.8%     102.08 ± 26%     -36.5%      66.00 ± 30%  interrupts.CPU16.PMI:Performance_monitoring_interrupts
    111.00 ±239%     -47.2%      58.58 ±123%     -88.8%      12.44 ± 76%  interrupts.CPU16.RES:Rescheduling_interrupts
      1.67 ±109%     +45.0%       2.42 ±124%    +153.3%       4.22 ± 24%  interrupts.CPU16.TLB:TLB_shootdowns
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU17.295:PCI-MSI.65536-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU17.303:PCI-MSI.79872-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU17.304:PCI-MSI.79872-edge.ioat-msix
     21.22 ±282%     -25.0%      15.92 ±331%      +0.0%      21.22 ±282%  interrupts.CPU17.306:PCI-MSI.288768-edge.ahci[0000:00:11.5]
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU17.307:PCI-MSI.376832-edge.ahci[0000:00:17.0]
      0.00       +3.2e+103%      31.83 ±223%    -100.0%       0.00        interrupts.CPU17.308:PCI-MSI.288768-edge.ahci[0000:00:11.5]
      0.00       +1.6e+103%      15.92 ±331%    -100.0%       0.00        interrupts.CPU17.310:PCI-MSI.288768-edge.ahci[0000:00:11.5]
      0.00          -100.0%       0.00       +2.1e+103%      21.22 ±282%  interrupts.CPU17.311:PCI-MSI.288768-edge.ahci[0000:00:11.5]
     21.22 ±282%    -100.0%       0.00          -100.0%       0.00        interrupts.CPU17.312:PCI-MSI.288768-edge.ahci[0000:00:11.5]
     21.22 ±282%    -100.0%       0.00          -100.0%       0.00        interrupts.CPU17.314:PCI-MSI.288768-edge.ahci[0000:00:11.5]
      9.00 ±282%    -100.0%       0.00            -9.9%       8.11 ±282%  interrupts.CPU17.315:PCI-MSI.376832-edge.ahci[0000:00:17.0]
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU17.3:IO-APIC.3-edge
      0.33 ±141%     -50.0%       0.17 ±223%     -66.7%       0.11 ±282%  interrupts.CPU17.40:PCI-MSI.31981573-edge.i40e-eth0-TxRx-4
     17.33 ±187%    +499.0%     103.83 ±326%     +40.4%      24.33 ±200%  interrupts.CPU17.53:PCI-MSI.31981586-edge.i40e-eth0-TxRx-17
      0.22 ±187%    -100.0%       0.00           -50.0%       0.11 ±282%  interrupts.CPU17.88:PCI-MSI.31981621-edge.i40e-eth0-TxRx-52
    644.00 ± 47%    +314.1%       2666 ±263%   +1194.3%       8335 ±235%  interrupts.CPU17.CAL:Function_call_interrupts
      0.00       +2.5e+101%       0.25 ±238%    -100.0%       0.00        interrupts.CPU17.IWI:IRQ_work_interrupts
    774589 ±  3%      -6.7%     722419 ± 18%      +1.2%     783648        interrupts.CPU17.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU17.MCP:Machine_check_polls
    105.89 ± 19%    +377.7%     505.83 ±261%     -34.0%      69.89 ± 28%  interrupts.CPU17.NMI:Non-maskable_interrupts
    105.89 ± 19%    +377.7%     505.83 ±261%     -34.0%      69.89 ± 28%  interrupts.CPU17.PMI:Performance_monitoring_interrupts
     58.33 ±204%     -65.3%      20.25 ± 70%     -81.1%      11.00 ± 30%  interrupts.CPU17.RES:Rescheduling_interrupts
      1.56 ±113%     +50.0%       2.33 ±132%    +228.6%       5.11 ± 17%  interrupts.CPU17.TLB:TLB_shootdowns
     21.22 ±282%     -25.0%      15.92 ±331%      +0.0%      21.22 ±282%  interrupts.CPU18.306:PCI-MSI.288768-edge.ahci[0000:00:11.5]
      8.56 ±282%    -100.0%       0.00          -100.0%       0.00        interrupts.CPU18.315:PCI-MSI.376832-edge.ahci[0000:00:17.0]
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU18.3:IO-APIC.3-edge
      0.11 ±282%     +50.0%       0.17 ±223%    -100.0%       0.00        interrupts.CPU18.41:PCI-MSI.31981574-edge.i40e-eth0-TxRx-5
     21.11 ±282%     +64.2%      34.67 ±314%    +265.8%      77.22 ±274%  interrupts.CPU18.54:PCI-MSI.31981587-edge.i40e-eth0-TxRx-18
      0.00          -100.0%       0.00       +1.1e+101%       0.11 ±282%  interrupts.CPU18.89:PCI-MSI.31981622-edge.i40e-eth0-TxRx-53
      3956 ±154%     -33.9%       2613 ±224%    +171.8%      10754 ±259%  interrupts.CPU18.CAL:Function_call_interrupts
      0.00       +8.3e+100%       0.08 ±331%    -100.0%       0.00        interrupts.CPU18.IWI:IRQ_work_interrupts
    774468 ±  3%      -6.7%     722385 ± 18%      +1.2%     783567        interrupts.CPU18.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU18.MCP:Machine_check_polls
    102.78 ± 16%    +511.8%     628.75 ±265%    +245.8%     355.44 ±224%  interrupts.CPU18.NMI:Non-maskable_interrupts
    102.78 ± 16%    +511.8%     628.75 ±265%    +245.8%     355.44 ±224%  interrupts.CPU18.PMI:Performance_monitoring_interrupts
     20.22 ± 47%    +137.0%      47.92 ±121%    +161.0%      52.78 ±180%  interrupts.CPU18.RES:Rescheduling_interrupts
      2.00 ± 94%     +33.3%       2.67 ±122%    +133.3%       4.67 ± 17%  interrupts.CPU18.TLB:TLB_shootdowns
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU19.297:PCI-MSI.67584-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU19.302:PCI-MSI.77824-edge.ioat-msix
      8.67 ±282%    -100.0%       0.00          -100.0%       0.00        interrupts.CPU19.315:PCI-MSI.376832-edge.ahci[0000:00:17.0]
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU19.3:IO-APIC.3-edge
      0.00       +8.3e+100%       0.08 ±331% +5.6e+101%       0.56 ± 89%  interrupts.CPU19.42:PCI-MSI.31981575-edge.i40e-eth0-TxRx-6
      5.11 ±282%    -100.0%       0.00          -100.0%       0.00        interrupts.CPU19.4:IO-APIC.4-edge.ttyS0
     69.44 ±281%  +10043.2%       7043 ±331%     -88.8%       7.78 ±187%  interrupts.CPU19.55:PCI-MSI.31981588-edge.i40e-eth0-TxRx-19
      0.00       +1.7e+101%       0.17 ±223%    -100.0%       0.00        interrupts.CPU19.90:PCI-MSI.31981623-edge.i40e-eth0-TxRx-54
    556.56 ±  6%   +1092.4%       6636 ±224%    +200.0%       1669 ±127%  interrupts.CPU19.CAL:Function_call_interrupts
    774599 ±  3%      -6.7%     722577 ± 18%      +1.2%     783636        interrupts.CPU19.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU19.MCP:Machine_check_polls
    108.22 ± 19%      -8.3%      99.25 ± 26%     -33.6%      71.89 ± 26%  interrupts.CPU19.NMI:Non-maskable_interrupts
    108.22 ± 19%      -8.3%      99.25 ± 26%     -33.6%      71.89 ± 26%  interrupts.CPU19.PMI:Performance_monitoring_interrupts
    240.00 ±262%     -93.6%      15.33 ± 68%     -92.4%      18.33 ±108%  interrupts.CPU19.RES:Rescheduling_interrupts
      1.78 ±101%     +59.4%       2.83 ±147%    +162.5%       4.67 ± 34%  interrupts.CPU19.TLB:TLB_shootdowns
      0.00       +1.7e+101%       0.17 ±223% +1.1e+101%       0.11 ±282%  interrupts.CPU2.121:PCI-MSI.31981654-edge.i40e-eth0-TxRx-85
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU2.297:PCI-MSI.67584-edge.ioat-msix
      3.89 ±240%   +2315.0%      93.92 ±311%   +4345.7%     172.89 ±261%  interrupts.CPU2.38:PCI-MSI.31981571-edge.i40e-eth0-TxRx-2
      0.11 ±282%    +200.0%       0.33 ±141%    +100.0%       0.22 ±187%  interrupts.CPU2.73:PCI-MSI.31981606-edge.i40e-eth0-TxRx-37
    649.33 ± 25%     +31.8%     855.58 ± 66%     -13.0%     565.00 ±  5%  interrupts.CPU2.CAL:Function_call_interrupts
    774442 ±  3%      -6.7%     722580 ± 18%      +1.2%     783803        interrupts.CPU2.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU2.MCP:Machine_check_polls
    654.78 ±159%     -85.1%      97.75 ± 24%     -50.4%     325.00 ±218%  interrupts.CPU2.NMI:Non-maskable_interrupts
    654.78 ±159%     -85.1%      97.75 ± 24%     -50.4%     325.00 ±218%  interrupts.CPU2.PMI:Performance_monitoring_interrupts
     15.67 ± 64%    +233.5%      52.25 ±229%     -26.2%      11.56 ± 83%  interrupts.CPU2.RES:Rescheduling_interrupts
      3.00 ±115%     -33.3%       2.00 ±132%     +55.6%       4.67 ± 48%  interrupts.CPU2.TLB:TLB_shootdowns
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU20.295:PCI-MSI.65536-edge.ioat-msix
      0.11 ±282%    -100.0%       0.00          +200.0%       0.33 ±141%  interrupts.CPU20.43:PCI-MSI.31981576-edge.i40e-eth0-TxRx-7
      5.11 ±282%    -100.0%       0.00          -100.0%       0.00        interrupts.CPU20.4:IO-APIC.4-edge.ttyS0
      1.44 ±282%  +11876.9%     173.00 ±226%   +2607.7%      39.11 ±281%  interrupts.CPU20.56:PCI-MSI.31981589-edge.i40e-eth0-TxRx-20
      0.33 ±141%     -25.0%       0.25 ±173%     -33.3%       0.22 ±187%  interrupts.CPU20.91:PCI-MSI.31981624-edge.i40e-eth0-TxRx-55
      6289 ±233%     -89.3%     670.08 ± 38%     -90.4%     605.22 ± 28%  interrupts.CPU20.CAL:Function_call_interrupts
      0.11 ±282%    -100.0%       0.00          +100.0%       0.22 ±187%  interrupts.CPU20.IWI:IRQ_work_interrupts
    774446 ±  3%      -6.7%     722456 ± 18%      +1.2%     783709        interrupts.CPU20.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU20.MCP:Machine_check_polls
    841.89 ±182%     -88.3%      98.17 ± 30%     -18.4%     687.11 ±249%  interrupts.CPU20.NMI:Non-maskable_interrupts
    841.89 ±182%     -88.3%      98.17 ± 30%     -18.4%     687.11 ±249%  interrupts.CPU20.PMI:Performance_monitoring_interrupts
     19.11 ± 80%    +142.0%      46.25 ±196%     +66.3%      31.78 ± 96%  interrupts.CPU20.RES:Rescheduling_interrupts
      1.67 ±116%     +35.0%       2.25 ±145%    +180.0%       4.67 ± 36%  interrupts.CPU20.TLB:TLB_shootdowns
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU21.295:PCI-MSI.65536-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU21.298:PCI-MSI.69632-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU21.3:IO-APIC.3-edge
      0.11 ±282%     +50.0%       0.17 ±223%    +100.0%       0.22 ±187%  interrupts.CPU21.44:PCI-MSI.31981577-edge.i40e-eth0-TxRx-8
      1.67 ±187%    +270.0%       6.17 ±294%    +120.0%       3.67 ±282%  interrupts.CPU21.57:PCI-MSI.31981590-edge.i40e-eth0-TxRx-21
      0.00       +8.3e+100%       0.08 ±331% +2.2e+101%       0.22 ±187%  interrupts.CPU21.92:PCI-MSI.31981625-edge.i40e-eth0-TxRx-56
    931.00 ±115%     -38.1%     576.58 ± 18%     -34.5%     609.78 ± 29%  interrupts.CPU21.CAL:Function_call_interrupts
      0.00       +8.3e+100%       0.08 ±331%    -100.0%       0.00        interrupts.CPU21.IWI:IRQ_work_interrupts
    774438 ±  3%      -6.7%     722473 ± 18%      +1.2%     783660        interrupts.CPU21.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU21.MCP:Machine_check_polls
    148.44 ± 84%     -30.2%     103.67 ± 30%     -54.8%      67.11 ± 33%  interrupts.CPU21.NMI:Non-maskable_interrupts
    148.44 ± 84%     -30.2%     103.67 ± 30%     -54.8%      67.11 ± 33%  interrupts.CPU21.PMI:Performance_monitoring_interrupts
     50.67 ±211%     -43.3%      28.75 ± 72%     -38.2%      31.33 ± 65%  interrupts.CPU21.RES:Rescheduling_interrupts
      1.56 ±101%     +39.3%       2.17 ±151%    +221.4%       5.00 ± 18%  interrupts.CPU21.TLB:TLB_shootdowns
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU22.295:PCI-MSI.65536-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU22.297:PCI-MSI.67584-edge.ioat-msix
      0.44 ±111%     -62.5%       0.17 ±223%     -75.0%       0.11 ±282%  interrupts.CPU22.45:PCI-MSI.31981578-edge.i40e-eth0-TxRx-9
      0.00       +8.4e+103%      83.50 ±297% +2.7e+102%       2.67 ±269%  interrupts.CPU22.58:PCI-MSI.31981591-edge.i40e-eth0-TxRx-22
      0.11 ±282%     -25.0%       0.08 ±331%    -100.0%       0.00        interrupts.CPU22.93:PCI-MSI.31981626-edge.i40e-eth0-TxRx-57
    515.56 ± 11%     +15.9%     597.42 ± 31%     +36.1%     701.78 ± 42%  interrupts.CPU22.CAL:Function_call_interrupts
    774528 ±  3%      -6.7%     722580 ± 18%      +1.2%     783639        interrupts.CPU22.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU22.MCP:Machine_check_polls
    104.00 ± 16%      -6.4%      97.33 ± 25%     -35.7%      66.89 ± 33%  interrupts.CPU22.NMI:Non-maskable_interrupts
    104.00 ± 16%      -6.4%      97.33 ± 25%     -35.7%      66.89 ± 33%  interrupts.CPU22.PMI:Performance_monitoring_interrupts
      9.67 ± 35%    +461.2%      54.25 ±220%    +129.9%      22.22 ±115%  interrupts.CPU22.RES:Rescheduling_interrupts
      1.44 ±103%     +90.4%       2.75 ±123%    +238.5%       4.89 ± 15%  interrupts.CPU22.TLB:TLB_shootdowns
      0.00       +1.6e+103%      15.92 ±331%    -100.0%       0.00        interrupts.CPU23.296:PCI-MSI.288768-edge.ahci[0000:00:11.5]
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU23.297:PCI-MSI.67584-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU23.3:IO-APIC.3-edge
      0.22 ±187%    -100.0%       0.00            +0.0%       0.22 ±187%  interrupts.CPU23.46:PCI-MSI.31981579-edge.i40e-eth0-TxRx-10
      0.00          -100.0%       0.00       +5.7e+102%       5.67 ±282%  interrupts.CPU23.4:IO-APIC.4-edge.ttyS0
     66.22 ±232%      -6.3%      62.08 ±285%    +773.0%     578.11 ±239%  interrupts.CPU23.59:PCI-MSI.31981592-edge.i40e-eth0-TxRx-23
      0.00       +1.7e+101%       0.17 ±223%    -100.0%       0.00        interrupts.CPU23.94:PCI-MSI.31981627-edge.i40e-eth0-TxRx-58
      1293 ±164%    +267.3%       4749 ±290%     -58.1%     542.11 ±  5%  interrupts.CPU23.CAL:Function_call_interrupts
      0.22 ±187%     -25.0%       0.17 ±223%    -100.0%       0.00        interrupts.CPU23.IWI:IRQ_work_interrupts
    774462 ±  3%      -6.4%     725129 ± 18%      +1.2%     783628        interrupts.CPU23.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU23.MCP:Machine_check_polls
      1249 ±171%     -44.3%     696.25 ±213%     -94.2%      72.22 ± 29%  interrupts.CPU23.NMI:Non-maskable_interrupts
      1249 ±171%     -44.3%     696.25 ±213%     -94.2%      72.22 ± 29%  interrupts.CPU23.PMI:Performance_monitoring_interrupts
     16.78 ± 90%     +54.0%      25.83 ± 56%     -26.5%      12.33 ± 64%  interrupts.CPU23.RES:Rescheduling_interrupts
      2.56 ± 94%     +14.1%       2.92 ±118%     +78.3%       4.56 ± 25%  interrupts.CPU23.TLB:TLB_shootdowns
     54.44 ±214%    +316.3%     226.67 ±238%    +238.0%     184.00 ±242%  interrupts.CPU24.60:PCI-MSI.31981593-edge.i40e-eth0-TxRx-24
    783.67 ± 88%      -0.6%     778.75 ± 95%     -17.1%     649.44 ± 43%  interrupts.CPU24.CAL:Function_call_interrupts
      0.44 ±154%     -25.0%       0.33 ±141%    -100.0%       0.00        interrupts.CPU24.IWI:IRQ_work_interrupts
    742136 ± 15%      +3.6%     769181 ±  3%      +5.6%     783608        interrupts.CPU24.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU24.MCP:Machine_check_polls
    142.44 ± 42%      -7.9%     131.25 ± 50%     -58.1%      59.67 ± 36%  interrupts.CPU24.NMI:Non-maskable_interrupts
    142.44 ± 42%      -7.9%     131.25 ± 50%     -58.1%      59.67 ± 36%  interrupts.CPU24.PMI:Performance_monitoring_interrupts
     32.00 ±140%     -66.7%      10.67 ± 76%     -61.5%      12.33 ±131%  interrupts.CPU24.RES:Rescheduling_interrupts
      1.44 ±108%     +44.2%       2.08 ±149%    +261.5%       5.22 ± 19%  interrupts.CPU24.TLB:TLB_shootdowns
      1582 ±282%     -99.2%      13.17 ±329%     -98.8%      19.56 ±187%  interrupts.CPU25.61:PCI-MSI.31981594-edge.i40e-eth0-TxRx-25
      1717 ±194%     -35.5%       1107 ± 96%    +324.6%       7291 ±205%  interrupts.CPU25.CAL:Function_call_interrupts
      0.00       +1.7e+101%       0.17 ±223% +1.1e+101%       0.11 ±282%  interrupts.CPU25.IWI:IRQ_work_interrupts
    743048 ± 15%      +3.5%     769173 ±  4%      +5.5%     783645        interrupts.CPU25.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU25.MCP:Machine_check_polls
    121.67 ± 35%      -3.2%     117.83 ± 51%     -46.6%      65.00 ± 34%  interrupts.CPU25.NMI:Non-maskable_interrupts
    121.67 ± 35%      -3.2%     117.83 ± 51%     -46.6%      65.00 ± 34%  interrupts.CPU25.PMI:Performance_monitoring_interrupts
     14.78 ±124%     +31.4%      19.42 ± 80%     -21.8%      11.56 ± 66%  interrupts.CPU25.RES:Rescheduling_interrupts
      1.89 ±101%     +14.7%       2.17 ±138%    +170.6%       5.11 ± 25%  interrupts.CPU25.TLB:TLB_shootdowns
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU26.305:PCI-MSI.67174400-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU26.306:PCI-MSI.67174400-edge.ioat-msix
      0.00       +2.8e+102%       2.75 ±243% +1.1e+101%       0.11 ±282%  interrupts.CPU26.62:PCI-MSI.31981595-edge.i40e-eth0-TxRx-26
    535.89          +169.4%       1443 ±119%    +186.5%       1535 ±179%  interrupts.CPU26.CAL:Function_call_interrupts
    742092 ± 15%      +3.6%     769143 ±  3%      +5.6%     783610        interrupts.CPU26.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU26.MCP:Machine_check_polls
    121.00 ± 33%      +1.8%     123.17 ± 40%     -47.9%      63.00 ± 28%  interrupts.CPU26.NMI:Non-maskable_interrupts
    121.00 ± 33%      +1.8%     123.17 ± 40%     -47.9%      63.00 ± 28%  interrupts.CPU26.PMI:Performance_monitoring_interrupts
      7.56 ± 44%    +161.4%      19.75 ± 74%    +480.9%      43.89 ±237%  interrupts.CPU26.RES:Rescheduling_interrupts
      2.00 ±102%      -8.3%       1.83 ±163%    +150.0%       5.00 ± 29%  interrupts.CPU26.TLB:TLB_shootdowns
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU27.307:PCI-MSI.67176448-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU27.308:PCI-MSI.67176448-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU27.309:PCI-MSI.67176448-edge.ioat-msix
     63.00 ±282%     -83.5%      10.42 ±331%     -88.4%       7.33 ±273%  interrupts.CPU27.63:PCI-MSI.31981596-edge.i40e-eth0-TxRx-27
    966.00 ±126%     -15.3%     818.17 ± 66%     +24.5%       1202 ±161%  interrupts.CPU27.CAL:Function_call_interrupts
      0.11 ±282%     -25.0%       0.08 ±331%    -100.0%       0.00        interrupts.CPU27.IWI:IRQ_work_interrupts
    742143 ± 15%      +3.6%     769182 ±  4%      +5.6%     783623        interrupts.CPU27.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU27.MCP:Machine_check_polls
    125.33 ± 37%    +397.7%     623.75 ±267%     -47.5%      65.78 ± 24%  interrupts.CPU27.NMI:Non-maskable_interrupts
    125.33 ± 37%    +397.7%     623.75 ±267%     -47.5%      65.78 ± 24%  interrupts.CPU27.PMI:Performance_monitoring_interrupts
     14.78 ± 90%    +131.8%      34.25 ± 66%    +179.7%      41.33 ±196%  interrupts.CPU27.RES:Rescheduling_interrupts
      2.22 ± 98%      -2.5%       2.17 ±146%    +115.0%       4.78 ± 32%  interrupts.CPU27.TLB:TLB_shootdowns
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU28.307:PCI-MSI.67176448-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU28.308:PCI-MSI.67176448-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU28.308:PCI-MSI.67178496-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU28.309:PCI-MSI.67178496-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU28.310:PCI-MSI.67178496-edge.ioat-msix
    210.89 ±257%     -93.3%      14.17 ±304%     -21.6%     165.33 ±282%  interrupts.CPU28.64:PCI-MSI.31981597-edge.i40e-eth0-TxRx-28
      4658 ±167%     -60.0%       1862 ±233%     -59.7%       1877 ±185%  interrupts.CPU28.CAL:Function_call_interrupts
    742109 ± 15%      +3.6%     769079 ±  4%      +5.6%     783584        interrupts.CPU28.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU28.MCP:Machine_check_polls
    122.00 ± 34%      +1.6%     123.92 ± 36%     -45.7%      66.22 ± 24%  interrupts.CPU28.NMI:Non-maskable_interrupts
    122.00 ± 34%      +1.6%     123.92 ± 36%     -45.7%      66.22 ± 24%  interrupts.CPU28.PMI:Performance_monitoring_interrupts
     16.00 ± 80%     +25.5%      20.08 ± 64%     -25.7%      11.89 ±144%  interrupts.CPU28.RES:Rescheduling_interrupts
      2.56 ± 86%      -5.4%       2.42 ±139%    +100.0%       5.11 ± 29%  interrupts.CPU28.TLB:TLB_shootdowns
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU29.309:PCI-MSI.67178496-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU29.309:PCI-MSI.67180544-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU29.310:PCI-MSI.67180544-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU29.311:PCI-MSI.67180544-edge.ioat-msix
      8.78 ±274%     -63.9%       3.17 ±322%    +829.1%      81.56 ±261%  interrupts.CPU29.65:PCI-MSI.31981598-edge.i40e-eth0-TxRx-29
      1037 ±136%     -47.9%     540.42 ±  5%    +272.5%       3864 ±165%  interrupts.CPU29.CAL:Function_call_interrupts
      0.11 ±282%    +275.0%       0.42 ±206%    -100.0%       0.00        interrupts.CPU29.IWI:IRQ_work_interrupts
    742047 ± 15%      +3.6%     769129 ±  3%      +5.6%     783578        interrupts.CPU29.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU29.MCP:Machine_check_polls
    125.89 ± 35%     +13.1%     142.33 ± 47%     -50.1%      62.78 ± 24%  interrupts.CPU29.NMI:Non-maskable_interrupts
    125.89 ± 35%     +13.1%     142.33 ± 47%     -50.1%      62.78 ± 24%  interrupts.CPU29.PMI:Performance_monitoring_interrupts
      6.11 ± 71%    +100.5%      12.25 ± 86%     -25.5%       4.56 ± 52%  interrupts.CPU29.RES:Rescheduling_interrupts
      2.22 ±105%      +8.7%       2.42 ±139%    +140.0%       5.33 ± 21%  interrupts.CPU29.TLB:TLB_shootdowns
      0.00       +8.3e+100%       0.08 ±331% +1.1e+101%       0.11 ±282%  interrupts.CPU3.122:PCI-MSI.31981655-edge.i40e-eth0-TxRx-86
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU3.297:PCI-MSI.67584-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU3.298:PCI-MSI.69632-edge.ioat-msix
     38.11 ±232%     -90.8%       3.50 ±331%    +184.0%     108.22 ±270%  interrupts.CPU3.39:PCI-MSI.31981572-edge.i40e-eth0-TxRx-3
      0.22 ±187%     -25.0%       0.17 ±223%     -50.0%       0.11 ±282%  interrupts.CPU3.74:PCI-MSI.31981607-edge.i40e-eth0-TxRx-38
      4407 ±225%     -81.7%     806.25 ± 74%     +66.4%       7331 ±246%  interrupts.CPU3.CAL:Function_call_interrupts
    774494 ±  3%      -6.7%     722513 ± 18%      +1.2%     783510        interrupts.CPU3.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU3.MCP:Machine_check_polls
    103.56 ± 17%      -8.4%      94.83 ± 27%     -27.7%      74.89 ± 17%  interrupts.CPU3.NMI:Non-maskable_interrupts
    103.56 ± 17%      -8.4%      94.83 ± 27%     -27.7%      74.89 ± 17%  interrupts.CPU3.PMI:Performance_monitoring_interrupts
     71.89 ±245%     -75.7%      17.50 ± 98%     -75.1%      17.89 ±118%  interrupts.CPU3.RES:Rescheduling_interrupts
      1.44 ± 92%     +84.6%       2.67 ±150%    +153.8%       3.67 ± 34%  interrupts.CPU3.TLB:TLB_shootdowns
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU30.310:PCI-MSI.67180544-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU30.310:PCI-MSI.67182592-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU30.311:PCI-MSI.67182592-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU30.312:PCI-MSI.67182592-edge.ioat-msix
      0.00       +1.8e+103%      18.17 ±176% +1.8e+103%      17.56 ±280%  interrupts.CPU30.66:PCI-MSI.31981599-edge.i40e-eth0-TxRx-30
    637.44 ± 45%    +453.4%       3527 ±273%     -16.0%     535.22 ±  4%  interrupts.CPU30.CAL:Function_call_interrupts
      0.22 ±187%     -25.0%       0.17 ±223%    -100.0%       0.00        interrupts.CPU30.IWI:IRQ_work_interrupts
    742040 ± 15%      +3.7%     769199 ±  3%      +5.6%     783637        interrupts.CPU30.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU30.MCP:Machine_check_polls
    129.89 ± 36%      +1.5%     131.83 ± 41%     -45.6%      70.67 ± 17%  interrupts.CPU30.NMI:Non-maskable_interrupts
    129.89 ± 36%      +1.5%     131.83 ± 41%     -45.6%      70.67 ± 17%  interrupts.CPU30.PMI:Performance_monitoring_interrupts
     11.00 ± 83%     +50.0%      16.50 ±144%     -34.3%       7.22 ± 91%  interrupts.CPU30.RES:Rescheduling_interrupts
      2.22 ±105%      +8.7%       2.42 ±134%    +110.0%       4.67 ± 31%  interrupts.CPU30.TLB:TLB_shootdowns
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU31.310:PCI-MSI.67180544-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU31.311:PCI-MSI.67182592-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU31.311:PCI-MSI.67184640-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU31.312:PCI-MSI.67184640-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU31.313:PCI-MSI.67184640-edge.ioat-msix
     14016 ±282%     -97.8%     306.08 ±202%     -99.9%      11.33 ±273%  interrupts.CPU31.67:PCI-MSI.31981600-edge.i40e-eth0-TxRx-31
    852.56 ±103%    +250.6%       2989 ±236%    +192.3%       2491 ±221%  interrupts.CPU31.CAL:Function_call_interrupts
      0.11 ±282%     +50.0%       0.17 ±223%    -100.0%       0.00        interrupts.CPU31.IWI:IRQ_work_interrupts
    742048 ± 15%      +3.7%     769150 ±  3%      +5.6%     783615        interrupts.CPU31.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU31.MCP:Machine_check_polls
    126.11 ± 32%      -2.5%     122.92 ± 42%     -39.6%      76.11 ± 16%  interrupts.CPU31.NMI:Non-maskable_interrupts
    126.11 ± 32%      -2.5%     122.92 ± 42%     -39.6%      76.11 ± 16%  interrupts.CPU31.PMI:Performance_monitoring_interrupts
     21.22 ±201%     -44.2%      11.83 ±128%     -55.0%       9.56 ±110%  interrupts.CPU31.RES:Rescheduling_interrupts
      2.00 ±120%     +20.8%       2.42 ±139%    +172.2%       5.44 ± 23%  interrupts.CPU31.TLB:TLB_shootdowns
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU32.311:PCI-MSI.67182592-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU32.312:PCI-MSI.67184640-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU32.312:PCI-MSI.67186688-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU32.313:PCI-MSI.67186688-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU32.314:PCI-MSI.67186688-edge.ioat-msix
    102.00 ±233%    +177.3%     282.83 ±127%     -99.9%       0.11 ±282%  interrupts.CPU32.68:PCI-MSI.31981601-edge.i40e-eth0-TxRx-32
      4767 ±236%     -88.7%     540.17            -4.7%       4541 ±251%  interrupts.CPU32.CAL:Function_call_interrupts
    742042 ± 15%      +3.6%     769101 ±  3%      +5.6%     783647        interrupts.CPU32.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU32.MCP:Machine_check_polls
    122.11 ± 31%      +1.8%     124.25 ± 35%     -35.8%      78.44 ± 13%  interrupts.CPU32.NMI:Non-maskable_interrupts
    122.11 ± 31%      +1.8%     124.25 ± 35%     -35.8%      78.44 ± 13%  interrupts.CPU32.PMI:Performance_monitoring_interrupts
     38.33 ±210%     -60.7%      15.08 ±152%     -75.7%       9.33 ± 84%  interrupts.CPU32.RES:Rescheduling_interrupts
      1.78 ±114%     +31.2%       2.33 ±150%    +181.2%       5.00 ± 31%  interrupts.CPU32.TLB:TLB_shootdowns
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU33.312:PCI-MSI.67184640-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU33.313:PCI-MSI.67186688-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU33.313:PCI-MSI.67188736-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU33.314:PCI-MSI.67188736-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU33.315:PCI-MSI.67188736-edge.ioat-msix
    219.56 ±193%     -95.9%       8.92 ±270%    -100.0%       0.00        interrupts.CPU33.69:PCI-MSI.31981602-edge.i40e-eth0-TxRx-33
    531.78 ± 15%    +106.9%       1100 ±169%      +2.1%     543.11        interrupts.CPU33.CAL:Function_call_interrupts
      0.00       +1.7e+101%       0.17 ±223% +1.1e+101%       0.11 ±282%  interrupts.CPU33.IWI:IRQ_work_interrupts
    742063 ± 15%      +3.6%     769096 ±  3%      +5.6%     783632        interrupts.CPU33.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU33.MCP:Machine_check_polls
    120.00 ± 33%    +359.0%     550.83 ±259%     -30.4%      83.56 ± 20%  interrupts.CPU33.NMI:Non-maskable_interrupts
    120.00 ± 33%    +359.0%     550.83 ±259%     -30.4%      83.56 ± 20%  interrupts.CPU33.PMI:Performance_monitoring_interrupts
     12.00 ± 86%     +16.7%      14.00 ±111%     +27.8%      15.33 ± 82%  interrupts.CPU33.RES:Rescheduling_interrupts
      1.89 ±104%     +23.5%       2.33 ±138%    +176.5%       5.22 ± 28%  interrupts.CPU33.TLB:TLB_shootdowns
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU34.313:PCI-MSI.67186688-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU34.314:PCI-MSI.67188736-edge.ioat-msix
     11.67 ±245%    +706.4%      94.08 ±331%     -94.3%       0.67 ±234%  interrupts.CPU34.70:PCI-MSI.31981603-edge.i40e-eth0-TxRx-34
    997.67 ±127%     -44.4%     555.17 ± 17%     -32.2%     676.11 ± 51%  interrupts.CPU34.CAL:Function_call_interrupts
    742089 ± 15%      +3.6%     769096 ±  4%      +5.6%     783695        interrupts.CPU34.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU34.MCP:Machine_check_polls
    123.89 ± 31%      -8.9%     112.92 ± 40%     -39.0%      75.56 ± 12%  interrupts.CPU34.NMI:Non-maskable_interrupts
    123.89 ± 31%      -8.9%     112.92 ± 40%     -39.0%      75.56 ± 12%  interrupts.CPU34.PMI:Performance_monitoring_interrupts
     24.67 ±127%     -56.1%      10.83 ±131%     -61.7%       9.44 ± 56%  interrupts.CPU34.RES:Rescheduling_interrupts
      2.22 ± 94%      +1.2%       2.25 ±145%    +140.0%       5.33 ± 26%  interrupts.CPU34.TLB:TLB_shootdowns
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU35.314:PCI-MSI.67188736-edge.ioat-msix
      2.00 ±265%    +150.0%       5.00 ±228%  +19227.8%     386.56 ±282%  interrupts.CPU35.71:PCI-MSI.31981604-edge.i40e-eth0-TxRx-35
    543.44 ±  5%     +96.5%       1067 ±159%      +1.0%     549.11 ±  8%  interrupts.CPU35.CAL:Function_call_interrupts
      0.22 ±187%    -100.0%       0.00          -100.0%       0.00        interrupts.CPU35.IWI:IRQ_work_interrupts
    742075 ± 15%      +3.6%     769098 ±  4%      +5.6%     783608        interrupts.CPU35.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU35.MCP:Machine_check_polls
    131.33 ± 38%     -11.5%     116.17 ± 41%     -41.3%      77.11 ± 12%  interrupts.CPU35.NMI:Non-maskable_interrupts
    131.33 ± 38%     -11.5%     116.17 ± 41%     -41.3%      77.11 ± 12%  interrupts.CPU35.PMI:Performance_monitoring_interrupts
     13.67 ± 85%     -22.0%      10.67 ±113%     -24.4%      10.33 ± 74%  interrupts.CPU35.RES:Rescheduling_interrupts
      2.00 ±113%     +25.0%       2.50 ±146%    +205.6%       6.11 ± 43%  interrupts.CPU35.TLB:TLB_shootdowns
    207.44 ±195%     -61.6%      79.58 ±331%    +794.4%       1855 ±272%  interrupts.CPU36.72:PCI-MSI.31981605-edge.i40e-eth0-TxRx-36
    542.33 ±  2%     +40.8%     763.42 ± 97%     +60.5%     870.33 ±109%  interrupts.CPU36.CAL:Function_call_interrupts
      0.00       +1.7e+101%       0.17 ±223%    -100.0%       0.00        interrupts.CPU36.IWI:IRQ_work_interrupts
    742038 ± 15%      +3.7%     769131 ±  4%      +5.6%     783606        interrupts.CPU36.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU36.MCP:Machine_check_polls
    120.67 ± 34%      +0.1%     120.83 ± 46%    +139.8%     289.33 ±211%  interrupts.CPU36.NMI:Non-maskable_interrupts
    120.67 ± 34%      +0.1%     120.83 ± 46%    +139.8%     289.33 ±211%  interrupts.CPU36.PMI:Performance_monitoring_interrupts
      8.78 ± 72%     +88.0%      16.50 ±144%    +246.8%      30.44 ±173%  interrupts.CPU36.RES:Rescheduling_interrupts
      1.78 ± 94%     +35.9%       2.42 ±137%    +187.5%       5.11 ± 26%  interrupts.CPU36.TLB:TLB_shootdowns
      0.11 ±282%  +3.4e+05%     378.42 ±315%  +23900.0%      26.67 ±196%  interrupts.CPU37.73:PCI-MSI.31981606-edge.i40e-eth0-TxRx-37
    542.22 ±  2%     +12.5%     609.75 ± 34%      +3.1%     559.11 ±  7%  interrupts.CPU37.CAL:Function_call_interrupts
      0.11 ±282%     -25.0%       0.08 ±331%      +0.0%       0.11 ±282%  interrupts.CPU37.IWI:IRQ_work_interrupts
    742093 ± 15%      +3.6%     769102 ±  4%      +5.6%     783496        interrupts.CPU37.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU37.MCP:Machine_check_polls
    127.00 ± 38%      -9.1%     115.50 ± 52%     -34.1%      83.67 ± 29%  interrupts.CPU37.NMI:Non-maskable_interrupts
    127.00 ± 38%      -9.1%     115.50 ± 52%     -34.1%      83.67 ± 29%  interrupts.CPU37.PMI:Performance_monitoring_interrupts
      8.00 ± 33%    +141.7%      19.33 ±112%     +77.8%      14.22 ± 81%  interrupts.CPU37.RES:Rescheduling_interrupts
      1.78 ±101%     +50.0%       2.67 ±138%    +212.5%       5.56 ± 22%  interrupts.CPU37.TLB:TLB_shootdowns
    182.11 ±265%    +277.1%     686.75 ±327%     -48.2%      94.33 ±282%  interrupts.CPU38.74:PCI-MSI.31981607-edge.i40e-eth0-TxRx-38
    535.78            +0.4%     537.75            +1.7%     545.11        interrupts.CPU38.CAL:Function_call_interrupts
      0.11 ±282%    -100.0%       0.00          -100.0%       0.00        interrupts.CPU38.IWI:IRQ_work_interrupts
    742247 ± 15%      +3.6%     769074 ±  3%      +5.6%     783741        interrupts.CPU38.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU38.MCP:Machine_check_polls
    125.89 ± 34%     -11.4%     111.58 ± 46%     -39.4%      76.33 ± 13%  interrupts.CPU38.NMI:Non-maskable_interrupts
    125.89 ± 34%     -11.4%     111.58 ± 46%     -39.4%      76.33 ± 13%  interrupts.CPU38.PMI:Performance_monitoring_interrupts
     19.44 ±142%     -45.6%      10.58 ±100%      -1.7%      19.11 ±103%  interrupts.CPU38.RES:Rescheduling_interrupts
      2.11 ± 95%     +22.4%       2.58 ±130%    +168.4%       5.67 ± 22%  interrupts.CPU38.TLB:TLB_shootdowns
    344.44 ±273%     -89.5%      36.25 ±319%     -51.6%     166.56 ±282%  interrupts.CPU39.75:PCI-MSI.31981608-edge.i40e-eth0-TxRx-39
    536.67            +0.6%     539.83            +1.4%     544.00        interrupts.CPU39.CAL:Function_call_interrupts
    742253 ± 15%      +3.6%     769164 ±  3%      +5.6%     783614        interrupts.CPU39.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU39.MCP:Machine_check_polls
    116.78 ± 40%      -5.7%     110.17 ± 46%     -33.5%      77.67 ± 14%  interrupts.CPU39.NMI:Non-maskable_interrupts
    116.78 ± 40%      -5.7%     110.17 ± 46%     -33.5%      77.67 ± 14%  interrupts.CPU39.PMI:Performance_monitoring_interrupts
      9.44 ± 76%     +87.1%      17.67 ±152%     +61.2%      15.22 ± 49%  interrupts.CPU39.RES:Rescheduling_interrupts
      2.22 ± 94%     +20.0%       2.67 ±127%    +160.0%       5.78 ± 32%  interrupts.CPU39.TLB:TLB_shootdowns
      0.11 ±282%    -100.0%       0.00          +200.0%       0.33 ±141%  interrupts.CPU4.123:PCI-MSI.31981656-edge.i40e-eth0-TxRx-87
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU4.298:PCI-MSI.69632-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU4.29:PCI-MSI.48791552-edge.PCIe.PME,pciehp
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU4.302:PCI-MSI.77824-edge.ioat-msix
     34.44 ±191%    +487.7%     202.42 ±304%     -99.7%       0.11 ±282%  interrupts.CPU4.40:PCI-MSI.31981573-edge.i40e-eth0-TxRx-4
      0.22 ±187%    -100.0%       0.00          -100.0%       0.00        interrupts.CPU4.75:PCI-MSI.31981608-edge.i40e-eth0-TxRx-39
    621.33 ± 23%     +40.4%     872.50 ± 53%     +73.5%       1077 ±103%  interrupts.CPU4.CAL:Function_call_interrupts
    774520 ±  3%      -6.7%     722765 ± 18%      +1.2%     783742        interrupts.CPU4.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU4.MCP:Machine_check_polls
     94.89 ± 21%      +5.6%     100.25 ± 25%     -14.8%      80.89 ± 15%  interrupts.CPU4.NMI:Non-maskable_interrupts
     94.89 ± 21%      +5.6%     100.25 ± 25%     -14.8%      80.89 ± 15%  interrupts.CPU4.PMI:Performance_monitoring_interrupts
     27.00 ±183%    +156.2%      69.17 ±188%     -37.9%      16.78 ± 97%  interrupts.CPU4.RES:Rescheduling_interrupts
      1.33 ±106%     +37.5%       1.83 ±150%    +158.3%       3.44 ± 30%  interrupts.CPU4.TLB:TLB_shootdowns
    101.78 ±282%     +15.4%     117.50 ±331%     +27.2%     129.44 ±282%  interrupts.CPU40.76:PCI-MSI.31981609-edge.i40e-eth0-TxRx-40
    543.89 ±  4%      +5.8%     575.17 ± 15%     +12.4%     611.56 ± 31%  interrupts.CPU40.CAL:Function_call_interrupts
    742032 ± 15%      +3.7%     769181 ±  3%      +5.6%     783633        interrupts.CPU40.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU40.MCP:Machine_check_polls
    117.89 ± 40%      -1.9%     115.67 ± 44%     -35.6%      75.89 ± 14%  interrupts.CPU40.NMI:Non-maskable_interrupts
    117.89 ± 40%      -1.9%     115.67 ± 44%     -35.6%      75.89 ± 14%  interrupts.CPU40.PMI:Performance_monitoring_interrupts
      7.11 ± 79%     +69.9%      12.08 ± 50%    +120.3%      15.67 ± 96%  interrupts.CPU40.RES:Rescheduling_interrupts
      2.33 ± 85%      +7.1%       2.50 ±125%    +147.6%       5.78 ± 29%  interrupts.CPU40.TLB:TLB_shootdowns
     37.56 ±225%     -94.9%       1.92 ±300%     +92.3%      72.22 ±245%  interrupts.CPU41.77:PCI-MSI.31981610-edge.i40e-eth0-TxRx-41
    571.11 ± 18%     +38.9%     793.00 ± 95%      -4.8%     543.44        interrupts.CPU41.CAL:Function_call_interrupts
      0.11 ±282%     -25.0%       0.08 ±331%    -100.0%       0.00        interrupts.CPU41.IWI:IRQ_work_interrupts
    742021 ± 15%      +3.6%     769093 ±  3%      +5.6%     783628        interrupts.CPU41.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU41.MCP:Machine_check_polls
    127.11 ± 37%      -5.5%     120.08 ± 44%     -38.3%      78.44 ± 16%  interrupts.CPU41.NMI:Non-maskable_interrupts
    127.11 ± 37%      -5.5%     120.08 ± 44%     -38.3%      78.44 ± 16%  interrupts.CPU41.PMI:Performance_monitoring_interrupts
      9.89 ± 71%    +465.4%      55.92 ±186%     +22.5%      12.11 ± 59%  interrupts.CPU41.RES:Rescheduling_interrupts
      2.33 ±104%      +7.1%       2.50 ±113%    +138.1%       5.56 ± 28%  interrupts.CPU41.TLB:TLB_shootdowns
     16.44 ±280%    +363.2%      76.17 ±319%     -78.4%       3.56 ±282%  interrupts.CPU42.78:PCI-MSI.31981611-edge.i40e-eth0-TxRx-42
    587.78 ± 29%      -1.7%     577.75 ± 22%     +31.4%     772.22 ± 83%  interrupts.CPU42.CAL:Function_call_interrupts
      0.00       +8.3e+100%       0.08 ±331%    -100.0%       0.00        interrupts.CPU42.IWI:IRQ_work_interrupts
    742138 ± 15%      +3.6%     769180 ±  4%      +5.6%     783669        interrupts.CPU42.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU42.MCP:Machine_check_polls
    114.22 ± 31%      +4.8%     119.67 ± 47%     -37.7%      71.11 ± 22%  interrupts.CPU42.NMI:Non-maskable_interrupts
    114.22 ± 31%      +4.8%     119.67 ± 47%     -37.7%      71.11 ± 22%  interrupts.CPU42.PMI:Performance_monitoring_interrupts
     15.00 ±115%     +12.8%      16.92 ± 90%    +152.6%      37.89 ±202%  interrupts.CPU42.RES:Rescheduling_interrupts
      2.11 ± 95%     +18.4%       2.50 ±127%    +157.9%       5.44 ± 27%  interrupts.CPU42.TLB:TLB_shootdowns
      0.00       +8.4e+102%       8.42 ±328% +1.2e+103%      12.22 ±210%  interrupts.CPU43.79:PCI-MSI.31981612-edge.i40e-eth0-TxRx-43
      2710 ±226%     -77.4%     613.17 ± 21%     -79.9%     543.44        interrupts.CPU43.CAL:Function_call_interrupts
    742228 ± 15%      +3.6%     769138 ±  4%      +5.6%     783603        interrupts.CPU43.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU43.MCP:Machine_check_polls
    107.89 ± 33%      +4.7%     113.00 ± 42%     -31.2%      74.22 ± 23%  interrupts.CPU43.NMI:Non-maskable_interrupts
    107.89 ± 33%      +4.7%     113.00 ± 42%     -31.2%      74.22 ± 23%  interrupts.CPU43.PMI:Performance_monitoring_interrupts
     26.33 ±136%     -65.8%       9.00 ±106%     -48.5%      13.56 ± 78%  interrupts.CPU43.RES:Rescheduling_interrupts
      2.11 ± 93%     +22.4%       2.58 ±127%    +168.4%       5.67 ± 23%  interrupts.CPU43.TLB:TLB_shootdowns
     33.56 ±281%    +112.6%      71.33 ±324%    +196.7%      99.56 ±282%  interrupts.CPU44.80:PCI-MSI.31981613-edge.i40e-eth0-TxRx-44
    543.44 ±  2%    +271.5%       2018 ±231%    +133.0%       1266 ±161%  interrupts.CPU44.CAL:Function_call_interrupts
    742102 ± 15%      +3.6%     769068 ±  4%      +5.6%     783686        interrupts.CPU44.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU44.MCP:Machine_check_polls
    123.44 ± 32%      -5.0%     117.33 ± 45%    +133.9%     288.78 ±212%  interrupts.CPU44.NMI:Non-maskable_interrupts
    123.44 ± 32%      -5.0%     117.33 ± 45%    +133.9%     288.78 ±212%  interrupts.CPU44.PMI:Performance_monitoring_interrupts
      9.67 ± 95%     -27.6%       7.00 ± 57%    +274.7%      36.22 ±116%  interrupts.CPU44.RES:Rescheduling_interrupts
      2.11 ± 93%     +50.0%       3.17 ± 99%    +168.4%       5.67 ± 23%  interrupts.CPU44.TLB:TLB_shootdowns
      0.00       +1.7e+104%     165.33 ±254% +1.7e+103%      17.44 ±282%  interrupts.CPU45.81:PCI-MSI.31981614-edge.i40e-eth0-TxRx-45
    537.78            +0.5%     540.33           +14.0%     613.11 ± 32%  interrupts.CPU45.CAL:Function_call_interrupts
      0.00       +8.3e+100%       0.08 ±331%    -100.0%       0.00        interrupts.CPU45.IWI:IRQ_work_interrupts
    742106 ± 15%      +3.6%     769107 ±  3%      +5.6%     783593        interrupts.CPU45.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU45.MCP:Machine_check_polls
    121.00 ± 34%      -1.3%     119.42 ± 47%     -41.5%      70.78 ± 24%  interrupts.CPU45.NMI:Non-maskable_interrupts
    121.00 ± 34%      -1.3%     119.42 ± 47%     -41.5%      70.78 ± 24%  interrupts.CPU45.PMI:Performance_monitoring_interrupts
     10.00 ± 64%     -41.7%       5.83 ± 88%     +65.6%      16.56 ± 72%  interrupts.CPU45.RES:Rescheduling_interrupts
      2.44 ± 79%     +26.1%       3.08 ±104%    +150.0%       6.11 ± 21%  interrupts.CPU45.TLB:TLB_shootdowns
    108.33 ±253%      +9.4%     118.50 ±235%     -98.3%       1.89 ±282%  interrupts.CPU46.82:PCI-MSI.31981615-edge.i40e-eth0-TxRx-46
    537.89 ±  7%      +0.5%     540.75            +1.0%     543.44        interrupts.CPU46.CAL:Function_call_interrupts
      0.11 ±282%    -100.0%       0.00          -100.0%       0.00        interrupts.CPU46.IWI:IRQ_work_interrupts
    742087 ± 15%      +3.6%     769137 ±  3%      +5.6%     783596        interrupts.CPU46.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU46.MCP:Machine_check_polls
    113.33 ± 44%      +4.4%     118.33 ± 45%     -37.2%      71.22 ± 23%  interrupts.CPU46.NMI:Non-maskable_interrupts
    113.33 ± 44%      +4.4%     118.33 ± 45%     -37.2%      71.22 ± 23%  interrupts.CPU46.PMI:Performance_monitoring_interrupts
      9.44 ± 85%     +20.0%      11.33 ±122%     -55.3%       4.22 ± 62%  interrupts.CPU46.RES:Rescheduling_interrupts
      2.56 ± 78%     +30.4%       3.33 ±101%    +139.1%       6.11 ± 14%  interrupts.CPU46.TLB:TLB_shootdowns
     11.33 ±245%     +79.4%      20.33 ±224%     -96.1%       0.44 ±282%  interrupts.CPU47.83:PCI-MSI.31981616-edge.i40e-eth0-TxRx-47
    906.78 ± 52%     -19.9%     726.75 ± 14%     -20.6%     719.78 ± 11%  interrupts.CPU47.CAL:Function_call_interrupts
      0.33 ±141%     -50.0%       0.17 ±223%    -100.0%       0.00        interrupts.CPU47.IWI:IRQ_work_interrupts
    742100 ± 15%      +3.6%     769147 ±  3%      +5.6%     783659        interrupts.CPU47.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU47.MCP:Machine_check_polls
    124.33 ± 47%      +3.5%     128.67 ± 49%     -43.8%      69.89 ± 30%  interrupts.CPU47.NMI:Non-maskable_interrupts
    124.33 ± 47%      +3.5%     128.67 ± 49%     -43.8%      69.89 ± 30%  interrupts.CPU47.PMI:Performance_monitoring_interrupts
     19.67 ±126%     -17.8%      16.17 ± 86%     +61.0%      31.67 ±149%  interrupts.CPU47.RES:Rescheduling_interrupts
      2.89 ± 73%     +26.9%       3.67 ± 91%    +115.4%       6.22 ± 18%  interrupts.CPU47.TLB:TLB_shootdowns
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU48.295:PCI-MSI.65536-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU48.3:IO-APIC.3-edge
      0.22 ±187%     -25.0%       0.17 ±223%      +0.0%       0.22 ±187%  interrupts.CPU48.47:PCI-MSI.31981580-edge.i40e-eth0-TxRx-11
      0.00          -100.0%       0.00       +5.1e+102%       5.11 ±282%  interrupts.CPU48.4:IO-APIC.4-edge.ttyS0
    134.22 ±273%     -99.9%       0.17 ±223%     -99.6%       0.56 ±226%  interrupts.CPU48.84:PCI-MSI.31981617-edge.i40e-eth0-TxRx-48
      0.00       +1.7e+101%       0.17 ±223%    -100.0%       0.00        interrupts.CPU48.95:PCI-MSI.31981628-edge.i40e-eth0-TxRx-59
    589.89 ± 21%     +46.8%     865.75 ±120%      -3.7%     568.00 ± 11%  interrupts.CPU48.CAL:Function_call_interrupts
      0.00       +8.3e+100%       0.08 ±331%    -100.0%       0.00        interrupts.CPU48.IWI:IRQ_work_interrupts
    774480 ±  3%      -6.7%     722507 ± 18%      +1.2%     783708        interrupts.CPU48.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU48.MCP:Machine_check_polls
     93.78 ± 31%    +434.7%     501.42 ±261%     -27.4%      68.11 ± 27%  interrupts.CPU48.NMI:Non-maskable_interrupts
     93.78 ± 31%    +434.7%     501.42 ±261%     -27.4%      68.11 ± 27%  interrupts.CPU48.PMI:Performance_monitoring_interrupts
     19.89 ± 72%     +37.8%      27.42 ±143%     -64.8%       7.00 ±104%  interrupts.CPU48.RES:Rescheduling_interrupts
      3.67 ±126%     -31.8%       2.50 ±124%     +51.5%       5.56 ± 22%  interrupts.CPU48.TLB:TLB_shootdowns
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU49.295:PCI-MSI.65536-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU49.3:IO-APIC.3-edge
      0.11 ±282%    +125.0%       0.25 ±173%    +100.0%       0.22 ±187%  interrupts.CPU49.48:PCI-MSI.31981581-edge.i40e-eth0-TxRx-12
     56.56 ±277%  +20032.6%      11386 ±233%     -94.9%       2.89 ±178%  interrupts.CPU49.85:PCI-MSI.31981618-edge.i40e-eth0-TxRx-49
      0.22 ±187%     -25.0%       0.17 ±223%     +50.0%       0.33 ±141%  interrupts.CPU49.96:PCI-MSI.31981629-edge.i40e-eth0-TxRx-60
    852.33 ± 69%      -8.7%     777.83 ± 91%     -35.6%     548.67 ±  2%  interrupts.CPU49.CAL:Function_call_interrupts
    774464 ±  3%      -6.7%     722496 ± 18%      +1.2%     783587        interrupts.CPU49.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU49.MCP:Machine_check_polls
    100.44 ± 22%    +311.8%     413.58 ±194%     -13.2%      87.22 ± 67%  interrupts.CPU49.NMI:Non-maskable_interrupts
    100.44 ± 22%    +311.8%     413.58 ±194%     -13.2%      87.22 ± 67%  interrupts.CPU49.PMI:Performance_monitoring_interrupts
     12.67 ± 50%     +40.1%      17.75 ± 95%    +171.9%      34.44 ±162%  interrupts.CPU49.RES:Rescheduling_interrupts
      2.22 ± 98%      -2.5%       2.17 ±144%    +175.0%       6.11 ± 36%  interrupts.CPU49.TLB:TLB_shootdowns
      0.00       +3.3e+101%       0.33 ±141% +1.1e+101%       0.11 ±282%  interrupts.CPU5.124:PCI-MSI.31981657-edge.i40e-eth0-TxRx-88
     21.22 ±282%    -100.0%       0.00          -100.0%       0.00        interrupts.CPU5.298:PCI-MSI.288768-edge.ahci[0000:00:11.5]
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU5.298:PCI-MSI.69632-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU5.299:PCI-MSI.71680-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU5.30:PCI-MSI.48807936-edge.PCIe.PME,pciehp
      2667 ±282%     -97.6%      63.25 ±325%     -99.8%       5.78 ±282%  interrupts.CPU5.41:PCI-MSI.31981574-edge.i40e-eth0-TxRx-5
      0.11 ±282%    -100.0%       0.00          -100.0%       0.00        interrupts.CPU5.76:PCI-MSI.31981609-edge.i40e-eth0-TxRx-40
      3265 ±232%     -80.0%     653.50 ± 39%     -75.1%     811.56 ± 79%  interrupts.CPU5.CAL:Function_call_interrupts
      0.00          -100.0%       0.00       +1.1e+101%       0.11 ±282%  interrupts.CPU5.IWI:IRQ_work_interrupts
    774423 ±  3%      -6.7%     722634 ± 18%      +1.2%     783690        interrupts.CPU5.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU5.MCP:Machine_check_polls
     85.56 ± 32%     +14.3%      97.75 ± 26%    +650.8%     642.33 ±253%  interrupts.CPU5.NMI:Non-maskable_interrupts
     85.56 ± 32%     +14.3%      97.75 ± 26%    +650.8%     642.33 ±253%  interrupts.CPU5.PMI:Performance_monitoring_interrupts
      5.56 ± 68%    +611.0%      39.50 ±194%     +52.0%       8.44 ± 84%  interrupts.CPU5.RES:Rescheduling_interrupts
      1.56 ± 80%     +28.6%       2.00 ±139%    +214.3%       4.89 ± 61%  interrupts.CPU5.TLB:TLB_shootdowns
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU50.297:PCI-MSI.67584-edge.ioat-msix
      0.33 ±141%     -50.0%       0.17 ±223%     -66.7%       0.11 ±282%  interrupts.CPU50.49:PCI-MSI.31981582-edge.i40e-eth0-TxRx-13
      0.00          -100.0%       0.00       +5.7e+102%       5.67 ±282%  interrupts.CPU50.4:IO-APIC.4-edge.ttyS0
     18.44 ±265%   +1356.6%     268.67 ±325%     +62.0%      29.89 ±282%  interrupts.CPU50.86:PCI-MSI.31981619-edge.i40e-eth0-TxRx-50
      0.11 ±282%    +125.0%       0.25 ±173%    +200.0%       0.33 ±141%  interrupts.CPU50.97:PCI-MSI.31981630-edge.i40e-eth0-TxRx-61
    542.78 ±  2%      -0.4%     540.58            +2.4%     555.78 ±  4%  interrupts.CPU50.CAL:Function_call_interrupts
      0.00          -100.0%       0.00       +1.1e+101%       0.11 ±282%  interrupts.CPU50.IWI:IRQ_work_interrupts
    774379 ±  3%      -6.7%     722434 ± 18%      +1.2%     783727        interrupts.CPU50.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU50.MCP:Machine_check_polls
    763.78 ±168%     -87.4%      96.17 ± 35%     -31.5%     523.22 ±241%  interrupts.CPU50.NMI:Non-maskable_interrupts
    763.78 ±168%     -87.4%      96.17 ± 35%     -31.5%     523.22 ±241%  interrupts.CPU50.PMI:Performance_monitoring_interrupts
     21.78 ±101%     -41.8%      12.67 ±128%     -26.0%      16.11 ±140%  interrupts.CPU50.RES:Rescheduling_interrupts
      2.89 ±134%     -22.1%       2.25 ±139%    +100.0%       5.78 ± 19%  interrupts.CPU50.TLB:TLB_shootdowns
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU51.295:PCI-MSI.65536-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU51.297:PCI-MSI.67584-edge.ioat-msix
      8.33 ±282%    -100.0%       0.00          -100.0%       0.00        interrupts.CPU51.315:PCI-MSI.376832-edge.ahci[0000:00:17.0]
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU51.3:IO-APIC.3-edge
      0.22 ±187%    -100.0%       0.00            +0.0%       0.22 ±187%  interrupts.CPU51.50:PCI-MSI.31981583-edge.i40e-eth0-TxRx-14
    114.11 ±278%     +23.9%     141.33 ±195%     +37.6%     157.00 ±236%  interrupts.CPU51.87:PCI-MSI.31981620-edge.i40e-eth0-TxRx-51
      0.11 ±282%    +350.0%       0.50 ±100%      +0.0%       0.11 ±282%  interrupts.CPU51.98:PCI-MSI.31981631-edge.i40e-eth0-TxRx-62
    913.11 ± 79%     -41.0%     539.08            -0.4%     909.78 ±109%  interrupts.CPU51.CAL:Function_call_interrupts
    774452 ±  3%      -6.7%     722596 ± 18%      +1.2%     783589        interrupts.CPU51.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU51.MCP:Machine_check_polls
    101.67 ± 15%      +0.2%     101.83 ± 29%     -25.0%      76.22 ± 22%  interrupts.CPU51.NMI:Non-maskable_interrupts
    101.67 ± 15%      +0.2%     101.83 ± 29%     -25.0%      76.22 ± 22%  interrupts.CPU51.PMI:Performance_monitoring_interrupts
      9.11 ± 54%    +122.3%      20.25 ±166%    +529.3%      57.33 ±213%  interrupts.CPU51.RES:Rescheduling_interrupts
      2.33 ± 92%     +10.7%       2.58 ±128%    +104.8%       4.78 ± 23%  interrupts.CPU51.TLB:TLB_shootdowns
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU52.295:PCI-MSI.65536-edge.ioat-msix
      0.00          -100.0%       0.00       +5.1e+102%       5.11 ±282%  interrupts.CPU52.4:IO-APIC.4-edge.ttyS0
      0.11 ±282%    +200.0%       0.33 ±141%      +0.0%       0.11 ±282%  interrupts.CPU52.51:PCI-MSI.31981584-edge.i40e-eth0-TxRx-15
      1.22 ±203%   +9704.5%     119.83 ±295%    +345.5%       5.44 ±121%  interrupts.CPU52.88:PCI-MSI.31981621-edge.i40e-eth0-TxRx-52
      0.22 ±187%     -62.5%       0.08 ±331%     -50.0%       0.11 ±282%  interrupts.CPU52.99:PCI-MSI.31981632-edge.i40e-eth0-TxRx-63
    811.11 ± 71%     -33.3%     541.33           -28.4%     580.56 ± 17%  interrupts.CPU52.CAL:Function_call_interrupts
      0.00       +8.3e+100%       0.08 ±331%    -100.0%       0.00        interrupts.CPU52.IWI:IRQ_work_interrupts
    774498 ±  3%      -6.7%     722808 ± 18%      +1.2%     783640        interrupts.CPU52.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU52.MCP:Machine_check_polls
    102.67 ± 18%      +0.0%     102.67 ± 31%     -27.2%      74.78 ± 22%  interrupts.CPU52.NMI:Non-maskable_interrupts
    102.67 ± 18%      +0.0%     102.67 ± 31%     -27.2%      74.78 ± 22%  interrupts.CPU52.PMI:Performance_monitoring_interrupts
      8.33 ± 47%    +119.0%      18.25 ±137%    +120.0%      18.33 ±162%  interrupts.CPU52.RES:Rescheduling_interrupts
      2.33 ± 78%     +21.4%       2.83 ±122%    +133.3%       5.44 ± 21%  interrupts.CPU52.TLB:TLB_shootdowns
      0.00       +1.7e+101%       0.17 ±223% +2.2e+101%       0.22 ±187%  interrupts.CPU53.100:PCI-MSI.31981633-edge.i40e-eth0-TxRx-64
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU53.3:IO-APIC.3-edge
      0.00       +4.3e+102%       4.33 ±331%    -100.0%       0.00        interrupts.CPU53.4:IO-APIC.4-edge.ttyS0
      0.00       +1.7e+101%       0.17 ±223% +1.1e+101%       0.11 ±282%  interrupts.CPU53.52:PCI-MSI.31981585-edge.i40e-eth0-TxRx-16
      5.22 ±282%     -18.6%       4.25 ±227%     -44.7%       2.89 ±282%  interrupts.CPU53.89:PCI-MSI.31981622-edge.i40e-eth0-TxRx-53
    540.00 ±  2%     +12.4%     607.17 ± 34%      +1.5%     547.89 ±  2%  interrupts.CPU53.CAL:Function_call_interrupts
      0.00       +8.3e+100%       0.08 ±331% +2.2e+101%       0.22 ±282%  interrupts.CPU53.IWI:IRQ_work_interrupts
    774539 ±  3%      -6.7%     722657 ± 18%      +1.2%     783620        interrupts.CPU53.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU53.MCP:Machine_check_polls
     99.22 ± 26%     +12.1%     111.25 ± 33%    +295.9%     392.78 ±231%  interrupts.CPU53.NMI:Non-maskable_interrupts
     99.22 ± 26%     +12.1%     111.25 ± 33%    +295.9%     392.78 ±231%  interrupts.CPU53.PMI:Performance_monitoring_interrupts
      9.89 ± 63%    +219.4%      31.58 ±142%     +31.5%      13.00 ±127%  interrupts.CPU53.RES:Rescheduling_interrupts
      2.22 ± 89%     +16.2%       2.58 ±135%    +140.0%       5.33 ± 27%  interrupts.CPU53.TLB:TLB_shootdowns
      0.22 ±187%     -62.5%       0.08 ±331%     -50.0%       0.11 ±282%  interrupts.CPU54.101:PCI-MSI.31981634-edge.i40e-eth0-TxRx-65
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU54.297:PCI-MSI.67584-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU54.298:PCI-MSI.69632-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU54.3:IO-APIC.3-edge
      5.89 ±282%    -100.0%       0.00          -100.0%       0.00        interrupts.CPU54.4:IO-APIC.4-edge.ttyS0
      0.00          -100.0%       0.00       +1.1e+101%       0.11 ±282%  interrupts.CPU54.53:PCI-MSI.31981586-edge.i40e-eth0-TxRx-17
      9.89 ±234%    +850.6%      94.00 ±329%     -66.3%       3.33 ±282%  interrupts.CPU54.90:PCI-MSI.31981623-edge.i40e-eth0-TxRx-54
    535.67          +104.0%       1093 ±112%      -0.0%     535.56 ±  5%  interrupts.CPU54.CAL:Function_call_interrupts
      0.00       +8.3e+100%       0.08 ±331%    -100.0%       0.00        interrupts.CPU54.IWI:IRQ_work_interrupts
    774484 ±  3%      -6.7%     722929 ± 18%      +1.2%     783621        interrupts.CPU54.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU54.MCP:Machine_check_polls
     95.56 ± 26%    +163.6%     251.92 ±218%    +134.0%     223.56 ±186%  interrupts.CPU54.NMI:Non-maskable_interrupts
     95.56 ± 26%    +163.6%     251.92 ±218%    +134.0%     223.56 ±186%  interrupts.CPU54.PMI:Performance_monitoring_interrupts
      7.44 ± 63%    +376.9%      35.50 ±128%     +62.7%      12.11 ± 75%  interrupts.CPU54.RES:Rescheduling_interrupts
      2.78 ± 84%      +5.0%       2.92 ±120%    +116.0%       6.00 ± 56%  interrupts.CPU54.TLB:TLB_shootdowns
      0.11 ±282%     -25.0%       0.08 ±331%      +0.0%       0.11 ±282%  interrupts.CPU55.102:PCI-MSI.31981635-edge.i40e-eth0-TxRx-66
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU55.295:PCI-MSI.65536-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU55.298:PCI-MSI.69632-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU55.3:IO-APIC.3-edge
      0.00       +8.3e+100%       0.08 ±331% +2.2e+101%       0.22 ±187%  interrupts.CPU55.54:PCI-MSI.31981587-edge.i40e-eth0-TxRx-18
      0.22 ±187%   +6162.5%      13.92 ±180%  +59950.0%     133.44 ±282%  interrupts.CPU55.91:PCI-MSI.31981624-edge.i40e-eth0-TxRx-55
    547.56 ±  3%      +1.0%     553.00 ±  5%     +18.6%     649.44 ± 40%  interrupts.CPU55.CAL:Function_call_interrupts
      0.00       +8.3e+100%       0.08 ±331% +1.1e+101%       0.11 ±282%  interrupts.CPU55.IWI:IRQ_work_interrupts
    774560 ±  3%      -6.7%     722465 ± 18%      +1.2%     783715        interrupts.CPU55.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU55.MCP:Machine_check_polls
    314.33 ±196%     -68.8%      98.00 ± 37%     +86.2%     585.33 ±246%  interrupts.CPU55.NMI:Non-maskable_interrupts
    314.33 ±196%     -68.8%      98.00 ± 37%     +86.2%     585.33 ±246%  interrupts.CPU55.PMI:Performance_monitoring_interrupts
     26.00 ±135%     -50.3%      12.92 ±128%     -49.1%      13.22 ±131%  interrupts.CPU55.RES:Rescheduling_interrupts
      2.44 ± 88%      -4.5%       2.33 ±130%    +136.4%       5.78 ± 15%  interrupts.CPU55.TLB:TLB_shootdowns
      0.44 ±111%     -81.2%       0.08 ±331%     -50.0%       0.22 ±187%  interrupts.CPU56.103:PCI-MSI.31981636-edge.i40e-eth0-TxRx-67
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU56.298:PCI-MSI.69632-edge.ioat-msix
      0.00       +4.2e+102%       4.17 ±331%    -100.0%       0.00        interrupts.CPU56.4:IO-APIC.4-edge.ttyS0
      0.22 ±187%     +50.0%       0.33 ±141%     -50.0%       0.11 ±282%  interrupts.CPU56.55:PCI-MSI.31981588-edge.i40e-eth0-TxRx-19
     33.78 ±282%     -92.8%       2.42 ±319%  +43008.9%      14561 ±282%  interrupts.CPU56.92:PCI-MSI.31981625-edge.i40e-eth0-TxRx-56
    834.00 ± 99%      +3.6%     864.25 ±103%     -34.5%     545.89        interrupts.CPU56.CAL:Function_call_interrupts
    774616 ±  3%      -6.7%     722502 ± 18%      +1.2%     783645        interrupts.CPU56.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU56.MCP:Machine_check_polls
    100.56 ± 19%      -1.8%      98.75 ± 28%     -25.9%      74.56 ± 20%  interrupts.CPU56.NMI:Non-maskable_interrupts
    100.56 ± 19%      -1.8%      98.75 ± 28%     -25.9%      74.56 ± 20%  interrupts.CPU56.PMI:Performance_monitoring_interrupts
     18.89 ± 91%     +28.8%      24.33 ±131%     -47.6%       9.89 ± 74%  interrupts.CPU56.RES:Rescheduling_interrupts
      2.89 ± 91%     -16.3%       2.42 ±145%    +103.8%       5.89 ± 12%  interrupts.CPU56.TLB:TLB_shootdowns
      0.11 ±282%    -100.0%       0.00            +0.0%       0.11 ±282%  interrupts.CPU57.104:PCI-MSI.31981637-edge.i40e-eth0-TxRx-68
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU57.295:PCI-MSI.65536-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU57.299:PCI-MSI.71680-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU57.3:IO-APIC.3-edge
      0.00       +3.8e+102%       3.83 ±331%    -100.0%       0.00        interrupts.CPU57.4:IO-APIC.4-edge.ttyS0
      0.11 ±282%     +50.0%       0.17 ±223%    +100.0%       0.22 ±187%  interrupts.CPU57.56:PCI-MSI.31981589-edge.i40e-eth0-TxRx-20
    289.78 ±219%     -99.5%       1.58 ±313%     -96.7%       9.67 ±275%  interrupts.CPU57.93:PCI-MSI.31981626-edge.i40e-eth0-TxRx-57
      1141 ±100%     -52.5%     541.83           -45.2%     625.78 ± 36%  interrupts.CPU57.CAL:Function_call_interrupts
      0.00       +8.3e+100%       0.08 ±331% +1.1e+101%       0.11 ±282%  interrupts.CPU57.IWI:IRQ_work_interrupts
    774466 ±  3%      -6.7%     722530 ± 18%      +1.2%     783693        interrupts.CPU57.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU57.MCP:Machine_check_polls
     96.00 ± 25%      +0.3%      96.25 ± 33%    +534.5%     609.11 ±245%  interrupts.CPU57.NMI:Non-maskable_interrupts
     96.00 ± 25%      +0.3%      96.25 ± 33%    +534.5%     609.11 ±245%  interrupts.CPU57.PMI:Performance_monitoring_interrupts
      8.78 ± 71%      +6.3%       9.33 ±125%    +103.8%      17.89 ±125%  interrupts.CPU57.RES:Rescheduling_interrupts
      2.33 ± 90%      -3.6%       2.25 ±143%    +128.6%       5.33 ± 12%  interrupts.CPU57.TLB:TLB_shootdowns
      0.00       +2.5e+101%       0.25 ±173% +1.1e+101%       0.11 ±282%  interrupts.CPU58.105:PCI-MSI.31981638-edge.i40e-eth0-TxRx-69
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU58.297:PCI-MSI.67584-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU58.3:IO-APIC.3-edge
      0.33 ±141%     -50.0%       0.17 ±223%    -100.0%       0.00        interrupts.CPU58.57:PCI-MSI.31981590-edge.i40e-eth0-TxRx-21
      0.00       +3.5e+104%     349.42 ±315% +1.2e+104%     115.44 ±238%  interrupts.CPU58.94:PCI-MSI.31981627-edge.i40e-eth0-TxRx-58
    894.44 ±113%     -38.6%     549.17 ±  3%     +19.8%       1071 ±142%  interrupts.CPU58.CAL:Function_call_interrupts
      0.00       +8.3e+100%       0.08 ±331%    -100.0%       0.00        interrupts.CPU58.IWI:IRQ_work_interrupts
    774497 ±  3%      -6.7%     722640 ± 18%      +1.2%     783760        interrupts.CPU58.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU58.MCP:Machine_check_polls
    101.44 ± 15%      -5.0%      96.33 ± 40%     -22.1%      79.00 ± 10%  interrupts.CPU58.NMI:Non-maskable_interrupts
    101.44 ± 15%      -5.0%      96.33 ± 40%     -22.1%      79.00 ± 10%  interrupts.CPU58.PMI:Performance_monitoring_interrupts
      5.00 ± 43%     +78.3%       8.92 ± 74%    +268.9%      18.44 ±116%  interrupts.CPU58.RES:Rescheduling_interrupts
      2.11 ±105%     +42.1%       3.00 ±120%    +163.2%       5.56 ± 14%  interrupts.CPU58.TLB:TLB_shootdowns
      0.11 ±282%    +350.0%       0.50 ±100%      +0.0%       0.11 ±282%  interrupts.CPU59.106:PCI-MSI.31981639-edge.i40e-eth0-TxRx-70
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU59.297:PCI-MSI.67584-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU59.3:IO-APIC.3-edge
      0.00       +1.7e+101%       0.17 ±223% +1.1e+101%       0.11 ±282%  interrupts.CPU59.58:PCI-MSI.31981591-edge.i40e-eth0-TxRx-22
      0.00       +4.1e+103%      40.58 ±289% +2.6e+103%      26.11 ±187%  interrupts.CPU59.95:PCI-MSI.31981628-edge.i40e-eth0-TxRx-59
    543.22 ±  2%    +311.4%       2234 ±248%      +0.6%     546.33        interrupts.CPU59.CAL:Function_call_interrupts
      0.00       +1.7e+101%       0.17 ±223% +1.1e+101%       0.11 ±282%  interrupts.CPU59.IWI:IRQ_work_interrupts
    774502 ±  3%      -6.7%     722579 ± 18%      +1.2%     783773        interrupts.CPU59.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU59.MCP:Machine_check_polls
     95.00 ± 30%    +144.9%     232.67 ±183%    +496.7%     566.89 ±244%  interrupts.CPU59.NMI:Non-maskable_interrupts
     95.00 ± 30%    +144.9%     232.67 ±183%    +496.7%     566.89 ±244%  interrupts.CPU59.PMI:Performance_monitoring_interrupts
     16.00 ±135%     -28.6%      11.42 ± 82%     -32.6%      10.78 ± 86%  interrupts.CPU59.RES:Rescheduling_interrupts
      2.33 ± 96%     +42.9%       3.33 ±109%    +133.3%       5.44 ± 23%  interrupts.CPU59.TLB:TLB_shootdowns
      0.00       +1.7e+101%       0.17 ±223% +2.2e+101%       0.22 ±187%  interrupts.CPU6.125:PCI-MSI.31981658-edge.i40e-eth0-TxRx-89
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU6.297:PCI-MSI.67584-edge.ioat-msix
     11109 ±282%     -99.2%      84.42 ±317%     -98.4%     176.00 ±166%  interrupts.CPU6.42:PCI-MSI.31981575-edge.i40e-eth0-TxRx-6
      0.33 ±141%     -25.0%       0.25 ±173%     -33.3%       0.22 ±187%  interrupts.CPU6.77:PCI-MSI.31981610-edge.i40e-eth0-TxRx-41
    795.44 ± 80%     +44.0%       1145 ±110%     -24.6%     599.67 ± 13%  interrupts.CPU6.CAL:Function_call_interrupts
      0.00       +8.3e+100%       0.08 ±331%    -100.0%       0.00        interrupts.CPU6.IWI:IRQ_work_interrupts
    774651 ±  3%      -6.7%     722932 ± 18%      +1.2%     783584        interrupts.CPU6.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU6.MCP:Machine_check_polls
    104.67 ± 20%    +468.5%     595.00 ±282%     +14.3%     119.67 ±113%  interrupts.CPU6.NMI:Non-maskable_interrupts
    104.67 ± 20%    +468.5%     595.00 ±282%     +14.3%     119.67 ±113%  interrupts.CPU6.PMI:Performance_monitoring_interrupts
     17.00 ± 80%    +271.6%      63.17 ±208%      +8.5%      18.44 ± 84%  interrupts.CPU6.RES:Rescheduling_interrupts
      1.56 ±105%     +23.2%       1.92 ±142%    +150.0%       3.89 ± 35%  interrupts.CPU6.TLB:TLB_shootdowns
      0.11 ±282%     +50.0%       0.17 ±223%    +100.0%       0.22 ±187%  interrupts.CPU60.107:PCI-MSI.31981640-edge.i40e-eth0-TxRx-71
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU60.295:PCI-MSI.65536-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU60.3:IO-APIC.3-edge
      0.44 ±111%     -62.5%       0.17 ±223%     -50.0%       0.22 ±187%  interrupts.CPU60.59:PCI-MSI.31981592-edge.i40e-eth0-TxRx-23
    281.00 ±281%     -64.3%     100.25 ±296%      +7.4%     301.89 ±142%  interrupts.CPU60.96:PCI-MSI.31981629-edge.i40e-eth0-TxRx-60
    864.11 ±107%     -36.6%     548.25 ±  3%      -3.3%     835.89 ± 87%  interrupts.CPU60.CAL:Function_call_interrupts
    774542 ±  3%      -6.7%     722360 ± 18%      +1.2%     783781        interrupts.CPU60.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU60.MCP:Machine_check_polls
     88.56 ± 36%     +15.7%     102.42 ± 58%     -11.4%      78.44 ± 18%  interrupts.CPU60.NMI:Non-maskable_interrupts
     88.56 ± 36%     +15.7%     102.42 ± 58%     -11.4%      78.44 ± 18%  interrupts.CPU60.PMI:Performance_monitoring_interrupts
     15.89 ±132%    +134.4%      37.25 ±205%     -21.7%      12.44 ± 49%  interrupts.CPU60.RES:Rescheduling_interrupts
      1.89 ±112%     +19.1%       2.25 ±135%    +182.4%       5.33 ± 19%  interrupts.CPU60.TLB:TLB_shootdowns
      0.22 ±187%     -25.0%       0.17 ±223%     -50.0%       0.11 ±282%  interrupts.CPU61.108:PCI-MSI.31981641-edge.i40e-eth0-TxRx-72
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU61.295:PCI-MSI.65536-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU61.297:PCI-MSI.67584-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU61.298:PCI-MSI.69632-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU61.3:IO-APIC.3-edge
      0.00       +3.8e+102%       3.83 ±331%    -100.0%       0.00        interrupts.CPU61.4:IO-APIC.4-edge.ttyS0
      0.33 ±141%     -50.0%       0.17 ±223%    -100.0%       0.00        interrupts.CPU61.60:PCI-MSI.31981593-edge.i40e-eth0-TxRx-24
    201.78 ±279%     -26.9%     147.50 ±324%     -97.4%       5.33 ±172%  interrupts.CPU61.97:PCI-MSI.31981630-edge.i40e-eth0-TxRx-61
      1844 ±146%     -71.3%     528.92 ± 10%     -67.1%     607.00 ± 28%  interrupts.CPU61.CAL:Function_call_interrupts
      0.11 ±282%    -100.0%       0.00          -100.0%       0.00        interrupts.CPU61.IWI:IRQ_work_interrupts
    774551 ±  3%      -6.7%     722486 ± 18%      +1.2%     783806        interrupts.CPU61.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU61.MCP:Machine_check_polls
    799.33 ±191%     -73.6%     211.33 ±196%     -91.2%      70.67 ± 25%  interrupts.CPU61.NMI:Non-maskable_interrupts
    799.33 ±191%     -73.6%     211.33 ±196%     -91.2%      70.67 ± 25%  interrupts.CPU61.PMI:Performance_monitoring_interrupts
     19.44 ±125%      +5.9%      20.58 ±133%     -38.3%      12.00 ± 79%  interrupts.CPU61.RES:Rescheduling_interrupts
      2.67 ± 75%     -12.5%       2.33 ±137%    +112.5%       5.67 ± 22%  interrupts.CPU61.TLB:TLB_shootdowns
      0.33 ±141%     -75.0%       0.08 ±331%     -33.3%       0.22 ±187%  interrupts.CPU62.109:PCI-MSI.31981642-edge.i40e-eth0-TxRx-73
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU62.297:PCI-MSI.67584-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU62.299:PCI-MSI.71680-edge.ioat-msix
      0.11 ±282%     -25.0%       0.08 ±331%      +0.0%       0.11 ±282%  interrupts.CPU62.61:PCI-MSI.31981594-edge.i40e-eth0-TxRx-25
    349.00 ±217%    +161.3%     911.92 ±195%     -96.9%      10.78 ±279%  interrupts.CPU62.98:PCI-MSI.31981631-edge.i40e-eth0-TxRx-62
    545.33 ±  3%      +0.2%     546.67 ±  4%     +19.1%     649.22 ± 40%  interrupts.CPU62.CAL:Function_call_interrupts
    774524 ±  3%      -6.7%     722603 ± 18%      +1.2%     783769        interrupts.CPU62.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU62.MCP:Machine_check_polls
    235.67 ±177%     -42.2%     136.17 ± 88%     -71.1%      68.22 ± 29%  interrupts.CPU62.NMI:Non-maskable_interrupts
    235.67 ±177%     -42.2%     136.17 ± 88%     -71.1%      68.22 ± 29%  interrupts.CPU62.PMI:Performance_monitoring_interrupts
     10.00 ± 55%     +85.0%      18.50 ±184%      -1.1%       9.89 ± 83%  interrupts.CPU62.RES:Rescheduling_interrupts
      2.33 ± 85%      +3.6%       2.42 ±126%    +133.3%       5.44 ± 23%  interrupts.CPU62.TLB:TLB_shootdowns
      0.00       +2.5e+101%       0.25 ±173% +1.1e+101%       0.11 ±282%  interrupts.CPU63.110:PCI-MSI.31981643-edge.i40e-eth0-TxRx-74
      0.00       +8.3e+100%       0.08 ±331% +1.1e+101%       0.11 ±282%  interrupts.CPU63.62:PCI-MSI.31981595-edge.i40e-eth0-TxRx-26
    178.00 ±268%     +48.4%     264.08 ±242%   +1037.1%       2024 ±282%  interrupts.CPU63.99:PCI-MSI.31981632-edge.i40e-eth0-TxRx-63
    536.56           +67.8%     900.58 ± 97%      +3.6%     556.11 ±  6%  interrupts.CPU63.CAL:Function_call_interrupts
    774640 ±  3%      -6.7%     722593 ± 18%      +1.2%     783749        interrupts.CPU63.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU63.MCP:Machine_check_polls
     98.89 ± 27%    +199.3%     296.00 ±229%     -36.6%      62.67 ± 35%  interrupts.CPU63.NMI:Non-maskable_interrupts
     98.89 ± 27%    +199.3%     296.00 ±229%     -36.6%      62.67 ± 35%  interrupts.CPU63.PMI:Performance_monitoring_interrupts
     12.00 ± 60%     +26.4%      15.17 ± 81%     -26.9%       8.78 ± 61%  interrupts.CPU63.RES:Rescheduling_interrupts
      2.89 ± 68%     -13.5%       2.50 ±119%     +92.3%       5.56 ± 26%  interrupts.CPU63.TLB:TLB_shootdowns
     48.11 ±282%     -32.8%      32.33 ±265%     -92.8%       3.44 ±272%  interrupts.CPU64.100:PCI-MSI.31981633-edge.i40e-eth0-TxRx-64
      0.22 ±187%     -62.5%       0.08 ±331%    -100.0%       0.00        interrupts.CPU64.111:PCI-MSI.31981644-edge.i40e-eth0-TxRx-75
      0.00       +1.6e+103%      15.92 ±331%    -100.0%       0.00        interrupts.CPU64.296:PCI-MSI.288768-edge.ahci[0000:00:11.5]
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU64.297:PCI-MSI.67584-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU64.298:PCI-MSI.69632-edge.ioat-msix
      0.22 ±187%    -100.0%       0.00           +50.0%       0.33 ±141%  interrupts.CPU64.63:PCI-MSI.31981596-edge.i40e-eth0-TxRx-27
    624.67 ± 26%     +50.5%     940.42 ± 96%     +24.4%     777.00 ± 53%  interrupts.CPU64.CAL:Function_call_interrupts
      0.00          -100.0%       0.00       +1.1e+101%       0.11 ±282%  interrupts.CPU64.IWI:IRQ_work_interrupts
    774613 ±  3%      -6.7%     722509 ± 18%      +1.2%     783840        interrupts.CPU64.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU64.MCP:Machine_check_polls
    100.22 ± 26%     -13.4%      86.83 ± 34%     -33.5%      66.67 ± 43%  interrupts.CPU64.NMI:Non-maskable_interrupts
    100.22 ± 26%     -13.4%      86.83 ± 34%     -33.5%      66.67 ± 43%  interrupts.CPU64.PMI:Performance_monitoring_interrupts
     12.44 ± 89%      -0.9%      12.33 ± 54%     -29.5%       8.78 ± 55%  interrupts.CPU64.RES:Rescheduling_interrupts
      3.00 ± 98%     -11.1%       2.67 ±108%     +81.5%       5.44 ± 19%  interrupts.CPU64.TLB:TLB_shootdowns
     16.11 ±196%    +434.8%      86.17 ±293%     -11.7%      14.22 ±210%  interrupts.CPU65.101:PCI-MSI.31981634-edge.i40e-eth0-TxRx-65
      0.11 ±282%     -25.0%       0.08 ±331%    +100.0%       0.22 ±187%  interrupts.CPU65.112:PCI-MSI.31981645-edge.i40e-eth0-TxRx-76
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU65.298:PCI-MSI.67584-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU65.300:PCI-MSI.73728-edge.ioat-msix
      0.00          -100.0%       0.00       +2.2e+101%       0.22 ±187%  interrupts.CPU65.64:PCI-MSI.31981597-edge.i40e-eth0-TxRx-28
    563.00 ±  3%      -5.1%     534.08 ± 11%     +29.6%     729.44 ± 49%  interrupts.CPU65.CAL:Function_call_interrupts
    774537 ±  3%      -6.7%     722512 ± 18%      +1.2%     783856        interrupts.CPU65.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU65.MCP:Machine_check_polls
    102.44 ± 18%    +119.5%     224.92 ±190%     -35.9%      65.67 ± 33%  interrupts.CPU65.NMI:Non-maskable_interrupts
    102.44 ± 18%    +119.5%     224.92 ±190%     -35.9%      65.67 ± 33%  interrupts.CPU65.PMI:Performance_monitoring_interrupts
     12.22 ± 77%     -16.8%      10.17 ± 82%     -36.4%       7.78 ± 80%  interrupts.CPU65.RES:Rescheduling_interrupts
      2.56 ± 78%      +7.6%       2.75 ±109%    +156.5%       6.56 ± 38%  interrupts.CPU65.TLB:TLB_shootdowns
     28.00 ±282%    +161.9%      73.33 ±331%     -32.5%      18.89 ±264%  interrupts.CPU66.102:PCI-MSI.31981635-edge.i40e-eth0-TxRx-66
      0.00       +1.7e+101%       0.17 ±223% +3.3e+101%       0.33 ±141%  interrupts.CPU66.113:PCI-MSI.31981646-edge.i40e-eth0-TxRx-77
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU66.298:PCI-MSI.69632-edge.ioat-msix
      0.22 ±187%     -62.5%       0.08 ±331%     -50.0%       0.11 ±282%  interrupts.CPU66.65:PCI-MSI.31981598-edge.i40e-eth0-TxRx-29
    716.44 ± 63%      +6.5%     762.75 ± 87%     -18.2%     586.11 ± 15%  interrupts.CPU66.CAL:Function_call_interrupts
    774568 ±  3%      -6.7%     722454 ± 18%      +1.2%     783635        interrupts.CPU66.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU66.MCP:Machine_check_polls
     98.56 ± 26%    +437.9%     530.08 ±210%    +286.4%     380.78 ±236%  interrupts.CPU66.NMI:Non-maskable_interrupts
     98.56 ± 26%    +437.9%     530.08 ±210%    +286.4%     380.78 ±236%  interrupts.CPU66.PMI:Performance_monitoring_interrupts
     58.22 ±185%      -8.3%      53.42 ±269%     -77.3%      13.22 ±135%  interrupts.CPU66.RES:Rescheduling_interrupts
      2.44 ± 92%      +5.7%       2.58 ±118%    +113.6%       5.22 ± 29%  interrupts.CPU66.TLB:TLB_shootdowns
     18.22 ±154%    +980.2%     196.83 ±307%     -98.8%       0.22 ±187%  interrupts.CPU67.103:PCI-MSI.31981636-edge.i40e-eth0-TxRx-67
      0.00       +8.3e+100%       0.08 ±331% +2.2e+101%       0.22 ±187%  interrupts.CPU67.114:PCI-MSI.31981647-edge.i40e-eth0-TxRx-78
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU67.297:PCI-MSI.67584-edge.ioat-msix
      5.11 ±282%    -100.0%       0.00          -100.0%       0.00        interrupts.CPU67.4:IO-APIC.4-edge.ttyS0
      0.00       +1.7e+101%       0.17 ±223% +1.1e+101%       0.11 ±282%  interrupts.CPU67.66:PCI-MSI.31981599-edge.i40e-eth0-TxRx-30
    827.56 ± 95%     +63.1%       1349 ±195%     +19.8%     991.67 ±125%  interrupts.CPU67.CAL:Function_call_interrupts
    774578 ±  3%      -6.7%     722688 ± 18%      +1.2%     783762        interrupts.CPU67.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU67.MCP:Machine_check_polls
    107.89 ± 18%      -7.5%      99.75 ± 27%     -35.2%      69.89 ± 32%  interrupts.CPU67.NMI:Non-maskable_interrupts
    107.89 ± 18%      -7.5%      99.75 ± 27%     -35.2%      69.89 ± 32%  interrupts.CPU67.PMI:Performance_monitoring_interrupts
     26.89 ±173%     -58.8%      11.08 ±113%     -65.7%       9.22 ± 68%  interrupts.CPU67.RES:Rescheduling_interrupts
      2.89 ± 82%     -22.1%       2.25 ±123%    +107.7%       6.00 ± 17%  interrupts.CPU67.TLB:TLB_shootdowns
     74.56 ±252%     -90.4%       7.17 ±331%    +108.3%     155.33 ±282%  interrupts.CPU68.104:PCI-MSI.31981637-edge.i40e-eth0-TxRx-68
      0.11 ±282%     +50.0%       0.17 ±223%    -100.0%       0.00        interrupts.CPU68.115:PCI-MSI.31981648-edge.i40e-eth0-TxRx-79
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU68.297:PCI-MSI.67584-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU68.299:PCI-MSI.71680-edge.ioat-msix
      0.11 ±282%    +125.0%       0.25 ±173%      +0.0%       0.11 ±282%  interrupts.CPU68.67:PCI-MSI.31981600-edge.i40e-eth0-TxRx-31
    645.89 ± 24%      -6.9%     601.58 ± 31%     +81.4%       1171 ± 98%  interrupts.CPU68.CAL:Function_call_interrupts
      0.11 ±282%    -100.0%       0.00          -100.0%       0.00        interrupts.CPU68.IWI:IRQ_work_interrupts
    774540 ±  3%      -6.7%     722502 ± 18%      +1.2%     783678        interrupts.CPU68.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU68.MCP:Machine_check_polls
    945.78 ±200%     -89.6%      98.50 ± 27%     -77.3%     214.67 ±181%  interrupts.CPU68.NMI:Non-maskable_interrupts
    945.78 ±200%     -89.6%      98.50 ± 27%     -77.3%     214.67 ±181%  interrupts.CPU68.PMI:Performance_monitoring_interrupts
     30.00 ±141%    +159.7%      77.92 ±233%     -34.4%      19.67 ± 60%  interrupts.CPU68.RES:Rescheduling_interrupts
      2.00 ± 74%     +41.7%       2.83 ±122%    +183.3%       5.67 ± 18%  interrupts.CPU68.TLB:TLB_shootdowns
      0.00       +8.3e+102%       8.33 ±232%   +1e+104%     100.00 ±282%  interrupts.CPU69.105:PCI-MSI.31981638-edge.i40e-eth0-TxRx-69
      0.22 ±187%     +50.0%       0.33 ±141%    +100.0%       0.44 ±111%  interrupts.CPU69.116:PCI-MSI.31981649-edge.i40e-eth0-TxRx-80
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU69.3:IO-APIC.3-edge
      0.11 ±282%    +125.0%       0.25 ±173%      +0.0%       0.11 ±282%  interrupts.CPU69.68:PCI-MSI.31981601-edge.i40e-eth0-TxRx-32
    563.00 ±  8%     +47.6%     831.08 ±111%     +19.1%     670.44 ± 47%  interrupts.CPU69.CAL:Function_call_interrupts
    774477 ±  3%      -6.7%     722587 ± 18%      +1.2%     783831        interrupts.CPU69.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU69.MCP:Machine_check_polls
    245.67 ±164%     -59.5%      99.58 ± 22%     -71.4%      70.22 ± 36%  interrupts.CPU69.NMI:Non-maskable_interrupts
    245.67 ±164%     -59.5%      99.58 ± 22%     -71.4%      70.22 ± 36%  interrupts.CPU69.PMI:Performance_monitoring_interrupts
     12.67 ± 89%     +14.5%      14.50 ±104%     +94.7%      24.67 ± 66%  interrupts.CPU69.RES:Rescheduling_interrupts
      2.78 ± 75%      +5.0%       2.92 ±107%     +88.0%       5.22 ± 21%  interrupts.CPU69.TLB:TLB_shootdowns
      0.11 ±282%     -25.0%       0.08 ±331%    +100.0%       0.22 ±187%  interrupts.CPU7.126:PCI-MSI.31981659-edge.i40e-eth0-TxRx-90
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU7.297:PCI-MSI.67584-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU7.299:PCI-MSI.71680-edge.ioat-msix
      1.33 ±282%    +756.2%      11.42 ±331%    +533.3%       8.44 ±154%  interrupts.CPU7.43:PCI-MSI.31981576-edge.i40e-eth0-TxRx-7
      0.22 ±187%    -100.0%       0.00            +0.0%       0.22 ±187%  interrupts.CPU7.78:PCI-MSI.31981611-edge.i40e-eth0-TxRx-42
    961.67 ± 80%     -36.6%     609.58 ± 34%     -25.0%     720.78 ± 44%  interrupts.CPU7.CAL:Function_call_interrupts
      0.22 ±282%    -100.0%       0.00          -100.0%       0.00        interrupts.CPU7.IWI:IRQ_work_interrupts
    774552 ±  3%      -6.7%     722407 ± 18%      +1.2%     783708        interrupts.CPU7.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU7.MCP:Machine_check_polls
    764.22 ±250%     -87.6%      94.50 ± 32%     -65.4%     264.22 ±199%  interrupts.CPU7.NMI:Non-maskable_interrupts
    764.22 ±250%     -87.6%      94.50 ± 32%     -65.4%     264.22 ±199%  interrupts.CPU7.PMI:Performance_monitoring_interrupts
     26.56 ±107%     +18.6%      31.50 ±161%     -38.1%      16.44 ± 75%  interrupts.CPU7.RES:Rescheduling_interrupts
      2.00 ± 88%      +4.2%       2.08 ±149%    +100.0%       4.00 ± 33%  interrupts.CPU7.TLB:TLB_shootdowns
      4.22 ±282%  +32840.8%       1390 ±330%   +2934.2%     128.11 ±198%  interrupts.CPU70.106:PCI-MSI.31981639-edge.i40e-eth0-TxRx-70
      0.22 ±187%     -25.0%       0.17 ±223%     -50.0%       0.11 ±282%  interrupts.CPU70.117:PCI-MSI.31981650-edge.i40e-eth0-TxRx-81
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU70.297:PCI-MSI.67584-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU70.298:PCI-MSI.69632-edge.ioat-msix
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU70.3:IO-APIC.3-edge
      0.11 ±282%    +125.0%       0.25 ±173%    -100.0%       0.00        interrupts.CPU70.69:PCI-MSI.31981602-edge.i40e-eth0-TxRx-33
    549.22 ±  3%      +7.8%     591.92 ± 22%    +125.8%       1240 ±133%  interrupts.CPU70.CAL:Function_call_interrupts
    774504 ±  3%      -6.7%     722659 ± 18%      +1.2%     783673        interrupts.CPU70.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU70.MCP:Machine_check_polls
    101.89 ± 15%      +2.2%     104.08 ± 23%     -34.5%      66.78 ± 29%  interrupts.CPU70.NMI:Non-maskable_interrupts
    101.89 ± 15%      +2.2%     104.08 ± 23%     -34.5%      66.78 ± 29%  interrupts.CPU70.PMI:Performance_monitoring_interrupts
     69.56 ±257%    +111.2%     146.92 ±190%     +46.2%     101.67 ±236%  interrupts.CPU70.RES:Rescheduling_interrupts
      2.67 ± 86%      +3.1%       2.75 ±111%    +108.3%       5.56 ± 12%  interrupts.CPU70.TLB:TLB_shootdowns
      3.67 ±273%  +94829.5%       3480 ±319%   +5657.6%     211.11 ±282%  interrupts.CPU71.107:PCI-MSI.31981640-edge.i40e-eth0-TxRx-71
      0.11 ±282%     -25.0%       0.08 ±331%      +0.0%       0.11 ±282%  interrupts.CPU71.118:PCI-MSI.31981651-edge.i40e-eth0-TxRx-82
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU71.297:PCI-MSI.67584-edge.ioat-msix
      0.11 ±282%    -100.0%       0.00          +100.0%       0.22 ±187%  interrupts.CPU71.70:PCI-MSI.31981603-edge.i40e-eth0-TxRx-34
    545.67 ±  3%      +0.7%     549.50 ±  5%    +306.3%       2216 ±213%  interrupts.CPU71.CAL:Function_call_interrupts
      0.00       +8.3e+100%       0.08 ±331%    -100.0%       0.00        interrupts.CPU71.IWI:IRQ_work_interrupts
    774504 ±  3%      -6.4%     725220 ± 17%      +1.2%     783668        interrupts.CPU71.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU71.MCP:Machine_check_polls
    739.56 ±160%     +13.7%     840.58 ±213%     -90.2%      72.22 ± 27%  interrupts.CPU71.NMI:Non-maskable_interrupts
    739.56 ±160%     +13.7%     840.58 ±213%     -90.2%      72.22 ± 27%  interrupts.CPU71.PMI:Performance_monitoring_interrupts
     17.56 ±139%    +191.0%      51.08 ±144%     -50.6%       8.67 ± 83%  interrupts.CPU71.RES:Rescheduling_interrupts
      2.33 ± 96%      +7.1%       2.50 ±117%    +147.6%       5.78 ± 19%  interrupts.CPU71.TLB:TLB_shootdowns
     13.22 ±202%     -54.6%       6.00 ±312%  +54798.3%       7258 ±282%  interrupts.CPU72.108:PCI-MSI.31981641-edge.i40e-eth0-TxRx-72
      1332 ±169%     -51.3%     648.83 ± 49%     -56.0%     586.00 ± 10%  interrupts.CPU72.CAL:Function_call_interrupts
      0.11 ±282%    +200.0%       0.33 ±141%    -100.0%       0.00        interrupts.CPU72.IWI:IRQ_work_interrupts
    742073 ± 15%      +3.7%     769160 ±  3%      +5.6%     783675        interrupts.CPU72.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU72.MCP:Machine_check_polls
    129.67 ± 38%      +5.8%     137.25 ± 50%     -49.7%      65.22 ± 33%  interrupts.CPU72.NMI:Non-maskable_interrupts
    129.67 ± 38%      +5.8%     137.25 ± 50%     -49.7%      65.22 ± 33%  interrupts.CPU72.PMI:Performance_monitoring_interrupts
     12.00 ± 88%     +63.9%      19.67 ±101%     -30.6%       8.33 ± 80%  interrupts.CPU72.RES:Rescheduling_interrupts
      3.11 ± 80%     -11.6%       2.75 ±111%     +89.3%       5.89 ± 14%  interrupts.CPU72.TLB:TLB_shootdowns
    188.67 ±282%     -61.3%      73.00 ±331%     -70.6%      55.44 ±251%  interrupts.CPU73.109:PCI-MSI.31981642-edge.i40e-eth0-TxRx-73
    582.22 ± 21%      -4.9%     553.67 ±  4%      -6.7%     543.33 ±  6%  interrupts.CPU73.CAL:Function_call_interrupts
      0.11 ±282%    -100.0%       0.00          -100.0%       0.00        interrupts.CPU73.IWI:IRQ_work_interrupts
    742997 ± 15%      +3.5%     769176 ±  4%      +5.5%     783661        interrupts.CPU73.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU73.MCP:Machine_check_polls
    119.89 ± 45%      -2.5%     116.92 ± 45%     -47.3%      63.22 ± 34%  interrupts.CPU73.NMI:Non-maskable_interrupts
    119.89 ± 45%      -2.5%     116.92 ± 45%     -47.3%      63.22 ± 34%  interrupts.CPU73.PMI:Performance_monitoring_interrupts
     11.33 ± 85%     +27.9%      14.50 ± 68%     -42.2%       6.56 ± 44%  interrupts.CPU73.RES:Rescheduling_interrupts
      3.11 ± 73%      +7.1%       3.33 ± 92%     +85.7%       5.78 ± 21%  interrupts.CPU73.TLB:TLB_shootdowns
    103.56 ±282%     -87.9%      12.50 ±248%     -14.4%      88.67 ±279%  interrupts.CPU74.110:PCI-MSI.31981643-edge.i40e-eth0-TxRx-74
    596.89 ± 25%      +4.2%     621.83 ± 29%    +362.2%       2758 ±217%  interrupts.CPU74.CAL:Function_call_interrupts
    742194 ± 15%      +3.6%     769228 ±  4%      +5.6%     783704        interrupts.CPU74.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU74.MCP:Machine_check_polls
    113.11 ± 39%      +6.1%     120.00 ± 44%     -42.4%      65.11 ± 33%  interrupts.CPU74.NMI:Non-maskable_interrupts
    113.11 ± 39%      +6.1%     120.00 ± 44%     -42.4%      65.11 ± 33%  interrupts.CPU74.PMI:Performance_monitoring_interrupts
     33.22 ±128%    +214.0%     104.33 ±280%     -58.2%      13.89 ±127%  interrupts.CPU74.RES:Rescheduling_interrupts
      3.11 ± 63%      -3.6%       3.00 ±104%    +107.1%       6.44 ± 24%  interrupts.CPU74.TLB:TLB_shootdowns
      0.22 ±187%     +12.5%       0.25 ±238%  +58200.0%     129.56 ±242%  interrupts.CPU75.111:PCI-MSI.31981644-edge.i40e-eth0-TxRx-75
    548.67 ±  5%      -0.2%     547.50 ±  2%    +108.5%       1143 ±133%  interrupts.CPU75.CAL:Function_call_interrupts
      0.11 ±282%    -100.0%       0.00          -100.0%       0.00        interrupts.CPU75.IWI:IRQ_work_interrupts
    742124 ± 15%      +3.6%     769144 ±  4%      +5.6%     783651        interrupts.CPU75.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU75.MCP:Machine_check_polls
    119.00 ± 43%     +62.3%     193.08 ±131%     -44.7%      65.78 ± 26%  interrupts.CPU75.NMI:Non-maskable_interrupts
    119.00 ± 43%     +62.3%     193.08 ±131%     -44.7%      65.78 ± 26%  interrupts.CPU75.PMI:Performance_monitoring_interrupts
      8.78 ± 64%     +49.1%      13.08 ± 70%     +26.6%      11.11 ±153%  interrupts.CPU75.RES:Rescheduling_interrupts
      3.78 ± 51%     -20.6%       3.00 ± 95%     +55.9%       5.89 ± 14%  interrupts.CPU75.TLB:TLB_shootdowns
      8.67 ±278%    +873.1%      84.33 ±331%    +957.7%      91.67 ±282%  interrupts.CPU76.112:PCI-MSI.31981645-edge.i40e-eth0-TxRx-76
    519.78 ± 11%     +48.9%     773.92 ± 98%      +5.3%     547.22        interrupts.CPU76.CAL:Function_call_interrupts
      0.11 ±282%    -100.0%       0.00          -100.0%       0.00        interrupts.CPU76.IWI:IRQ_work_interrupts
    742164 ± 15%      +3.6%     769187 ±  4%      +5.6%     783638        interrupts.CPU76.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU76.MCP:Machine_check_polls
    118.56 ± 42%      -3.8%     114.00 ± 42%     -43.8%      66.67 ± 26%  interrupts.CPU76.NMI:Non-maskable_interrupts
    118.56 ± 42%      -3.8%     114.00 ± 42%     -43.8%      66.67 ± 26%  interrupts.CPU76.PMI:Performance_monitoring_interrupts
      6.22 ± 67%    +304.5%      25.17 ±172%     +46.4%       9.11 ± 69%  interrupts.CPU76.RES:Rescheduling_interrupts
      3.22 ± 65%      -1.7%       3.17 ± 91%     +75.9%       5.67 ± 22%  interrupts.CPU76.TLB:TLB_shootdowns
      3.33 ±282%    +950.0%      35.00 ±307%  +4.7e+05%      15833 ±251%  interrupts.CPU77.113:PCI-MSI.31981646-edge.i40e-eth0-TxRx-77
    815.22 ± 50%      +1.8%     829.83 ±115%     +44.3%       1176 ±121%  interrupts.CPU77.CAL:Function_call_interrupts
    742115 ± 15%      +3.6%     769173 ±  4%      +5.6%     783644        interrupts.CPU77.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU77.MCP:Machine_check_polls
    108.67 ± 40%     +11.2%     120.83 ± 45%     -29.9%      76.22 ± 13%  interrupts.CPU77.NMI:Non-maskable_interrupts
    108.67 ± 40%     +11.2%     120.83 ± 45%     -29.9%      76.22 ± 13%  interrupts.CPU77.PMI:Performance_monitoring_interrupts
     31.22 ±138%     -75.7%       7.58 ± 89%     -19.9%      25.00 ±160%  interrupts.CPU77.RES:Rescheduling_interrupts
      3.89 ± 73%      -7.9%       3.58 ±106%     +60.0%       6.22 ± 18%  interrupts.CPU77.TLB:TLB_shootdowns
      0.00       +3.8e+102%       3.75 ±331% +1.4e+102%       1.44 ±282%  interrupts.CPU78.114:PCI-MSI.31981647-edge.i40e-eth0-TxRx-78
    523.00 ± 12%      +3.3%     540.42            +9.4%     572.33 ± 14%  interrupts.CPU78.CAL:Function_call_interrupts
      0.22 ±187%     -62.5%       0.08 ±331%    -100.0%       0.00        interrupts.CPU78.IWI:IRQ_work_interrupts
    742116 ± 15%      +3.7%     769233 ±  3%      +5.6%     783672        interrupts.CPU78.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU78.MCP:Machine_check_polls
    112.89 ± 54%      +7.6%     121.42 ± 43%     -36.5%      71.67 ± 19%  interrupts.CPU78.NMI:Non-maskable_interrupts
    112.89 ± 54%      +7.6%     121.42 ± 43%     -36.5%      71.67 ± 19%  interrupts.CPU78.PMI:Performance_monitoring_interrupts
      7.56 ± 73%      +3.7%       7.83 ± 92%     -16.2%       6.33 ± 55%  interrupts.CPU78.RES:Rescheduling_interrupts
      3.89 ± 72%     -16.4%       3.25 ± 98%     +57.1%       6.11 ± 17%  interrupts.CPU78.TLB:TLB_shootdowns
    573.00 ±161%     -71.8%     161.50 ±303%    -100.0%       0.00        interrupts.CPU79.115:PCI-MSI.31981648-edge.i40e-eth0-TxRx-79
    541.67 ±  2%      -0.2%     540.42           +72.3%     933.44 ±108%  interrupts.CPU79.CAL:Function_call_interrupts
      0.11 ±282%    -100.0%       0.00          -100.0%       0.00        interrupts.CPU79.IWI:IRQ_work_interrupts
    742169 ± 15%      +3.6%     769208 ±  3%      +5.6%     783746        interrupts.CPU79.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU79.MCP:Machine_check_polls
    116.56 ± 41%      +6.2%     123.75 ± 35%     -46.2%      62.67 ± 29%  interrupts.CPU79.NMI:Non-maskable_interrupts
    116.56 ± 41%      +6.2%     123.75 ± 35%     -46.2%      62.67 ± 29%  interrupts.CPU79.PMI:Performance_monitoring_interrupts
      6.78 ± 74%     +15.6%       7.83 ± 41%     +42.6%       9.67 ±106%  interrupts.CPU79.RES:Rescheduling_interrupts
      3.89 ± 47%      -7.9%       3.58 ± 99%     +57.1%       6.11 ± 24%  interrupts.CPU79.TLB:TLB_shootdowns
      0.00       +8.3e+100%       0.08 ±331%    -100.0%       0.00        interrupts.CPU8.127:PCI-MSI.31981660-edge.i40e-eth0-TxRx-91
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.CPU8.315:PCI-MSI.376832-edge.ahci[0000:00:17.0]
      1.89 ±282%     -91.2%       0.17 ±223%   +5517.6%     106.11 ±255%  interrupts.CPU8.44:PCI-MSI.31981577-edge.i40e-eth0-TxRx-8
      0.00       +8.3e+100%       0.08 ±331%    -100.0%       0.00        interrupts.CPU8.79:PCI-MSI.31981612-edge.i40e-eth0-TxRx-43
    624.56 ± 23%    +413.6%       3207 ±238%   +1183.2%       8014 ±254%  interrupts.CPU8.CAL:Function_call_interrupts
    774520 ±  3%      -6.7%     722476 ± 18%      +1.2%     783724        interrupts.CPU8.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU8.MCP:Machine_check_polls
     85.22 ± 22%     +16.0%      98.83 ± 24%      -8.3%      78.11 ±  9%  interrupts.CPU8.NMI:Non-maskable_interrupts
     85.22 ± 22%     +16.0%      98.83 ± 24%      -8.3%      78.11 ±  9%  interrupts.CPU8.PMI:Performance_monitoring_interrupts
     15.89 ± 39%     -39.7%       9.58 ± 90%    +269.9%      58.78 ±209%  interrupts.CPU8.RES:Rescheduling_interrupts
      1.33 ±106%     +50.0%       2.00 ±139%    +225.0%       4.33 ± 21%  interrupts.CPU8.TLB:TLB_shootdowns
     34.89 ±281%      -1.6%      34.33 ±328%    +297.1%     138.56 ±272%  interrupts.CPU80.116:PCI-MSI.31981649-edge.i40e-eth0-TxRx-80
    655.22 ± 51%     -15.2%     555.50 ±  7%     -13.5%     566.89 ± 10%  interrupts.CPU80.CAL:Function_call_interrupts
    742107 ± 15%      +3.6%     769179 ±  3%      +5.6%     783643        interrupts.CPU80.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU80.MCP:Machine_check_polls
    115.44 ± 37%      +2.7%     118.58 ± 38%     -36.0%      73.89 ± 14%  interrupts.CPU80.NMI:Non-maskable_interrupts
    115.44 ± 37%      +2.7%     118.58 ± 38%     -36.0%      73.89 ± 14%  interrupts.CPU80.PMI:Performance_monitoring_interrupts
     12.11 ±115%      +9.4%      13.25 ±140%     -20.2%       9.67 ± 57%  interrupts.CPU80.RES:Rescheduling_interrupts
      4.11 ± 50%     -14.9%       3.50 ± 81%     +43.2%       5.89 ± 29%  interrupts.CPU80.TLB:TLB_shootdowns
      0.22 ±187%   +1212.5%       2.92 ±243%  +53900.0%     120.00 ±262%  interrupts.CPU81.117:PCI-MSI.31981650-edge.i40e-eth0-TxRx-81
    537.22            +1.5%     545.33 ±  3%     +56.3%     839.44 ± 88%  interrupts.CPU81.CAL:Function_call_interrupts
    742230 ± 15%      +3.6%     769205 ±  3%      +5.6%     783634        interrupts.CPU81.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU81.MCP:Machine_check_polls
    114.44 ± 39%    +205.0%     349.00 ±223%     -30.0%      80.11 ± 13%  interrupts.CPU81.NMI:Non-maskable_interrupts
    114.44 ± 39%    +205.0%     349.00 ±223%     -30.0%      80.11 ± 13%  interrupts.CPU81.PMI:Performance_monitoring_interrupts
      8.78 ± 65%     +52.8%      13.42 ±121%     +54.4%      13.56 ± 90%  interrupts.CPU81.RES:Rescheduling_interrupts
      3.67 ± 51%     -13.6%       3.17 ± 99%     +48.5%       5.44 ± 17%  interrupts.CPU81.TLB:TLB_shootdowns
      0.11 ±282%   +2225.0%       2.58 ±320%    +1e+05%     111.22 ±282%  interrupts.CPU82.118:PCI-MSI.31981651-edge.i40e-eth0-TxRx-82
    539.00 ±  2%      -1.4%     531.58 ±  6%    +125.6%       1215 ±154%  interrupts.CPU82.CAL:Function_call_interrupts
      0.00       +8.3e+100%       0.08 ±331%    -100.0%       0.00        interrupts.CPU82.IWI:IRQ_work_interrupts
    742047 ± 15%      +3.7%     769234 ±  3%      +5.6%     783642        interrupts.CPU82.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU82.MCP:Machine_check_polls
    118.78 ± 36%      -5.5%     112.25 ± 47%     -36.2%      75.78 ± 12%  interrupts.CPU82.NMI:Non-maskable_interrupts
    118.78 ± 36%      -5.5%     112.25 ± 47%     -36.2%      75.78 ± 12%  interrupts.CPU82.PMI:Performance_monitoring_interrupts
      8.56 ± 50%     -22.1%       6.67 ± 52%      +0.0%       8.56 ± 78%  interrupts.CPU82.RES:Rescheduling_interrupts
      3.56 ± 59%     -10.9%       3.17 ± 91%     +71.9%       6.11 ± 14%  interrupts.CPU82.TLB:TLB_shootdowns
    144.33 ±201%     -83.1%      24.33 ±324%     -38.0%      89.56 ±202%  interrupts.CPU83.119:PCI-MSI.31981652-edge.i40e-eth0-TxRx-83
    563.44 ± 19%      -4.0%     540.92           +18.6%     668.44 ± 48%  interrupts.CPU83.CAL:Function_call_interrupts
      0.11 ±282%    +275.0%       0.42 ±153%    -100.0%       0.00        interrupts.CPU83.IWI:IRQ_work_interrupts
    742088 ± 15%      +3.7%     769199 ±  4%      +5.6%     783671        interrupts.CPU83.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU83.MCP:Machine_check_polls
    122.67 ± 43%     +13.0%     138.58 ± 48%     -45.0%      67.44 ± 25%  interrupts.CPU83.NMI:Non-maskable_interrupts
    122.67 ± 43%     +13.0%     138.58 ± 48%     -45.0%      67.44 ± 25%  interrupts.CPU83.PMI:Performance_monitoring_interrupts
     10.00 ± 73%     -29.2%       7.08 ± 96%     +36.7%      13.67 ±106%  interrupts.CPU83.RES:Rescheduling_interrupts
      4.11 ± 63%     -18.9%       3.33 ± 97%     +54.1%       6.33 ± 14%  interrupts.CPU83.TLB:TLB_shootdowns
      0.11 ±282%    +200.0%       0.33 ±141%  +31300.0%      34.89 ±186%  interrupts.CPU84.120:PCI-MSI.31981653-edge.i40e-eth0-TxRx-84
    683.22 ± 40%     -20.7%     542.08           -15.0%     580.89 ±  8%  interrupts.CPU84.CAL:Function_call_interrupts
      0.22 ±187%    -100.0%       0.00            +0.0%       0.22 ±282%  interrupts.CPU84.IWI:IRQ_work_interrupts
    742084 ± 15%      +3.7%     769249 ±  4%      +5.6%     783664        interrupts.CPU84.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU84.MCP:Machine_check_polls
    129.67 ± 40%      -3.1%     125.67 ± 35%    +474.2%     744.56 ±256%  interrupts.CPU84.NMI:Non-maskable_interrupts
    129.67 ± 40%      -3.1%     125.67 ± 35%    +474.2%     744.56 ±256%  interrupts.CPU84.PMI:Performance_monitoring_interrupts
     15.00 ±145%     -30.6%      10.42 ± 99%     +88.1%      28.22 ±191%  interrupts.CPU84.RES:Rescheduling_interrupts
      3.89 ± 57%      -5.7%       3.67 ± 77%     +40.0%       5.44 ± 34%  interrupts.CPU84.TLB:TLB_shootdowns
     34.78 ±187%    +455.0%     193.00 ±218%     -99.7%       0.11 ±282%  interrupts.CPU85.121:PCI-MSI.31981654-edge.i40e-eth0-TxRx-85
    534.67 ± 15%     +12.7%     602.33 ± 26%     +26.7%     677.33 ± 45%  interrupts.CPU85.CAL:Function_call_interrupts
      0.00          -100.0%       0.00       +1.1e+101%       0.11 ±282%  interrupts.CPU85.IWI:IRQ_work_interrupts
    742148 ± 15%      +3.6%     769165 ±  4%      +5.6%     783771        interrupts.CPU85.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU85.MCP:Machine_check_polls
    119.00 ± 40%      +0.3%     119.33 ± 42%     -35.9%      76.22 ± 37%  interrupts.CPU85.NMI:Non-maskable_interrupts
    119.00 ± 40%      +0.3%     119.33 ± 42%     -35.9%      76.22 ± 37%  interrupts.CPU85.PMI:Performance_monitoring_interrupts
      6.44 ± 67%    +105.6%      13.25 ±161%     +37.9%       8.89 ± 78%  interrupts.CPU85.RES:Rescheduling_interrupts
      3.78 ± 49%      -0.7%       3.75 ± 77%     +85.3%       7.00 ± 19%  interrupts.CPU85.TLB:TLB_shootdowns
      0.00       +1.8e+104%     176.00 ±244%    -100.0%       0.00        interrupts.CPU86.122:PCI-MSI.31981655-edge.i40e-eth0-TxRx-86
    702.67 ± 48%     +20.9%     849.50 ±119%     +29.4%     909.00 ± 81%  interrupts.CPU86.CAL:Function_call_interrupts
      0.11 ±282%    -100.0%       0.00          -100.0%       0.00        interrupts.CPU86.IWI:IRQ_work_interrupts
    742225 ± 15%      +3.6%     769189 ±  4%      +5.6%     783591        interrupts.CPU86.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU86.MCP:Machine_check_polls
    121.33 ± 41%      -2.7%     118.00 ± 44%     -42.9%      69.22 ± 29%  interrupts.CPU86.NMI:Non-maskable_interrupts
    121.33 ± 41%      -2.7%     118.00 ± 44%     -42.9%      69.22 ± 29%  interrupts.CPU86.PMI:Performance_monitoring_interrupts
     16.56 ±125%     -47.1%       8.75 ± 73%     -46.3%       8.89 ± 62%  interrupts.CPU86.RES:Rescheduling_interrupts
      3.78 ± 49%      -0.7%       3.75 ± 78%     +70.6%       6.44 ± 19%  interrupts.CPU86.TLB:TLB_shootdowns
     38.44 ±282%     -93.5%       2.50 ±331%     -80.6%       7.44 ±278%  interrupts.CPU87.123:PCI-MSI.31981656-edge.i40e-eth0-TxRx-87
    554.44 ±  3%      +0.0%     554.58 ±  3%      -4.2%     530.89 ± 11%  interrupts.CPU87.CAL:Function_call_interrupts
    742363 ± 15%      +3.6%     769240 ±  3%      +5.6%     783735        interrupts.CPU87.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU87.MCP:Machine_check_polls
    119.22 ± 41%      -1.6%     117.33 ± 44%     -39.1%      72.56 ± 21%  interrupts.CPU87.NMI:Non-maskable_interrupts
    119.22 ± 41%      -1.6%     117.33 ± 44%     -39.1%      72.56 ± 21%  interrupts.CPU87.PMI:Performance_monitoring_interrupts
     13.33 ± 68%      -4.4%      12.75 ±105%     -45.0%       7.33 ± 51%  interrupts.CPU87.RES:Rescheduling_interrupts
      3.78 ± 44%      +5.9%       4.00 ± 70%     +79.4%       6.78 ± 25%  interrupts.CPU87.TLB:TLB_shootdowns
    564.11 ±282%     -80.7%     108.83 ±299%     -90.7%      52.44 ±282%  interrupts.CPU88.124:PCI-MSI.31981657-edge.i40e-eth0-TxRx-88
    911.56 ±114%      +0.0%     912.00 ±102%     -23.9%     693.67 ± 51%  interrupts.CPU88.CAL:Function_call_interrupts
      0.00       +8.3e+100%       0.08 ±331%    -100.0%       0.00        interrupts.CPU88.IWI:IRQ_work_interrupts
    742075 ± 15%      +3.7%     769202 ±  3%      +5.6%     783720        interrupts.CPU88.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU88.MCP:Machine_check_polls
    117.11 ± 39%      +4.7%     122.58 ± 40%     -42.2%      67.67 ± 30%  interrupts.CPU88.NMI:Non-maskable_interrupts
    117.11 ± 39%      +4.7%     122.58 ± 40%     -42.2%      67.67 ± 30%  interrupts.CPU88.PMI:Performance_monitoring_interrupts
      7.89 ± 71%     +82.7%      14.42 ± 76%     -16.9%       6.56 ± 62%  interrupts.CPU88.RES:Rescheduling_interrupts
      4.00 ± 47%      -8.3%       3.67 ± 72%     +58.3%       6.33 ± 27%  interrupts.CPU88.TLB:TLB_shootdowns
     11.44 ±240%    +333.3%      49.58 ±331%     -97.1%       0.33 ±200%  interrupts.CPU89.125:PCI-MSI.31981658-edge.i40e-eth0-TxRx-89
    541.78 ±  2%      +2.0%     552.75 ±  3%      +1.0%     547.00        interrupts.CPU89.CAL:Function_call_interrupts
      0.11 ±282%     -25.0%       0.08 ±331%      +0.0%       0.11 ±282%  interrupts.CPU89.IWI:IRQ_work_interrupts
    742048 ± 15%      +3.7%     769141 ±  3%      +5.6%     783643        interrupts.CPU89.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU89.MCP:Machine_check_polls
    125.11 ± 42%      -2.0%     122.67 ± 47%     -40.3%      74.67 ± 38%  interrupts.CPU89.NMI:Non-maskable_interrupts
    125.11 ± 42%      -2.0%     122.67 ± 47%     -40.3%      74.67 ± 38%  interrupts.CPU89.PMI:Performance_monitoring_interrupts
     18.67 ±159%     -42.4%      10.75 ± 84%     -73.2%       5.00 ± 73%  interrupts.CPU89.RES:Rescheduling_interrupts
      3.56 ± 40%     +19.5%       4.25 ± 63%    +100.0%       7.11 ± 12%  interrupts.CPU89.TLB:TLB_shootdowns
      0.11 ±282%     +50.0%       0.17 ±223%    +100.0%       0.22 ±187%  interrupts.CPU9.128:PCI-MSI.31981661-edge.i40e-eth0-TxRx-92
    143.33 ±233%    +418.4%     743.08 ±221%    -100.0%       0.00        interrupts.CPU9.45:PCI-MSI.31981578-edge.i40e-eth0-TxRx-9
      0.22 ±187%     +50.0%       0.33 ±141%      +0.0%       0.22 ±187%  interrupts.CPU9.80:PCI-MSI.31981613-edge.i40e-eth0-TxRx-44
    729.78 ± 67%     +43.1%       1044 ±113%     +42.2%       1037 ±131%  interrupts.CPU9.CAL:Function_call_interrupts
    774586 ±  3%      -6.7%     722508 ± 18%      +1.2%     783681        interrupts.CPU9.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU9.MCP:Machine_check_polls
    104.00 ± 18%     -13.0%      90.50 ± 30%    +323.3%     440.22 ±231%  interrupts.CPU9.NMI:Non-maskable_interrupts
    104.00 ± 18%     -13.0%      90.50 ± 30%    +323.3%     440.22 ±231%  interrupts.CPU9.PMI:Performance_monitoring_interrupts
     12.44 ± 64%   +1876.8%     246.00 ±306%     +27.7%      15.89 ± 75%  interrupts.CPU9.RES:Rescheduling_interrupts
      1.56 ±109%     +23.2%       1.92 ±138%    +185.7%       4.44 ± 28%  interrupts.CPU9.TLB:TLB_shootdowns
    538.67 ±282%   +1919.4%      10877 ±331%    -100.0%       0.00        interrupts.CPU90.126:PCI-MSI.31981659-edge.i40e-eth0-TxRx-90
    542.56            +1.3%     549.50 ±  3%      +8.0%     585.89 ± 18%  interrupts.CPU90.CAL:Function_call_interrupts
      0.11 ±282%     -25.0%       0.08 ±331%    -100.0%       0.00        interrupts.CPU90.IWI:IRQ_work_interrupts
    742180 ± 15%      +3.6%     769219 ±  4%      +5.6%     783638        interrupts.CPU90.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU90.MCP:Machine_check_polls
    126.44 ± 39%      +0.2%     126.67 ± 39%     -47.6%      66.22 ± 27%  interrupts.CPU90.NMI:Non-maskable_interrupts
    126.44 ± 39%      +0.2%     126.67 ± 39%     -47.6%      66.22 ± 27%  interrupts.CPU90.PMI:Performance_monitoring_interrupts
     10.89 ± 93%     +24.7%      13.58 ±119%      -2.0%      10.67 ± 78%  interrupts.CPU90.RES:Rescheduling_interrupts
      4.22 ± 33%      -3.3%       4.08 ± 66%     +50.0%       6.33 ± 28%  interrupts.CPU90.TLB:TLB_shootdowns
     13770 ±282%    -100.0%       0.00          -100.0%       0.00        interrupts.CPU91.127:PCI-MSI.31981660-edge.i40e-eth0-TxRx-91
    907.89 ±114%     -36.3%     578.58 ± 14%     -32.9%     609.33 ± 25%  interrupts.CPU91.CAL:Function_call_interrupts
      0.00          -100.0%       0.00       +1.1e+101%       0.11 ±282%  interrupts.CPU91.IWI:IRQ_work_interrupts
    742276 ± 15%      +3.6%     769195 ±  4%      +5.6%     783673        interrupts.CPU91.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU91.MCP:Machine_check_polls
    113.56 ± 40%      +5.2%     119.42 ± 41%     -31.3%      78.00 ± 29%  interrupts.CPU91.NMI:Non-maskable_interrupts
    113.56 ± 40%      +5.2%     119.42 ± 41%     -31.3%      78.00 ± 29%  interrupts.CPU91.PMI:Performance_monitoring_interrupts
     17.11 ±108%     -56.7%       7.42 ± 62%     -50.6%       8.44 ± 75%  interrupts.CPU91.RES:Rescheduling_interrupts
      4.11 ± 31%      +9.5%       4.50 ± 65%     +64.9%       6.78 ± 11%  interrupts.CPU91.TLB:TLB_shootdowns
      0.78 ±240%     -78.6%       0.17 ±223%   +5014.3%      39.78 ±280%  interrupts.CPU92.128:PCI-MSI.31981661-edge.i40e-eth0-TxRx-92
    558.33 ±  6%     +11.8%     624.33 ± 42%     +26.0%     703.22 ± 60%  interrupts.CPU92.CAL:Function_call_interrupts
      0.00          -100.0%       0.00       +2.2e+101%       0.22 ±282%  interrupts.CPU92.IWI:IRQ_work_interrupts
    742166 ± 15%      +3.6%     769146 ±  4%      +5.6%     783625        interrupts.CPU92.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU92.MCP:Machine_check_polls
    117.78 ± 38%      -1.2%     116.33 ± 45%    +534.5%     747.33 ±255%  interrupts.CPU92.NMI:Non-maskable_interrupts
    117.78 ± 38%      -1.2%     116.33 ± 45%    +534.5%     747.33 ±255%  interrupts.CPU92.PMI:Performance_monitoring_interrupts
      8.44 ± 83%     -23.0%       6.50 ± 61%     +39.5%      11.78 ± 98%  interrupts.CPU92.RES:Rescheduling_interrupts
      4.89 ± 28%      -8.0%       4.50 ± 51%     +29.5%       6.33 ± 36%  interrupts.CPU92.TLB:TLB_shootdowns
      2715 ±206%     -83.3%     454.00 ±327%    -100.0%       0.00        interrupts.CPU93.129:PCI-MSI.31981662-edge.i40e-eth0-TxRx-93
    725.00 ± 64%     -25.5%     540.00 ± 13%     -24.3%     549.11        interrupts.CPU93.CAL:Function_call_interrupts
    742161 ± 15%      +3.6%     769136 ±  4%      +5.6%     783658        interrupts.CPU93.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU93.MCP:Machine_check_polls
    117.44 ± 40%      -2.9%     114.00 ± 44%     -39.5%      71.11 ± 22%  interrupts.CPU93.NMI:Non-maskable_interrupts
    117.44 ± 40%      -2.9%     114.00 ± 44%     -39.5%      71.11 ± 22%  interrupts.CPU93.PMI:Performance_monitoring_interrupts
     15.56 ±112%     -45.9%       8.42 ± 69%     -62.1%       5.89 ± 85%  interrupts.CPU93.RES:Rescheduling_interrupts
      4.67 ± 24%     +10.7%       5.17 ± 45%     +69.0%       7.89 ± 17%  interrupts.CPU93.TLB:TLB_shootdowns
    595.11 ± 12%      +3.5%     616.17 ± 39%      -7.0%     553.33        interrupts.CPU94.CAL:Function_call_interrupts
    742121 ± 15%      +3.6%     769155 ±  3%      +5.6%     783729        interrupts.CPU94.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU94.MCP:Machine_check_polls
    117.00 ± 39%      -0.2%     116.75 ± 44%     -39.0%      71.33 ± 23%  interrupts.CPU94.NMI:Non-maskable_interrupts
    117.00 ± 39%      -0.2%     116.75 ± 44%     -39.0%      71.33 ± 23%  interrupts.CPU94.PMI:Performance_monitoring_interrupts
      8.78 ± 35%     +26.3%      11.08 ±101%     -68.4%       2.78 ± 62%  interrupts.CPU94.RES:Rescheduling_interrupts
      4.78 ± 19%      +9.9%       5.25 ± 40%     +58.1%       7.56 ± 11%  interrupts.CPU94.TLB:TLB_shootdowns
    519.33 ±  5%      +9.3%     567.42 ± 16%      +1.0%     524.56 ± 13%  interrupts.CPU95.CAL:Function_call_interrupts
      0.22 ±187%     -25.0%       0.17 ±223%    -100.0%       0.00        interrupts.CPU95.IWI:IRQ_work_interrupts
    742181 ± 15%      +3.6%     769213 ±  3%      +5.6%     783671        interrupts.CPU95.LOC:Local_timer_interrupts
      1.00            +0.0%       1.00            +0.0%       1.00        interrupts.CPU95.MCP:Machine_check_polls
    133.33 ± 43%      -6.1%     125.25 ± 52%     -45.4%      72.78 ± 31%  interrupts.CPU95.NMI:Non-maskable_interrupts
    133.33 ± 43%      -6.1%     125.25 ± 52%     -45.4%      72.78 ± 31%  interrupts.CPU95.PMI:Performance_monitoring_interrupts
     12.33 ± 80%     -16.9%      10.25 ± 64%     -20.7%       9.78 ±106%  interrupts.CPU95.RES:Rescheduling_interrupts
      0.11 ±282%    +275.0%       0.42 ±153%    +200.0%       0.33 ±200%  interrupts.CPU95.TLB:TLB_shootdowns
      4.56 ± 61%     +15.2%       5.25 ± 47%     -48.8%       2.33 ± 28%  interrupts.IWI:IRQ_work_interrupts
  72801215 ±  9%      -1.6%   71607644 ± 11%      +3.3%   75232525        interrupts.LOC:Local_timer_interrupts
     96.00            +0.0%      96.00            +0.0%      96.00        interrupts.MCP:Machine_check_polls
     17999 ± 16%      +1.6%      18282 ± 19%     -18.4%      14696 ± 10%  interrupts.NMI:Non-maskable_interrupts
     17999 ± 16%      +1.6%      18282 ± 19%     -18.4%      14696 ± 10%  interrupts.PMI:Performance_monitoring_interrupts
      2608 ± 45%     +22.0%       3182 ± 82%     -33.7%       1730 ± 13%  interrupts.RES:Rescheduling_interrupts
      0.00          -100.0%       0.00          -100.0%       0.00        interrupts.RTR:APIC_ICR_read_retries
    243.78 ± 68%      +7.8%     262.75 ±106%    +111.3%     515.22 ± 14%  interrupts.TLB:TLB_shootdowns
Thanks,
Amir.

Re: [fsnotify] 4c40d6efc8: unixbench.score -3.3% regression

From: Amir Goldstein <amir73il@gmail.com>
Date: 2021-08-03 16:19:17

quoted
Oliver,

Would it be possible to request a re-test with the branch:
https://github.com/amir73il/linux fsnotify-perf

The patch at the tip of that branch is the one this regression report
has blamed.

My expectation is that the patch at fsnotify-perf^ ("fsnotify: optimize the
case of no marks of any type") will improve performance of the test case
compared to baseline (v5.14-rc3) and that the patch at the tip of fsnotify-perf
would not regress performance.
we tested this branch and the results meet your expectation.

fsnotify-perf^ improves performance comparing to v5.14-rc3. tip is a little worse
than its parent (-3.3%), but still better than v5.14-rc3.

below is detail data.


=========================================================================================
compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase/ucode:
  gcc-9/performance/x86_64-rhel-8.3/1/debian-10.4-x86_64-20200603.cgz/300s/lkp-csl-2sp4/pipe/unixbench/0x4003006

commit:
  v5.14-rc3
  23050d041 ("fsnotify: optimize the case of no marks of any type")
  7446ba772 ("fsnotify: pass arguments of fsnotify() in struct fsnotify_event_info")

       v5.14-rc3 23050d0419441a02185e4ed5170 7446ba772ae107ab937cd04e880
---------------- --------------------------- ---------------------------
         %stddev     %change         %stddev     %change         %stddev
             \          |                \          |                \
      1562            +8.0%       1688            +4.5%       1633        unixbench.score
Hi Oliver,

Thanks a lot for testing!

I don't know what to make of the (-3.3%) degradation because I was expecting
that fsnotify-perf^ would optimize out the calls to fsnotify() and fsnotify-perf
only changes code from fsnotify() and below.

But I guess it doesn't matter much as Gabriel said, its a cleanup patch and
we can drop it.

But now that I have this report I can post the fsnotify-perf^ patches :-)

Thanks,
Amir.
Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help