Add couple of new macros to represent onchip L2 and onchip L3 accesses.
Patch also adds the decoding strings in the mem_lvlnum data structure.
Signed-off-by: Kajol Jain <redacted>
---
tools/include/uapi/linux/perf_event.h | 4 +++-
tools/perf/util/mem-events.c | 2 ++
2 files changed, 5 insertions(+), 1 deletion(-)
From: Michael Ellerman <mpe@ellerman.id.au> Date: 2021-09-08 07:18:32
Kajol Jain [off-list ref] writes:
Add couple of new macros to represent onchip L2 and onchip L3 accesses.
It would be "on chip". But I think this needs much more explanation,
this is a generic header so these definitions need to make sense, and
have an understood meaning, across all architectures.
I think most people are going to read "on chip" as differentiating
between an L2/L3 that is "on chip" vs "off chip".
But the case you're trying to express is "another core's L2/L3 on the
same chip as the CPU", vs "the current CPU's L2/L3".
@@ -1265,7 +1265,9 @@ union perf_mem_data_src {#define PERF_MEM_LVLNUM_L2 0x02 /* L2 */#define PERF_MEM_LVLNUM_L3 0x03 /* L3 */#define PERF_MEM_LVLNUM_L4 0x04 /* L4 */-/* 5-0xa available */+#define PERF_MEM_LVLNUM_OC_L2 0x05 /* On Chip L2 */+#define PERF_MEM_LVLNUM_OC_L3 0x06 /* On Chip L3 */
The obvious use for 5 is for "L5" and so on.
I'm not sure adding new levels is the best idea, because these don't fit
neatly into the hierarchy, they are off to the side.
I wonder if we should use the remote field.
ie. for another core's L2 we set:
mem_lvl = PERF_MEM_LVL_L2
mem_remote = 1
Which would mean "remote L2", but not remote enough to be
lvl = PERF_MEM_LVL_REM_CCE1.
It would be printed by the existing tools/perf code as "Remote L2", vs
"Remote cache (1 hop)", which seems OK.
ie. we'd be able to express:
Current core's L2: LVL_L2
Other core's L2: LVL_L2 | REMOTE
Other chip's L2: LVL_REM_CCE1 | REMOTE
And similarly for L3.
I think that makes sense? Unless people think remote should be reserved
to mean on another chip, though we already have REM_CCE1 for that.
cheers
@@ -1265,7 +1265,9 @@ union perf_mem_data_src {#define PERF_MEM_LVLNUM_L2 0x02 /* L2 */#define PERF_MEM_LVLNUM_L3 0x03 /* L3 */#define PERF_MEM_LVLNUM_L4 0x04 /* L4 */-/* 5-0xa available */+#define PERF_MEM_LVLNUM_OC_L2 0x05 /* On Chip L2 */+#define PERF_MEM_LVLNUM_OC_L3 0x06 /* On Chip L3 */
The obvious use for 5 is for "L5" and so on.
I'm not sure adding new levels is the best idea, because these don't fit
neatly into the hierarchy, they are off to the side.
I wonder if we should use the remote field.
ie. for another core's L2 we set:
mem_lvl = PERF_MEM_LVL_L2
mem_remote = 1
This mixes APIs (see below), IIUC the correct usage would be something
like: lvl_num=L2 remote=1
Which would mean "remote L2", but not remote enough to be
lvl = PERF_MEM_LVL_REM_CCE1.
It would be printed by the existing tools/perf code as "Remote L2", vs
"Remote cache (1 hop)", which seems OK.
ie. we'd be able to express:
Current core's L2: LVL_L2
Other core's L2: LVL_L2 | REMOTE
Other chip's L2: LVL_REM_CCE1 | REMOTE
And similarly for L3.
I think that makes sense? Unless people think remote should be reserved
to mean on another chip, though we already have REM_CCE1 for that.
IIRC the PERF_MEM_LVL_* namespace is somewhat depricated in favour of
the newer composite PERF_MEM_{LVLNUM_,REMOTE_,SNOOPX_} fields. Of
course, ABIs being what they are, we get to support both :/ But I'm not
sure mixing them is a great idea.
Also, clearly this could use a comment...
The 'new' composite doesnt have a hops field because the hardware that
nessecitated that change doesn't report it, but we could easily add a
field there.
Suppose we add, mem_hops:3 (would 6 hops be too small?) and the
corresponding PERF_MEM_HOPS_{NA, 0..6}
Then I suppose you can encode things like:
L2 - local L2
L2 | REMOTE - remote L2 at an unspecified distance (NA)
L2 | REMOTE | HOPS_0 - remote L2 on the same node
L2 | REMOTE | HOPS_1 - remote L2 on a node 1 removed
Would that work?
@@ -1265,7 +1265,9 @@ union perf_mem_data_src {#define PERF_MEM_LVLNUM_L2 0x02 /* L2 */#define PERF_MEM_LVLNUM_L3 0x03 /* L3 */#define PERF_MEM_LVLNUM_L4 0x04 /* L4 */-/* 5-0xa available */+#define PERF_MEM_LVLNUM_OC_L2 0x05 /* On Chip L2 */+#define PERF_MEM_LVLNUM_OC_L3 0x06 /* On Chip L3 */
The obvious use for 5 is for "L5" and so on.
I'm not sure adding new levels is the best idea, because these don't fit
neatly into the hierarchy, they are off to the side.
I wonder if we should use the remote field.
ie. for another core's L2 we set:
mem_lvl = PERF_MEM_LVL_L2
mem_remote = 1
This mixes APIs (see below), IIUC the correct usage would be something
like: lvl_num=L2 remote=1
Aha, I was wondering how lvl and lvl_num were supposed to interact.
quoted
Which would mean "remote L2", but not remote enough to be
lvl = PERF_MEM_LVL_REM_CCE1.
It would be printed by the existing tools/perf code as "Remote L2", vs
"Remote cache (1 hop)", which seems OK.
ie. we'd be able to express:
Current core's L2: LVL_L2
Other core's L2: LVL_L2 | REMOTE
Other chip's L2: LVL_REM_CCE1 | REMOTE
And similarly for L3.
I think that makes sense? Unless people think remote should be reserved
to mean on another chip, though we already have REM_CCE1 for that.
IIRC the PERF_MEM_LVL_* namespace is somewhat depricated in favour of
the newer composite PERF_MEM_{LVLNUM_,REMOTE_,SNOOPX_} fields. Of
course, ABIs being what they are, we get to support both :/ But I'm not
sure mixing them is a great idea.
OK.
Also, clearly this could use a comment...
The 'new' composite doesnt have a hops field because the hardware that
nessecitated that change doesn't report it, but we could easily add a
field there.
Suppose we add, mem_hops:3 (would 6 hops be too small?) and the
corresponding PERF_MEM_HOPS_{NA, 0..6}
It's really 7 if we use remote && hop = 0 to mean the first hop.
If we're wanting to use some of the hop levels to represent
intra-chip/package hops then we could possibly use them all on a really
big system.
eg. you could imagine something like:
L2 | - local L2
L2 | REMOTE | HOPS_0 - L2 of neighbour core
L2 | REMOTE | HOPS_1 - L2 of near core on same chip (same 1/2 of chip)
L2 | REMOTE | HOPS_2 - L2 of far core on same chip (other 1/2 of chip)
L2 | REMOTE | HOPS_3 - L2 of sibling chip in same package
L2 | REMOTE | HOPS_4 - L2 on separate package 1 hop away
L2 | REMOTE | HOPS_5 - L2 on separate package 2 hops away
L2 | REMOTE | HOPS_6 - L2 on separate package 3 hops away
Whether it's useful to represent all those levels I'm not sure, but it's
probably good if we have the ability.
I guess I'm 50/50 on whether that's enough levels, or whether we want
another bit to allow for future growth.
Then I suppose you can encode things like:
L2 - local L2
L2 | REMOTE - remote L2 at an unspecified distance (NA)
L2 | REMOTE | HOPS_0 - remote L2 on the same node
L2 | REMOTE | HOPS_1 - remote L2 on a node 1 removed
Would that work?
From: Peter Zijlstra <peterz@infradead.org> Date: 2021-09-09 14:46:41
On Thu, Sep 09, 2021 at 10:45:54PM +1000, Michael Ellerman wrote:
quoted
The 'new' composite doesnt have a hops field because the hardware that
nessecitated that change doesn't report it, but we could easily add a
field there.
Suppose we add, mem_hops:3 (would 6 hops be too small?) and the
corresponding PERF_MEM_HOPS_{NA, 0..6}
It's really 7 if we use remote && hop = 0 to mean the first hop.
I don't think we can do that, becaus of backward compat. Currently:
lvl_num=DRAM, remote=1
denites: "Remote DRAM of any distance". Effectively it would have the new
hops field filled with zeros though, so if you then decode with the hops
field added it suddenly becomes:
lvl_num=DRAM, remote=1, hops=0
and reads like: "Remote DRAM of 0 hops" which is quite daft. Therefore 0
really must denote a 'N/A'.
If we're wanting to use some of the hop levels to represent
intra-chip/package hops then we could possibly use them all on a really
big system.
eg. you could imagine something like:
L2 | - local L2
L2 | REMOTE | HOPS_0 - L2 of neighbour core
L2 | REMOTE | HOPS_1 - L2 of near core on same chip (same 1/2 of chip)
L2 | REMOTE | HOPS_2 - L2 of far core on same chip (other 1/2 of chip)
L2 | REMOTE | HOPS_3 - L2 of sibling chip in same package
L2 | REMOTE | HOPS_4 - L2 on separate package 1 hop away
L2 | REMOTE | HOPS_5 - L2 on separate package 2 hops away
L2 | REMOTE | HOPS_6 - L2 on separate package 3 hops away
Whether it's useful to represent all those levels I'm not sure, but it's
probably good if we have the ability.
I'm thinking we ought to keep hops as steps along the NUMA fabric, with
0 hops being the local node. That only gets us:
L2, remote=0, hops=HOPS_0 -- our L2
L2, remote=1, hops=HOPS_0 -- L2 on the local node but not ours
L2, remote=1, hops!=HOPS_0 -- L2 on a remote node
I guess I'm 50/50 on whether that's enough levels, or whether we want
another bit to allow for future growth.
Right, possibly safer to add one extra bit while we can.... I suppose.
From: Michael Ellerman <mpe@ellerman.id.au> Date: 2021-09-14 10:40:48
Peter Zijlstra [off-list ref] writes:
On Thu, Sep 09, 2021 at 10:45:54PM +1000, Michael Ellerman wrote:
quoted
quoted
The 'new' composite doesnt have a hops field because the hardware that
nessecitated that change doesn't report it, but we could easily add a
field there.
Suppose we add, mem_hops:3 (would 6 hops be too small?) and the
corresponding PERF_MEM_HOPS_{NA, 0..6}
It's really 7 if we use remote && hop = 0 to mean the first hop.
I don't think we can do that, becaus of backward compat. Currently:
lvl_num=DRAM, remote=1
denites: "Remote DRAM of any distance". Effectively it would have the new
hops field filled with zeros though, so if you then decode with the hops
field added it suddenly becomes:
lvl_num=DRAM, remote=1, hops=0
and reads like: "Remote DRAM of 0 hops" which is quite daft. Therefore 0
really must denote a 'N/A'.
Ah yeah, duh, it needs to be backward compatible.
quoted
If we're wanting to use some of the hop levels to represent
intra-chip/package hops then we could possibly use them all on a really
big system.
eg. you could imagine something like:
L2 | - local L2
L2 | REMOTE | HOPS_0 - L2 of neighbour core
L2 | REMOTE | HOPS_1 - L2 of near core on same chip (same 1/2 of chip)
L2 | REMOTE | HOPS_2 - L2 of far core on same chip (other 1/2 of chip)
L2 | REMOTE | HOPS_3 - L2 of sibling chip in same package
L2 | REMOTE | HOPS_4 - L2 on separate package 1 hop away
L2 | REMOTE | HOPS_5 - L2 on separate package 2 hops away
L2 | REMOTE | HOPS_6 - L2 on separate package 3 hops away
Whether it's useful to represent all those levels I'm not sure, but it's
probably good if we have the ability.
I'm thinking we ought to keep hops as steps along the NUMA fabric, with
0 hops being the local node. That only gets us:
L2, remote=0, hops=HOPS_0 -- our L2
L2, remote=1, hops=HOPS_0 -- L2 on the local node but not ours
L2, remote=1, hops!=HOPS_0 -- L2 on a remote node
Hmm. I'm not sure about tying it directly to NUMA hops. I worry we're
going to see more and more systems where there's a hierarchy within the
chip/package, in addition to the traditional NUMA hierarchy.
Although then I guess it becomes a question of what exactly is a NUMA
hop, maybe the answer is that on those future systems those
intra-chip/package hops should be represented as NUMA hops.
It's not like we have a hard definition of what a NUMA hop is?
quoted
I guess I'm 50/50 on whether that's enough levels, or whether we want
another bit to allow for future growth.
Right, possibly safer to add one extra bit while we can.... I suppose.
Equally it's not _that_ hard to add another bit later (if there's still
one free), makes the API a little uglier to use, but not the end of the
world.
cheers
From: Peter Zijlstra <peterz@infradead.org> Date: 2021-09-14 11:50:27
On Tue, Sep 14, 2021 at 08:40:38PM +1000, Michael Ellerman wrote:
Peter Zijlstra [off-list ref] writes:
quoted
I'm thinking we ought to keep hops as steps along the NUMA fabric, with
0 hops being the local node. That only gets us:
L2, remote=0, hops=HOPS_0 -- our L2
L2, remote=1, hops=HOPS_0 -- L2 on the local node but not ours
L2, remote=1, hops!=HOPS_0 -- L2 on a remote node
Hmm. I'm not sure about tying it directly to NUMA hops. I worry we're
going to see more and more systems where there's a hierarchy within the
chip/package, in addition to the traditional NUMA hierarchy.
Although then I guess it becomes a question of what exactly is a NUMA
hop, maybe the answer is that on those future systems those
intra-chip/package hops should be represented as NUMA hops.
It's not like we have a hard definition of what a NUMA hop is?
Not really, typically whatever the BIOS/DT/whatever tables tell us. I
think in case of Power you're mostly making things up in software :-)
But yeah, I think we have plenty wriggle room there.
From: Michael Ellerman <mpe@ellerman.id.au> Date: 2021-09-16 10:57:52
Peter Zijlstra [off-list ref] writes:
On Tue, Sep 14, 2021 at 08:40:38PM +1000, Michael Ellerman wrote:
quoted
Peter Zijlstra [off-list ref] writes:
quoted
quoted
I'm thinking we ought to keep hops as steps along the NUMA fabric, with
0 hops being the local node. That only gets us:
L2, remote=0, hops=HOPS_0 -- our L2
L2, remote=1, hops=HOPS_0 -- L2 on the local node but not ours
L2, remote=1, hops!=HOPS_0 -- L2 on a remote node
Hmm. I'm not sure about tying it directly to NUMA hops. I worry we're
going to see more and more systems where there's a hierarchy within the
chip/package, in addition to the traditional NUMA hierarchy.
Although then I guess it becomes a question of what exactly is a NUMA
hop, maybe the answer is that on those future systems those
intra-chip/package hops should be represented as NUMA hops.
It's not like we have a hard definition of what a NUMA hop is?
Not really, typically whatever the BIOS/DT/whatever tables tell us. I
think in case of Power you're mostly making things up in software :-)
Firmware is software so yes :)
But yeah, I think we have plenty wriggle room there.