Re: [PATCH v3 00/17] KVM: x86/pmu: Add support to enable Guest PEBS via DS

On 2021/1/29 10:52, Liuxiangdong (Aven, Cloud Infrastructure Service 
Product Dept.) wrote:

On 2021/1/26 15:08, Xu, Like wrote:

quoted

On 2021/1/25 22:47, Liuxiangdong (Aven, Cloud Infrastructure Service
Product Dept.) wrote:

quoted

Thanks for replying,

On 2021/1/25 10:41, Like Xu wrote:

quoted

+ kvm@vger.kernel.org

Hi Liuxiangdong,

On 2021/1/22 18:02, Liuxiangdong (Aven, Cloud Infrastructure Service
Product Dept.) wrote:

quoted

Hi Like,

Some questions about
https://lore.kernel.org/kvm/20210104131542.495413-1-like.xu@linux.intel.com/ (local) 

<https://lore.kernel.org/kvm/20210104131542.495413-1-like.xu@linux.intel.com/ (local)> 

Thanks for trying the PEBS feature in the guest,
and I assume you have correctly applied the QEMU patches for guest PEBS.

Is there any other patch that needs to be apply? I use qemu 5.2.0.
(download from github on January 14th)

Two qemu patches are attached against qemu tree
(commit 31ee895047bdcf7387e3570cbd2a473c6f744b08)
and then run the guest with "-cpu,pebs=true".

Note, this two patch are just for test and not finalized for qemu upstream.

Yes, we can use pebs in IceLake when qemu patches applied.
Thanks very much!

Thanks for your verification on this earlier version.

quoted

quoted

quoted

quoted

1)Test in IceLake

In the [PATCH v3 10/17] KVM: x86/pmu: Expose CPUIDs feature bits PDCM,
DS, DTES64, we only support Ice Lake with the following x86_model(s):

#define INTEL_FAM6_ICELAKE_X        0x6A
#define INTEL_FAM6_ICELAKE_D        0x6C

you can check the eax output of "cpuid -l 1 -1 -r",
for example "0x000606a4" meets this requirement.

It's INTEL_FAM6_ICELAKE_X

Yes, it's the target hardware.

quoted

cpuid -l 1 -1 -r

CPU:
    0x00000001 0x00: eax=0x000606a6 ebx=0xb4800800 ecx=0x7ffefbf7
edx=0xbfebfbff

quoted

quoted

HOST:

CPU family:                      6

Model:                           106

Model name:                      Intel(R) Xeon(R) Platinum 8378A CPU 
$@ $@

microcode: sig=0x606a6, pf=0x1, revision=0xd000122

As long as you get the latest BIOS from the provider,
you may check 'cat /proc/cpuinfo | grep code | uniq' with the latest one.

OK. I'll do it later.

quoted

quoted

Guest:  linux kernel 5.11.0-rc2

I assume it's the "upstream tag v5.11-rc2" which is fine.

Yes.

quoted

quoted

We can find pebs/intel_pt flag in guest cpuinfo, but there still exists
error when we use perf

Just a note, intel_pt and pebs are two features and we can write
pebs records to intel_pt buffer with extra hardware support.
(by default, pebs records are written to the pebs buffer)

You may check the output of "dmesg | grep PEBS" in the guest
to see if the guest PEBS cpuinfo is exposed and use "perf record
–e cycles:pp" to see if PEBS feature actually  works in the guest.

I apply only pebs patch set to linux kernel 5.11.0-rc2, test perf in
guest and dump stack when return -EOPNOTSUPP

Yes, you may apply the qemu patches and try it again.

quoted

(1)
# perf record -e instructions:pp
Error:
instructions:pp: PMU Hardware doesn't support
sampling/overflow-interrupts. Try 'perf stat'

[  117.793266] Call Trace:
[  117.793270]  dump_stack+0x57/0x6a
[  117.793275]  intel_pmu_setup_lbr_filter+0x137/0x190
[  117.793280]  intel_pmu_hw_config+0x18b/0x320
[  117.793288]  hsw_hw_config+0xe/0xa0
[  117.793290]  x86_pmu_event_init+0x8e/0x210
[  117.793293]  perf_try_init_event+0x40/0x130
[  117.793297]  perf_event_alloc.part.22+0x611/0xde0
[  117.793299]  ? alloc_fd+0xba/0x180
[  117.793302]  __do_sys_perf_event_open+0x1bd/0xd90
[  117.793305]  do_syscall_64+0x33/0x40
[  117.793308]  entry_SYSCALL_64_after_hwframe+0x44/0xa9

Do we need lbr when we use pebs?

No, lbr ane pebs are two features and we enable it separately.

quoted

I tried to apply lbr patch
set(https://lore.kernel.org/kvm/911adb63-ba05-ea93-c038-1c09cff15eda@intel.com/ (local)) 

to kernel and qemu, but there is still other problem.
Error:
The sys_perf_event_open() syscall returned with 22 (Invalid argument) for
event
...

We don't need that patch for PEBS feature.

quoted

(2)
# perf record -e instructions:ppp
Error:
instructions:ppp: PMU Hardware doesn't support
sampling/overflow-interrupts. Try 'perf stat'

[  115.188498] Call Trace:
[  115.188503]  dump_stack+0x57/0x6a
[  115.188509]  x86_pmu_hw_config+0x1eb/0x220
[  115.188515]  intel_pmu_hw_config+0x13/0x320
[  115.188519]  hsw_hw_config+0xe/0xa0
[  115.188521]  x86_pmu_event_init+0x8e/0x210
[  115.188524]  perf_try_init_event+0x40/0x130
[  115.188528]  perf_event_alloc.part.22+0x611/0xde0
[  115.188530]  ? alloc_fd+0xba/0x180
[  115.188534]  __do_sys_perf_event_open+0x1bd/0xd90
[  115.188538]  do_syscall_64+0x33/0x40
[  115.188541]  entry_SYSCALL_64_after_hwframe+0x44/0xa9

This is beacuse x86_pmu.intel_cap.pebs_format is always 0 in
x86_pmu_max_precise().

We rdmsr MSR_IA32_PERF_CAPABILITIES(0x00000345)  from HOST, it's f4c5.
 From guest, it's 2000

quoted

quoted

# perf record –e cycles:pp

Error:

cycles:pp: PMU Hardware doesn’t support sampling/overflow-interrupts.
Try ‘perf stat’

Could you give some advice?

If you have more specific comments or any concerns, just let me know.

quoted

2)Test in Skylake

HOST:

CPU family:                      6

Model:                           85

Model name:                      Intel(R) Xeon(R) Gold 6146 CPU @

                                    3.20GHz

microcode        : 0x2000064

Guest: linux 4.18

we cannot find intel_pt flag in guest cpuinfo because
cpu_has_vmx_intel_pt() return false.

You may check vmx_pebs_supported().

It's true.

quoted

quoted

SECONDARY_EXEC_PT_USE_GPA/VM_EXIT_CLEAR_IA32_RTIT_CTL/VM_ENTRY_LOAD_IA32_RTIT_CTL 

are both disable.

Is it because microcode is not supported?

And, isthere a new macrocode which can support these bits? How can we
get this?

Currently, this patch set doesn't support guest PEBS on the Skylake
platforms, and if we choose to support it, we will let you know.

And now, we want to use pebs in skylake. If we develop based on pebs
patch set, do you have any suggestions?

- At least you need to pin guest memory such as "-overcommit mem-lock=true"
for qemu
- You may rewrite the patches 13 - 17 for Skylake specific because the
records format is different with Ice Lake.

OK. So, is there anything else we need to pay attention to except record 
format when used for Skylake?

You may need:
- remove x86_match_cpu check in the vmx_pebs_supported()
- add intel_pmu_handle_guest_pebs() to the intel_pmu_drain_pebs_nhm()

I suggest that you may pick up the one-one mapping patch from the v1
so that you can get avoid of patches 13 - 17.

quoted

quoted

I think microcode requirements need to be satisfied.  Can we use
https://github.com/intel/Intel-Linux-Processor-Microcode-Data-Files ?

You may try it at your risk and again,
this patch set doesn't support guest PEBS on the Skylake platforms 
currently.

quoted

quoted

---
thx,likexu

quoted

Thanks,

Liuxiangdong

Thanks. Liuxiangdong