On 8/2/23 09:33, Tom Lendacky wrote:

On 8/2/23 09:25, Tom Lendacky wrote:

quoted

On 8/2/23 09:01, Sean Christopherson wrote:

quoted

On Wed, Aug 02, 2023, Wu Zongyo wrote:

quoted

On Mon, Jul 31, 2023 at 11:45:29PM +0800, wuzongyong wrote:

quoted

On 2023/7/31 23:03, Tom Lendacky wrote:

quoted

On 7/31/23 09:30, Sean Christopherson wrote:

quoted

On Sat, Jul 29, 2023, wuzongyong wrote:

quoted

Hi,
I am writing a firmware in Rust to support SEV based on project 
td-shim[1].
But when I create a SEV VM (just SEV, no SEV-ES and no SEV-SNP) 
with the firmware,
the linux kernel crashed because the int3 instruction in 
int3_selftest() cause a
#UD.

...

quoted

BTW, if a create a normal VM without SEV by qemu & OVMF, the int3 
instruction always generates a
#BP.
So I am confused now about the behaviour of int3 instruction, 
could anyone help to explain the behaviour?
Any suggestion is appreciated!

Have you tried my suggestions from the other thread[*]?

Firstly, I'm sorry for sending muliple mails with the same content. I 
thought the mails I sent previously
didn't be sent successfully.
And let's talk the problem here.

quoted

quoted

    : > > I'm curious how this happend. I cannot find any condition 
that would
    : > > cause the int3 instruction generate a #UD according to 
the AMD's spec.
    :
    : One possibility is that the value from memory that gets 
executed diverges from the
    : value that is read out be the #UD handler, e.g. due to 
patching (doesn't seem to
    : be the case in this test), stale cache/tlb entries, etc.
    :
    : > > BTW, it worked nomarlly with qemu and ovmf.
    : >
    : > Does this happen every time you boot the guest with your 
firmware? What
    : > processor are you running on?
    :

Yes, every time.
The processor I used is EPYC 7T83.

quoted

quoted

    : And have you ruled out KVM as the culprit?  I.e. verified 
that KVM is NOT injecting
    : a #UD.  That obviously shouldn't happen, but it should be 
easy to check via KVM
    : tracepoints.

I have a feeling that KVM is injecting the #UD, but it will take 
instrumenting KVM to see which path the #UD is being injected from.

Wu Zongyo, can you add some instrumentation to figure that out if 
the trace points towards KVM injecting the #UD?

Ok, I will try to do that.

You're right. The #UD is injected by KVM.

The path I found is:
     svm_vcpu_run
         svm_complete_interrupts
        kvm_requeue_exception // vector = 3
            kvm_make_request

     vcpu_enter_guest
         kvm_check_and_inject_events
        svm_inject_exception
            svm_update_soft_interrupt_rip
            __svm_skip_emulated_instruction
                x86_emulate_instruction
                svm_can_emulate_instruction
                    kvm_queue_exception(vcpu, UD_VECTOR)

Does this mean a #PF intercept occur when the guest try to deliver a
#BP through the IDT? But why?

I doubt it's a #PF.  A #NPF is much more likely, though it could be 
something
else entirely, but I'm pretty sure that would require bugs in both the 
host and
guest.

What is the last exit recorded by trace_kvm_exit() before the #UD is 
injected?

I'm guessing it was a #NPF, too. Could it be related to the changes that
went in around svm_update_soft_interrupt_rip()?

6ef88d6e36c2 ("KVM: SVM: Re-inject INT3/INTO instead of retrying the 
instruction")

Sorry, that should have been:

7e5b5ef8dca3 ("KVM: SVM: Re-inject INTn instead of retrying the insn on 
"failure"")

Doh! I was right the first time... sigh

6ef88d6e36c2 ("KVM: SVM: Re-inject INT3/INTO instead of retrying the instruction")

Thanks,
Tom

quoted

Before this the !nrips check would prevent the call into
svm_skip_emulated_instruction(). But now, there is a call to:

   svm_update_soft_interrupt_rip()
     __svm_skip_emulated_instruction()
       kvm_emulate_instruction()
         x86_emulate_instruction() (passed a NULL insn pointer)
           kvm_can_emulate_insn() (passed a NULL insn pointer)
             svm_can_emulate_instruction() (passed NULL insn pointer)

Because it is an SEV guest, it ends up in the "if (unlikely(!insn))" path
and injects the #UD.

Thanks,
Tom