From: Peter Zijlstra <peterz@infradead.org>
Date: Mon, 14 Aug 2023 13:44:35 +0200
Subject: x86/cpu/kvm: Provide UNTRAIN_RET_VM
Git-commit: 864bcaa38ee44ec6c0e43f79c2d2997b977e26b2
Patch-mainline: v6.5-rc7
References: git-fixes
Similar to how it doesn't make sense to have UNTRAIN_RET have two
untrain calls, it also doesn't make sense for VMEXIT to have an extra
IBPB call.
This cures VMEXIT doing potentially unret+IBPB or double IBPB.
Also, the (SEV) VMEXIT case seems to have been overlooked.
Redefine the meaning of the synthetic IBPB flags to:
- ENTRY_IBPB -- issue IBPB on entry (was: entry + VMEXIT)
- IBPB_ON_VMEXIT -- issue IBPB on VMEXIT
And have 'retbleed=ibpb' set *BOTH* feature flags to ensure it retains
the previous behaviour and issues IBPB on entry+VMEXIT.
The new 'srso=ibpb_vmexit' option only sets IBPB_ON_VMEXIT.
Create UNTRAIN_RET_VM specifically for the VMEXIT case, and have that
check IBPB_ON_VMEXIT.
All this avoids having the VMEXIT case having to check both ENTRY_IBPB
and IBPB_ON_VMEXIT and simplifies the alternatives.
Fixes: fb3bd914b3ec ("x86/srso: Add a Speculative RAS Overflow mitigation")
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Link: https://lore.kernel.org/r/20230814121149.109557833@infradead.org
Acked-by: nborisov <nik.borisov@suse.com>
---
arch/x86/include/asm/nospec-branch.h | 11 +++++++++++
arch/x86/kernel/cpu/bugs.c | 1 +
arch/x86/kvm/svm.c | 4 +---
3 files changed, 13 insertions(+), 3 deletions(-)
--- a/arch/x86/include/asm/nospec-branch.h
+++ b/arch/x86/include/asm/nospec-branch.h
@@ -190,6 +190,17 @@
#else /* __ASSEMBLY__ */
+#if defined(CONFIG_RETPOLINE) || defined(CONFIG_CPU_SRSO)
+#define UNTRAIN_RET_VM \
+ ALTERNATIVE_2( \
+ "", \
+ "call entry_untrain_ret", X86_FEATURE_UNRET, \
+ "call entry_ibpb", X86_FEATURE_IBPB_ON_VMEXIT)
+#else
+#define UNTRAIN_RET_VM
+#endif
+
+
#define ANNOTATE_NOSPEC_ALTERNATIVE \
"999:\n\t" \
".pushsection .discard.nospec\n\t" \
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -984,6 +984,7 @@ do_cmd_auto:
case RETBLEED_MITIGATION_IBPB:
retbleed_force_ibpb:
setup_force_cpu_cap(X86_FEATURE_ENTRY_IBPB);
+ setup_force_cpu_cap(X86_FEATURE_IBPB_ON_VMEXIT);
mitigate_smt = true;
break;
--- a/arch/x86/kvm/svm.c
+++ b/arch/x86/kvm/svm.c
@@ -5693,9 +5693,7 @@ static void svm_vcpu_run(struct kvm_vcpu
"mov %%r14, %c[r14](%[svm]) \n\t"
"mov %%r15, %c[r15](%[svm]) \n\t"
#endif
-
- ALTERNATIVE("", "call zen_untrain_ret", X86_FEATURE_UNRET)
- ALTERNATIVE("", "call entry_ibpb", X86_FEATURE_IBPB_ON_VMEXIT)
+ UNTRAIN_RET_VM
/*
* Clear host registers marked as clobbered to prevent