Linux: KVM: arm64: nv: Fix null ptr deref on nested wp/unmap, teardown race

In the Linux kernel, the following vulnerability has been resolved:

KVM: arm64: nv: Fix null ptr deref on nested wp/unmap, teardown race

Commit 7270cc9157f4 ("KVM: arm64: nv: Handle VNCR_EL2 invalidation from MMU notifiers") introduced VNCR_EL2 invalidation in both kvm_nested_s2_unmap() and kvm_nested_s2_wp().

However at the point of this being performed concurrent stage 2 teardown of a nested guest can cause kvm->arch.mmu.pgt to be set to NULL.

This happens in kvm_flush_shadow_all() -> kvm_arch_flush_shadow_all() -> kvm_free_stage2_pgd() and is performed under the kvm->mmu_lock.

Commit ec14c272408a ("KVM: arm64: nv: Unmap/flush shadow stage 2 page tables") introduced the teardown of the entire nested MMU range, which then invokes stage2_apply_range() with resched=true:

mmu_notifier_invalidate_range_start() -> ... -> kvm_mmu_notifier_invalidate_range_start() -> kvm_mmu_unmap_gfn_range() -> kvm_unmap_gfn_range() -> kvm_nested_s2_unmap() -> kvm_stage2_unmap_range() -> __unmap_stage2_range() -> stage2_apply_range()

This means that stage2_apply_range() can drop the kvm->mmu_lock and thus concurrent progress can be made in lockstep with kvm_arch_flush_shadow_all().

If kvm_arch_flush_shadow_all() advances ahead of stage2_apply_range() and completes its operation it guarantees a NULL pointer deref.

Since kvm_free_stage2_pgd() is performed under the kvm->mmu_lock this will either be observed NULL or not and serialised against kvm_free_stage2_pgd().

Resolve the issue by abstracting the invalidation to a new function, kvm_invalidate_vncr_ipa_all(), and check that the pgt is non-NULL before dereferencing it.