Skip to content

Latest commit

 

History

History
107 lines (78 loc) · 3.4 KB

research.rst

File metadata and controls

107 lines (78 loc) · 3.4 KB

KVM Research Doc

General KVM usage

KVM is a kernel module which is used by a user-mode process to create and interact with virtual machines.
All interactions between the user-mode and KVM are done using ioctls and file descriptors .

The typical boilerplate usage will look something like this:

  • open("/dev/kvm") -> kvm_fd
  • ioctl(kvm_fd, KVM_CREATE_VM) -> vm_fd
  • ioctl(vm_fd, KVM_CREATE_VCPU) -> vcpu_fd
  • ioctl(vm_fd, KVM_SET_USER_MEMORY_REGION, struct vm_memory_region *)
  • ...
  • ioctl(vcpu_fd, KVM_RUN)

This POC

In this POC I wanted to demonstrate how to inspect a VM's state (registers and memory).

To ensure the state doesn't change while we inspect it, we need our code to run after the user mode called the ioctl and before KVM executes the VM.

To do this, I hook kvm_vcpu_ioctl(), where KVM handles the KVM_RUN ioctl.
Ideally, we'll wait for all vcpus to call this ioctl and only then make the inspection.

From this point we can receive the following:

static long kvm_vcpu_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg)
{
   struct kvm_vcpu *vcpu = filp->private_data; // the vcpu to run
   struct kvm *kvm = vcpu->kvm;  // the owning virtual machine
   ...

Internal KVM memory structure

struct kvm {
   ...
   struct mm_struct *mm; /* userspace tied to this vm */
   struct kvm_memslots __rcu *memslots[KVM_ADDRESS_SPACE_NUM];  // KVM_ADDRESS_SPACE_NUM is 1 or 2
   struct kvm_vcpu *vcpus[KVM_MAX_VCPUS];
   atomic_t online_vcpus;
   ...
}

If KVM_ADDRESS_SPACE_NUM == 2 the second entry references the SMM memory address space. [1]

We can iterate through the memory slots with kvm_for_each_memslot() from include/linux/kvm_host.h.
From there we can find all memory mappings for this vm and scan the guest's memory.

Each memory slot contains the following:

struct kvm_memory_slot {
    gfn_t base_gfn;  // guest page number
    unsigned long npages;
    unsigned long *dirty_bitmap;
    struct kvm_arch_memory_slot arch;
    unsigned long userspace_addr;
    u32 flags;
    short id;  // a user-provided ID
};

We can read entire pages from the guest using kvm_read_guest_page(), since every memory slot must be page-aligned in address and length:

int __kvm_set_memory_region(struct kvm *kvm, const struct kvm_userspace_memory_region *mem) {
   ...
   if (mem->memory_size & (PAGE_SIZE - 1))
    goto out;
   if (mem->guest_phys_addr & (PAGE_SIZE - 1))
    goto out;
   ...
}

Scanning the VM's physical memory

Reading with kvm_read_guest_page() is memory intense (copying every page over and over again).
There's a neat function called apply_to_page_range() which was introduced by the Xen project [2] that can be handy,
it applies a function for every user page in a certain range.

Another (and probably better) option is to find the relevant page structures via kvm->mm and map them using vmap().

I use __get_user_pages_fast() to find the relevant page structure and vmap() to map these pages to our memory space.

[1]
[2]https://lwn.net/Articles/182495/