Windows XP: vmware to KVM migration

Like many others, I've become somewhat dependent on virtualization to reduce the number of computers and windows installs I have in my home. I recently took another stab at using open source virtualization, and I've had some success with WebVirtMgr, a libvirt-based VM management solution for Linux. This made me want to migrate some XP guests from vmware player to KVM, and I'm happy to say that this is relatively simple today once you figure out the precise sequence of events.

The Cheaper Way

I personally think that vmware is a fine piece of software, but as a bit of a skinflint I just can't see paying for [any version of] it. Apparently this does not trouble vmware much, as they give away the vmware player as a hook of sorts for drawing the user into their sphere of influence. There's a lot to be said for this strategy, because their software is actually of very high quality. It's certainly not perfect, and I've often managed to step on the VM by running seemingly innocuous software, but the support for 3d graphics is second to none, and the performance is really quite stellar so long as you host guests on their own devices, or on files on SSD. It's also extremely handy for doing maintenance on multi-booting hosts; you can connect real disks to virtual machines, and if you install Windows from vmware on the first disk in the system, install the drivers for the hardware, and then boot the install itself, you can freely boot the system either on the metal or in the virtual machine so long as you shut down and don't just hibernate or suspend.

If you decide to move these virtual machines, for example so that you can access them from disparate remote clients (at least more remote than the machine hosting them) then you might well move them to some libvirt-based solution. If you want web management, you could do a lot worse than WebVirtMgr. There is also Proxmox VE, which is free for personal use but which has some annoying limitations like one backup per virtual host. Installation of WebVirtMgr (etc.) is outside the scope of this document, but one hint I can give you is that you should use a host network and do all of your networking configuration manually. That's how you avoid libvirt starting dnsmasq with some rather poor options, which was a cheap way to avoid a race condition. Sadly, it has annoying ramifications for many typical systems today.

Virtualization

There are three open virtualization solutions for Linux, plus three main segregation strategies. Each one has its own particular advantages. The virtualization options are KVM, Xen, and VirtualBox. The primary segregation solutions are chroot, containers, and colinux virtual machines.

VirtualBox is a vmware player-like product which aims at end users, and while it has a number of features useful on the server, good management isn't one of them. Of the three FOSS options, it's the most fragile. KVM is slightly less mature than Xen, but it's part of the kernel now and it's become very popular. Xen was long difficult to manage, but that changed as it merged with XenSource and became XenServer. Since it's not a standard kernel feature, you need a special kernel to run it. The presumption is that you will prepare a server to run it, where KVM is a feature which can be added to any system. If KVM is comparable to vmware server, then Xen is like vmware ESX.

chroot is a system call (and command) that changes the apparent root directory as seen by a program and all that program's child programs. It can be used to segregate software away from other programs' files, but it still runs under the same kernel as all your other software so one privilege escalation hole and your whole system is compromised. Containers, whether they be provided by LXC, OpenVZ or Linux-Vserver, are like chroots but with memory protection, virtual network interfaces, and other features designed to provide more process segregation. colinux is Linux as a process under Linux, or another operating system. Each colinux process communicates with the world through virtual network interfaces shared with the host. Processes running in this virtual machine can only access memory which belongs to the kernel which hosts them.

Each of these options has its own distinct advantages and disadvantages. I chose KVM for two reasons. One, which is two reasons in itself, is that it's both up-and-coming and also already very good. The other reason is that the user space portions are based on QEMU. That means you can take your KVM virtual machines, strip them of their drivers, and load them up in QEMU even on another architecture. This would execute extremely slowly, but I prefer the kind of flexibility you get by planning ahead. This is a more-or-less future-proof scheme for the support of Windows, since it will run in QEMU without virtualization, on a completely emulated system.

Virtual Drivers

vmware and KVM each have their own set of virtual hardware which is emulated by the software, mostly intel chips. They also feature "paravirtualized" hardware drivers, which can be used to get much better performance from the virtualized guest. The normal virtual hardware has to waste a certain amount of time and effort pretending to be real hardware so that ordinary operating systems can interface with it. Paravirtualized hardware doesn't have this problem; the virtual hardware is designed to be operated by a simple driver, which doesn't have to account for the realities of hardware operating in the real world. This makes it both more reliable and faster.

The chief challenge in Windows migration is coping with mass storage drivers. These drivers must be coaxed into installation before full migration. The short form goes like this:

  1. Remove "vmware tools", which includes utilities and paravirtualization drivers used by vmware. This step is optional, but it may or may not work later. Some older guests may have older versions of vmware tools which refuse to uninstall if they do not detect vmware.
  2. Twiddle the Windows registry to support miscellaneous IDE controllers. As Microsoft says, you may get the error
    STOP: 0x0000007B (0xF741B84C,0xC0000034,0x00000000,0x00000000)
    INACCESSIBLE_BOOT_DEVICE
    "if the registry entries and the drivers for the mass storage controller hardware [...] are not installed in Windows XP." This step adds the registry entries to tell Windows to use the generic IDE drivers it always comes with to talk to all the controllers they know how to operate. This will get you up and booting with virtual hardware. Now you can shut down the virtual machine.
  3. Convert the virtual disk to the proper format. You probably will want either raw or qcow2 format. The qcow2 format permits a sparse file, which is one which will grow as it is actually used. This is how vmware behaves when you do not tell it to preallocate all disk space. You convert the file with the qemu-img program, like so: "qemu-img convert -O qcow2 diskimage.vmdk diskimage.qcow2" One of the slick things about recent versions of qemu-img is that they can handle multipart VMDKs. Just point it at the first one, the actual .vmdk file, and it will work out the rest. Drop the resulting file someplace sensible. If you're going to just start virtual machines from the command line, that could be anywhere. If you're using libvirt, then you probably know where your images dir is. Put it there.1
  4. Create/Boot a virtual machine with an added virtualized disk, and install the VirtIO mass storage driver. You want your settings to not use VirtIO drivers, and then you want to add in a VirtIO disk drive. When you boot, that drive won't affect booting in any way because your system won't have drivers for the paravirtualized storage controller. Then you'll install the VirtIO mass storage drivers. You can get the drivers from the Fedora project. Download the ISO, and put it into your iso images directory, or someplace otherwise accessible. You'll need to connect this to your guest so that you can install the rest of the drivers. Don't just let windows search the whole CD for drivers, especially if you are using Windows XP, as it will install the wrong ones. You need to drill down to the proper drivers for your operating system and architecture. You also need to set the type of your drive image! I created an empty 10MB file to serve as my temporary disk, and mounted it as the "raw" type. But when I specified my converted qcow2 disk, it was detected as raw, and I had to change it to "qcow2".
  5. Boot the machine with only paravirtual devices. If you're using WebVirtMgr or another libvirt-based solution, probably the easiest way is to just destroy the old instance and create another one, without deleting the image. But wait! If you're reading this all the way through before beginning, you have the chance to note that the easiest way to do this is to create the second instance first, the one with the VirtIO drivers that is, and don't boot it. Just use it as the basis to understand what to do to your config files, for now. You can then go copy the entries relating to the disk from the XML file, then create your temporary VM without VirtIO, and paste those entries into its XML config file. When you're done installing the network and storage drivers, you can delete the temporary instance, and just boot up the permanent one. As for video, you may have to manually change the type of the video device to "qxl".

  6. Finish the driver install. I unpacked the CD and added some more drivers to a new ISO image using genisoimage, notably The QXL video driver needed for the SPICE console and vdagent. However, once you have the network and mass storage drivers working, you should be able to get these files onto the host by other means. You will have to unpack RPM archives to get the agent files out and usable by Windows; I did this on Debian with rpm2cpio, like "rpm2cpio filename.rpm | cpio -dium". Put the two executable files appropriate to your architecture (32- or 64-bit) in a permanent directory of your choice and then run vdservice install from that location as an administrator to install the service. If you forgot to remove vmware tools, and you are migrating an older guest, it might well crash when the video driver is installed. If this happens, it will probably be after the driver is successfully installed, and it should work okay after boot. If not, Windows will let you know, and prompt you again to install the driver.

In theory, you should be done at this point. You shouldn't even need to reboot the guest!

XML Config File Mangling

It can be a little confusing knowing what to change in your configuration files, so here's some hints.

Disk Image Type

First, the image type will be a problem. When I created my first VM, and selected my converted image, the resulting XML looked like this:

    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/home/libvirt/images/disk.qcow2'/>
      <target dev='sda' bus='ide'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>

That didn't boot, because the file had a qcow header on the top of it, and didn't just begin with the boot loader like it would have with a raw file. Changing the type to "qcow2" solves the problem:

      <driver name='qemu' type='qcow2'/>

Adding a VirtIO disk

When you create a VM without VirtIO and one virtual HDD, the disk declaration looks like this:

    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/home/libvirt/images/disk.qcow2'/>
      <target dev='sda' bus='ide'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>

Whereas the version with VirtIO looks like this:

    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/home/libvirt/images/disk.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>

You don't want this to point at the same disk you're already using, so either create another storage volume through your management interface or just make an empty file with dd. It doesn't need to be of any particular size, I used 10MB (dd if=/dev/zero of=filename.img bs=1M count=10.) Then change the source file name. Note that you only get one device per "slot", so if you were going to just paste this into your XML, you'd need to scroll down and make sure that you didn't create a conflict. In my case, slot 0x04 is in use by memballoon by default:

    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </memballoon>

Since there's no slot 0x05 in use, we can just change that:

      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>

You only need to do this for the temporary instance.

QXL Video Driver for SPICE

The default console connection for libvirt guests seems to be VNC, which is highly compatible and requires no special drivers but which offers fairly mediocre performance. It's quite usable for most purposes, but a paravirtualized video card and driver can do much better. The video driver system is called QXL, and the Windows driver is called WinQXL. The default video card configured is an emulated Cirrus Logic GD 5446, which shows up in the XML file as type="cirrus":

    <video>
      <model type='cirrus' vram='9216' heads='1'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>

You can connect to the system with SPICE with this configuration, but the performance will be worse than with VNC. Instead, change the model type to "qxl", and perhaps mess with the vram value.2

      <model type='qxl' vram='9216' heads='1'/>

Appendix

Mass Storage Driver Floppy

Maybe you don't want to migrate a guest, maybe you want to install one to begin with. And maybe you want to start with VirtIO storage, from the beginning. In that case, you will either need to slipstream the virtio storage driver (at least) into the Windows ISO, or create a mass storage driver floppy disk image and configure your virtual machine to use it. Slipstreaming drivers is outside the scope of this document, but creating the driver floppy is actually fairly simple, as demonstrated in this completely borrowed snippet:

#create floppy & file system
dd bs=512 count=2880 if=/dev/zero of=viostor-0.1-xx-floppy.img
sudo mkfs.msdos viostor-0.1-xx-floppy.img
#mount images
sudo mkdir /mnt/img
sudo mount -o loop viostor-0.1-xx-floppy.img /mnt/img
sudo mkdir /mnt/iso
sudo mount -o loop virtio-win-0.1-xx.iso /mnt/iso

The instructions say to now copy the appropriate storage drivers to the floppy, but you could also simply download the floppy image, e.g. from kvm-guest-drivers-windows. At this point, you will still need to accommodate the floppy image in your XML configuration file:

<disk type='file' device='floppy'>
<source file='/home/libvirt/images/floppy.img'/>
<target dev='fda'/>
</disk>

Or otherwise specify the use of the floppy image to QEMU.

Bibliography

(Those links which did not appear in the body)

  • Rader, Udo. the not so easy migration from VMWare server to KVM for Windows guests. June 18, 2009.
    • 1. If you wrote your qcow2 file to a volume which supports sparse files, and it was generated from a VMDK which was not preallocated, then it should be a sparse file. That means that any unallocated space in the disk image is also unallocated in the qcow2 file. If you want to keep the file small like this, you can either write it directly to the target location (via NFS if necessary) or you can perform the copy using rsync (with -S), GNU tar (also with -S), or another command which understands sparse files.
    • 2. Set vram to whatever number of kilobytes of graphics memory you want. Psuedoarbitrary values seem to work fine. I have not studied the inner workings of the QXL driver, but in the case of real video cards you would ideally like to have at least twice as much graphics memory as you need to hold one framebuffer, For 1024x768*24bpp that comes out to 6144kB, or 6MB. The default is 9216 (as above) which is 9MB, or three times what you need to store a 24bpp XGA framebuffer, and that seems awfully coincidental, so it might be wise to triple your framebuffer size and then round up at least to the nearest megabyte.

    Add new comment

    Default

    • Use [fn]...[/fn] (or <fn>...</fn>) to insert automatically numbered footnotes.
    • You may link to images on this site using a special syntax
    • Web page addresses and e-mail addresses turn into links automatically.
    • To post pieces of code, surround them with <code>...</code> tags. For PHP code, you can use <?php ... ?>, which will also colour it based on syntax.
    • Internal paths in single or double quotes, written as "internal:node/99", for example, are replaced with the appropriate absolute URL or path. Paths to files in single or double quotes, written as "files:somefile.ext", for example, are replaced with the appropriate URL that can be used to download the file.
    • Filtered words will be replaced with the filtered version of the word.
    • Lines and paragraphs break automatically.
    • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd> <blockquote> <q>

    Issue

    • Lines and paragraphs break automatically.
    • To post pieces of code, surround them with <code>...</code> tags. For PHP code, you can use <?php ... ?>, which will also colour it based on syntax.
    • Allowed HTML tags: <a> <em> <strong> <cite> <blockquote> <code> <ul> <ol> <li> <dl> <dt> <dd>

    Drinking Game

    • Web page addresses and e-mail addresses turn into links automatically.
    • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd> <img> <p> <br> <pre> <h2> <h3> <h4>
    • Images may be embedded like: [image:node_id align=alignment hspace=n vspace=n border=n size=label width=n height=n nolink=(0|1) class=name style=style-data node=id] Leave off any attributes you don't want.
    • [img_assist|...] tags will be displayed, maybe. Please don't make more of them.

    Plain text

    • No HTML tags allowed.
    • Web page addresses and e-mail addresses turn into links automatically.
    • Lines and paragraphs break automatically.
    CAPTCHA
    This question is for testing whether you are a human visitor and to prevent automated spam submissions.
    Refresh Type the characters you see in this picture. Type the characters you see in the picture; if you can't read them, submit the form and a new image will be generated. Not case sensitive.  Switch to audio verification.