Xen, why I want PV/PVHs, and why HVMs are not a solution

So, explained earlier I wanted a PV/PVH of something, and got a response back hours later along the lines of “Oh you can solve your problem by using an HVM”.

So let's explain why that's terrible advice on all levels to someone who has voluntarily installed Xen and has already indicated they want a PV/PVH.

Back in the early days of “I heard you like operating systems so I put an operating system inside your operating system LOL”, there were really two types: emulation, and, as CPUs became more powerful, regular old virtualization where a wrapper would run your operating system and rely upon CPU traps (subroutines called if your program ran instructions it shouldn't) to give the underlying system the appearance it had the hardware all to itself.

Old people like me remember a big fuss about the 68010 coming out after the 68000 and being considered very, very, powerful for two reasons: (1) it was capable of having an MMU (which isn't 100% relevant here but that's definitely a nice to have) and (2) it protected a status register so that if an unprivileged process tried to use it, you could swoop in and emulate the instruction, because it happened that status register told you if you were running in user or supervisor mode, and if you were doing virtualization, you wanted operating systems running in user mode to think they were running in supervisor mode, and so on.

OK, anyway.

Along comes Xen. Xen introduces (well, on microcomputers anyway), a third approach called paravirtualization. Xen itself is a dedicated operating system, called a hypervisor, whose job is just to run operating systems. With paravirtualization, guest operating systems are required to be aware that they're running in a box, but that's good, because they can cooperate with one another, making the best use of the available resources. The program running the operating systems no longer needs to pretend, say, that there's an Adaptec UltraSCSI controller on PCI slot 00:01 with a 45Mb SCSI drive on LUN 2, and implement the layers of emulation needed to make it look like one, it can just tell the guest operating system “Hey, I got you some disk space, and a network device, just send me requests for data etc and I'll process them using a lightweight API.” In fairness, it goes beyond device drivers, it's also memory handling and time management and stuff like that. But you get the gist. The guest OS can basically ask the hypervisor for things that would take it a long time to do, and that the virtualization layer would have to intervene with all the time anyway, and so everything's more efficient on every level.

This is good! It's really efficient! Everyone is happy. People clamor for CPU makers to come up with ways to make it even more efficient but otherwise it's great!

Well, back up. Not everyone is happy. The Linux people, for reasons known only to themselves, decide they're not happy with this at all. Maybe it takes all the fun out of kernel development if they have to develop an efficient version that runs in a box it's happy to live with? Maybe they're just annoyed at the Xen developers. I'm not sure. I think it was mostly the latter from memory.

So instead they put a lot of effort into creating their own “hypervisor” which instead of being a lightweight container of operating systems is the giant clunky Linux kernel itself. And instead of implementing lightweight protocols for guest operating systems to use cooperatively, which would unlikely to be adopted because that'd mean Microsoft cooperating with Linux (back in the early 2000s that was not a thing), they decided instead to use a version of the “convince an OS it's running in a real PC” approach but doing what they can to make it efficient, using some of tricks Xen has introduced with lightweight device drivers etc, but ultimately making it resemble the earlier privileged wrapper approach that convinces operating systems they have the hardware to themselves. This is called KVM which stands for Kernel Virtualization Manager and is a pun because a KVM is a widget that allows you to control multiple computers from the same keyboard, video monitor, and mouse.

Now there's one advantage to this otherwise crappier approach, at the time, which is you can run operating systems that weren't modified to run in a box. Not that the Xen people haven't recognized this problem, they also create a hack, called HVM, to run operating systems in a similar way. But HVM, at the time, isn't the preferred way to run guest operating systems, for obvious reasons.

So the two coexist, but KVM, alas, takes a significant amount of mindshare from Xen because Linux has more devs than Xen. RedHat and others push KVM as THE solution to running multiple operating systems on the same box, while Microsoft works on its own, and Xen just flounders, with nobody to push it and reduced developer interest.

A major change.

In the mid 2000s the amd64 CPU standard, otherwise known as “64 bit PCs”, is introduced by AMD, adopted by Intel, and it introduces more efficient ways to sandbox an operating system. The CPU architecture rapidly takes off after being heavily promoted by Apple of all people. How efficient is amd64's virtualization layer? Super efficient! In fact, it's so efficient that even with the crappier emulation of devices that older virtualization systems did before paravirtualization, it's still more performant than the way Xen does things natively. The reason is because Xen has to perform certainly ugly tricks to allocate memory that amd64 does in a better way.

Xen works to improve its non-native guest support (HVM), and finds its developers actually recommending people use HVM instead of the paravirtualization mode PV to run guests.

Needless to say, this does not help in the Xen vs KVM war. With the release and popularization of amd64:

Xen's developers do try to work on a solution to the efficiency issues, PVH, that combines the best of HVM and the best of PV, to make a super virtualization mode that requires operating systems cooperate but is much, much, faster as a result. And... remember I said KVM took away the mindshare? Well, it did, and it takes a long time before PVH is considered production ready. Even today it's not even supported by XenServer.

I've tried PVH. It works. People have benchmarked it and found it's faster than both PV and HVM. Hopefully they'll get the kinks out and make it the default soon.

But back up a moment

So that's the history. So you've used Xen and you found PV too slow, and you're running around advising people to “just use HVM” rather than PVH for a configuration that should work fine under PVH.

Why?

Seriously, why are you using Xen at all if you only use HVM?

Xen isn't well supported. It took them the best part of a decade to get a better version of PV out, and you haven't even looked at it. HVMs are not user friendly. Xen (not XenServer) is kinda sorta friendly if you use xen-tools with a regular GNU/Linux as dom0, but xen-tools creates PV images, not HVM images.

So why are you not using KVM? Because you want to wave your “I love Xen” flag around?

I can be mad at what the KVM devs did, I think if they'd understood Xen and wanted to improve it and work on it, Xen would be in a much better position right now. But they didn't, history is history, I bet 90% of KVM devs were assigned to it by their RedHat or Canonical employed manager, they're unfamiliar with Xen, it's not really their fault.

But it's completely dumb to use Xen when you're not interested in the one feature that makes Xen special, and when everyone else has implemented superior versions of the remaining features.

If someone says “I am trying to get a PV/PVH that does X”, a response of “Use an HVM!” is as helpful as “Use Virtual Box” or “Why are you trying to run a Debian VM when you could run Fedora!”

I hope Xen gets its support back. But I can tell you that as long as its advocates keep pushing HVMs as the solution to every problem, it's not going to get it.