ARM virtualization issue

I will be the first to admit I don't know a lot about the ARM instruction set. However, I do know that instructions which behave differently at different privilege levels (and don't cause an interrupt) are a huge problem for virtualization.

The well-known "Formal Requirements for Virtualizable Third Generation Architectures" paper identified 17 problematic instructions on x86, the best example being POPF. The Intel Architecture Developer's Manual makes this understated observation:

The effect of the POPF/POPFD instructions on the EFLAGS register changes slightly, depending on the mode of operation of the processor.

In other words, you won't get a trap when trying to modify supervisor state with the POPF instruction in user mode.

In the PowerPC KVM implementation, we relied on the fact that a privileged instruction would trap. This enabled us to execute the vast majority of guest kernel instructions natively in user mode, since we would get a trap and could emulate any supervisor-only instructions. Ultimately, even without hardware support, we didn't need a complicated dynamic instruction translation engine (see VMware). Hardware support became a question of acceleration, rather than a requirement.

A colleague recently mentioned that ARM has a similar problem with the CPS instruction. Sure enough, from the ARM Architecture Reference Manual:

Exceptions: None.

Notes
User mode: CPS has no effect in User mode.

That's disappointing, because I had assumed that ARM, following similar RISCish principles to PowerPC, would have ended up with the same behavior. It took Intel years to add the necessary architecture changes for virtualization (VMX), and there is still no solution other than VMware's for the non-VMX processors.

From what I can tell, ARM TrustZone doesn't solve this problem... can anybody confirm?

Solaris: the awesomest virtualization ever?

There's an advocacy piece, in Forbes of all places, that offers a pretty unbalanced perspective on Solaris. (I hesitate to even link to it, because it feels like sensationalism just to generate huge numbers of outraged readers.) The author, Dan Woods, doesn't mention any negative points to Solaris at all, which should raise the suspicions of any reader with critical thinking skills, but I wanted to debunk some of the virtualization statements in particular:

Dan claims that Linux virtualization is the result of un-coordinated development from a number of companies, and Solaris virtualization is better because it's engineered "top to bottom" at a single company. Seamless integration can certainly offer advantages (see Apple), but I take issue with both his observations about the ecosystem and the conclusions he draws from them.

Aside from containers, Solaris uses hypervisors from Xen (marketed as xVM), and VirtualBox (from the innotek acquisition). Neither of those solutions were designed for Solaris; they were adopted years later to fill gaps in Sun's offerings. However, they are currently developed by Sun, so you still have the "single company" argument. About that...

Where I come from, being completely dependent on a single company is a bad thing, and I'm not even talking about the freedom of open source. It's called "vendor lock-in," and it's bad because there's no competition and customers are at the mercy of that single company's roadmap. Companies invest lots of money developing and supporting 3rd party ecosystems because it's a critically important to their customers. Anyways, looking at it from another angle, isn't it disturbing that virtualization ISVs don't consider Solaris important enough to target? Sun had to buy them outright or develop solutions in-house.

Dan claims Solaris containers cause a 2% performance degradation, vs. "about 20% for a hypervisor." While it's true that Forbes isn't a good forum for presenting performance analyses, without even a hint about where they came from, offering these numbers is ridiculous. It's often true that you can pick a specific benchmark and environment to support any argument, but Dan didn't even pretend.

Finally, I thought Dan's most interesting claim was the one for which he didn't offer any supporting arguments at all: that Solaris is now safe. Even if he's right, and Solaris is indeed the most awesome OS ever seen, that still doesn't guarantee it a slot on the Oracle roadmap.

WAN optimization

I've wondered how "WAN optimization" magic works, and I just came across a page to explain it in (a little) more detail than marketing. I hadn't heard of it until a couple years ago, when some Cisco people mentioned that their WAASrouters embed KVM for virtualization.

Why would they do that? Because if your branch office uses an Active Directory server in your centralized data center, and your WAN link dies, work at the branch office ceases. From what I understand, Cisco's WAAS routers run an Active Directory server inside a virtual machine on the router itself, to mitigate that problem. A little googling reveals that similar approaches may be taken by their competition, 3Com and Riverbed.

In general, I expect we'll see much more virtualization in this area in the future. For example, today Cisco's Application Extension Platform (AXP) products are physical x86 cards you stick into a router to run server workloads. It would be plain silly not to take advantage of the well-known consolidation benefits of virtualization to accomplish the same thing. (That's pure speculation, but as I said... silly.)

embedded virtualization use case: hot upgrade

Mark Hermeling at Wind River has written about a valuable use case for virtualization in embedded systems: No Downtime Upgrade. I admit I often omit this from my presentations because it doesn't seem as sexy as other use cases, but I'm glad he wrote about it because to some systems it's absolutely critical.

Basically it goes like this: to upgrade the software running in some embedded device, you do the upgrade in a virtual machine clone of the active code. Then, when it's all patched and running, you switch over from the old code, which has been running the whole time. You can imagine how this applies to High Availability in general, outside the software upgrade case.

To me it doesn't sound that sexy, at least not compared to consolidating multiple hardware systems (e.g. networking control and data planes) onto a single system without impacting reliability. But consider this: some of these embedded devices handle lots of IO traffic all the time. Think about a RAID controller in a busy server, or a network backbone. If it takes you 10 seconds to upgrade your software, how many packets or IOps have you missed?

(Of course, unless you have built an extra core into your hardware design, this is something you can't really do if you're stuck with strict hardware isolation...)