ARM virtualization issue

I will be the first to admit I don't know a lot about the ARM instruction set. However, I do know that instructions which behave differently at different privilege levels (and don't cause an interrupt) are a huge problem for virtualization.

The well-known "Formal Requirements for Virtualizable Third Generation Architectures" paper identified 17 problematic instructions on x86, the best example being POPF. The Intel Architecture Developer's Manual makes this understated observation:

The effect of the POPF/POPFD instructions on the EFLAGS register changes slightly, depending on the mode of operation of the processor.

In other words, you won't get a trap when trying to modify supervisor state with the POPF instruction in user mode.

In the PowerPC KVM implementation, we relied on the fact that a privileged instruction would trap. This enabled us to execute the vast majority of guest kernel instructions natively in user mode, since we would get a trap and could emulate any supervisor-only instructions. Ultimately, even without hardware support, we didn't need a complicated dynamic instruction translation engine (see VMware). Hardware support became a question of acceleration, rather than a requirement.

A colleague recently mentioned that ARM has a similar problem with the CPS instruction. Sure enough, from the ARM Architecture Reference Manual:

Exceptions: None.

Notes
User mode: CPS has no effect in User mode.

That's disappointing, because I had assumed that ARM, following similar RISCish principles to PowerPC, would have ended up with the same behavior. It took Intel years to add the necessary architecture changes for virtualization (VMX), and there is still no solution other than VMware's for the non-VMX processors.

From what I can tell, ARM TrustZone doesn't solve this problem... can anybody confirm?

Solaris: the awesomest virtualization ever?

There's an advocacy piece, in Forbes of all places, that offers a pretty unbalanced perspective on Solaris. (I hesitate to even link to it, because it feels like sensationalism just to generate huge numbers of outraged readers.) The author, Dan Woods, doesn't mention any negative points to Solaris at all, which should raise the suspicions of any reader with critical thinking skills, but I wanted to debunk some of the virtualization statements in particular:

Dan claims that Linux virtualization is the result of un-coordinated development from a number of companies, and Solaris virtualization is better because it's engineered "top to bottom" at a single company. Seamless integration can certainly offer advantages (see Apple), but I take issue with both his observations about the ecosystem and the conclusions he draws from them.

Aside from containers, Solaris uses hypervisors from Xen (marketed as xVM), and VirtualBox (from the innotek acquisition). Neither of those solutions were designed for Solaris; they were adopted years later to fill gaps in Sun's offerings. However, they are currently developed by Sun, so you still have the "single company" argument. About that...

Where I come from, being completely dependent on a single company is a bad thing, and I'm not even talking about the freedom of open source. It's called "vendor lock-in," and it's bad because there's no competition and customers are at the mercy of that single company's roadmap. Companies invest lots of money developing and supporting 3rd party ecosystems because it's a critically important to their customers. Anyways, looking at it from another angle, isn't it disturbing that virtualization ISVs don't consider Solaris important enough to target? Sun had to buy them outright or develop solutions in-house.

Dan claims Solaris containers cause a 2% performance degradation, vs. "about 20% for a hypervisor." While it's true that Forbes isn't a good forum for presenting performance analyses, without even a hint about where they came from, offering these numbers is ridiculous. It's often true that you can pick a specific benchmark and environment to support any argument, but Dan didn't even pretend.

Finally, I thought Dan's most interesting claim was the one for which he didn't offer any supporting arguments at all: that Solaris is now safe. Even if he's right, and Solaris is indeed the most awesome OS ever seen, that still doesn't guarantee it a slot on the Oracle roadmap.

Wind River Hypervisor release

Yesterday Wind River Systems released the hypervisor they announced last summer, complete with YouTube videos:

• Release announcement
• Mark Hermeling's blog entry
  
• LinuxDevices coverage
• EETimes coverage

Things I find interesting:

• From the demo video, we can see that the x86 port uses VT, but the PowerPC core doesn't have equivalent functionality (e500v2 is pre-ISA 2.06). That must mean they've got a paravirtual interface, likely the "virtual board interface" mentioned.
• EETimes quote: "We map I/O into the guest environment directly so it can directly talk to data registers to get higher performance." Given that they're releasing VxWorks MILS Platform 2.0 today for the hypervisor, which presumably requires high levels of isolation, I'm willing to bet that direct IO access is an option rather than a design assumption.
  
• Mark's demo videos had an emphasis on their Workbench IDE. I don't know if this was a conscious decision or not, but it does nicely reinforce the notion of the hypervisor as part of a solution, not a standalone product.
• They advertise their "MIPC" shared-memory inter-guest communication mechansim. I hope they just put a fancy name on virtio, but I doubt it. If they aren't using virtio, they're developing yet another set of virtual IO drivers for Linux. :(
• Their Linux patch isn't mentioned, but I hope they will be publishing it and merging it upstream, instead of the usual RTOS vendor approach to Linux...
• They consistently advertise the performance impact as "2-3% at worst." That's a nice marketing bullet, but is never the right answer to a vague performance question. The right answer is "it depends." In this case, it depends on the processor model, the workload, the partitioning configuration, and more. For example, VT-enabled x86 virtualization will have very different performance characteristics than paravirtualized PowerPC.

All in all, it's a significant event when a dominant embedded software company enters a new area like virtualization. As time goes by, it will be interesting to see how well they play with competitors' software... who is going to port Integrity to the Wind River virtual board interface? ;)

VMs on netbooks

OK, this post is about virtual machines in the JVM sense, not in the hypervisor sense. (The lines are getting a little blurry these days though.)

Back in the day, I once installed Windows NT on an RS/6000 (PowerPC) just to play with it. (It's funny how obsolete/impractical technology seemed so interesting back then, and these days it boggles my mind how anybody could care about Haiku, Amiga, etc.) So Windows NT: it installed OK, I started it up, and ran IE 2.0. That sucked (even at the time), but there were no updates for it. I ran the bundled Pinball game. That was the end of the story, because there was no ISV support. Just porting the kernel wasn't enough: an x86 WinNT application still couldn't run on PowerPC WinNT. The same fate could befall ARM netbooks (ahem "smartbooks").

This post suggests that .NET could be the answer. It starts by assuming that ARM netbooks will be common (a question on which the jury is still out), and then assumes Microsoft will somehow want to participate in that ecosystem (probably a safe bet: look at Windows ME on phones). Port some Windows kernel to ARM netbooks, provide a .NET runtime, and then just run .NET applications -- never worry about needing native ARM Windows binaries.

That has an existing parallel of course: J2ME on mobile phones. As a consumer, I'd call that a success. I love that I can download a random Java application and not worry about if the creator has built it for N phone vendors x M models. I'm sure J2ME has its limits, but it has made my life better.

And of course Google is walking down the same path with Dalvik. The cool thing about Java/Dalvik/.NET that it might just allow another processor architecture to compete with Intel without the legacy software issue. It will be interesting to see if Intel eventually enables Java on Moblin.

With Intel investing so many resources not just into the Linux kernel, but now into a full mobile Linux distribution (complete with UI), maybe Microsoft will annoy them right back by enabling ARM netbooks. You know both of them have to be looking to embedded systems for growth.

Anyways, I'd only buy a netbook that runs Linux. ;) I've heard good things about the ARM instruction set...

Oracle buys another hypervisor?!

Oracle made headlines for two acqusitions in the past month, Sun and Virtual Iron. By my count, that makes them the proud owners of no fewer than four x86 hypervisors: Oracle VM, Sun's xVM Server, Sun's xVM Desktop (a.k.a. VirtualBox), and Virtual Iron. (I've never really understood why Sun had two.) All but VirtualBox are based on Xen.

Even with this surprising success in the game of "collect the Xen implementations," that still leaves at least Red Hat, Novell, and of course Citrix itself offering competing Xen solutions. I'll admit the Unix wars predate me, but that seems like an impressive degree of fragmentation. Still, Red Hat has announced plans to abandon Xen for KVM, and even Novell included KVM as a tech preview in SLES11. There's no question that the number of Xen-based products is about to significantly shrink.

I've even seen one person speculate that the cost of maintaining Xen is so high, that with Red Hat pulling out, Oracle must have been worried about strengthening their Xen development capabilities.

What's worse though is that each of those Oracle hypervisors has its own management stack. Systems management is one of those areas that sounds really easy, but in practice never is. Management software contains so many layers that it's hard to find anybody who actually understands the end-to-end details of a particular code path. You need translation layers for components that are too old, too new, or fit at a different layer, or written for a different management stack. In this case, you might find one management stack built on xm, another built on libvirt, another built on CIM (and "enterprise frameworks" are a whole other world of complexity). Do they use OVF for image files? Should they? Every design question has tradeoffs and requires serious consideration.

Speaking from experience at a large company, I expect there will be at least 6 months of churn while architects furiously scribe presentations, rearrange block diagrams, create hypothetical development sizings, establish migration plans for legacy customers, escalate issues to management (which will be sorting itself out too), find and get input from related organizations ("how does this affect our relationship with VMware?"), and in general figure out what they're doing. After all the dust has settled, they'll still need to write the code.

Will the eventual result of this consolidation be a stronger Xen ecosystem a few years from now? To be honest, I couldn't care less... but it could be worth the cost of a bag of popcorn.

WAN optimization

I've wondered how "WAN optimization" magic works, and I just came across a page to explain it in (a little) more detail than marketing. I hadn't heard of it until a couple years ago, when some Cisco people mentioned that their WAASrouters embed KVM for virtualization.

Why would they do that? Because if your branch office uses an Active Directory server in your centralized data center, and your WAN link dies, work at the branch office ceases. From what I understand, Cisco's WAAS routers run an Active Directory server inside a virtual machine on the router itself, to mitigate that problem. A little googling reveals that similar approaches may be taken by their competition, 3Com and Riverbed.

In general, I expect we'll see much more virtualization in this area in the future. For example, today Cisco's Application Extension Platform (AXP) products are physical x86 cards you stick into a router to run server workloads. It would be plain silly not to take advantage of the well-known consolidation benefits of virtualization to accomplish the same thing. (That's pure speculation, but as I said... silly.)

package management and embedded Linux distributions

A while back, when it looked like the Kuro Box would actually go somewhere, I bought the original model. Among other things, this consists of a 200MHz Freescale e300 core (similar to PowerPC 603e) and 64MB of RAM. No serial port even, but through u-boot it has netconsole and netboot support. I had a decrepit version of Debian installed, and Debian's package management tools frustrate me to the point that I was helpless to get the thing upgraded (something about packages mysteriously "being held back").

For a system like this, the real value of a Linux distribution is the frequency of its security updates. There are some embedded distributions I could have messed with, but I have no idea how reliable their updates are, and I really don't want to be responsible for that myself. I started looking for a "normal" Linux distribution to install. (Unfortunately, the number of mainstream Linux distributions with PowerPC support are shrinking...)

I'm really a Fedora guy. Back in university I did some packaging for LinuxPPC and Yellow Dog Linux (both of which were Red Hat variants), and I'm comfortable enough with RPM to bootstrap a system from just about nothing. So I tried the Fedora installer, and when anaconda failed miserably with netconsole, I painfully installed it package by package.

Long story short, yum (the Fedora software installer) requires an absurd amount of RAM... way more than the 64MB my little Kuro has. The simplest yum operation was taking hours, and I could see I was well into swap. So after all that work, the box was still useless. I ran out of play time, unplugged the thing (since I couldn't install any security updates for it), and there it has sat for another 6 months.

Today I ran across a blog post comparing the speed and memory consumption of yum and zypper (OpenSUSE's package software installer). I don't know much about OpenSUSE, but I will be trying it next...