embedded VMware on the Nokia n800

Last year VMware acquired a French company called Trango, who had a tiny hypervisor written for ARM and MIPS. It's completely paravirtualization, i.e. modifications to the guest kernels that run on top of it (including Linux, though no patches are publicly available that I've found). (Mind you, it's only a thin hypervisor, but that probably makes sense for these low-end processors.)

So that's where this WinCE+Android demo comes from. It's cool, but is it just a demo? Who actually wants virtualization in consumer electronics? Well, another embedded virtualization vendor, OK Labs, claims they're installed in 250 million cell phones.

The funny thing is that I expect virtualization to be adopted more quickly in networking (Cisco, Juniper, et al) than consumer electronics. Of course, you probably won't see flashy demos from there, just more robustness, more features, and maybe even a better profit margin from the vendors (because they don't have to rewrite as much software).

Anyways, if there is still doubt about virtualization in embedded systems, I think demos like this should help evaporate it.

Update: Reacting to the same story, Ars Technica opines that embedded virtualization is "inevitable."

multicore and unicore virtualization

One of the factors driving interest in virtualization for embedded systems is the increasing penetration of multicore processors in the embedded world. Although multicore systems have been around in the server world for decades, most traditional embedded systems have been strictly uniprocessor.

However, multicore processors are a big deal in some segments of the embedded world these days. I think most of the traditional RTOSs have been multicore-enabled by now, but what about all the other layers in the software stack?

One answer for dealing with single-threaded software in a multicore world is virtualization, or more specifically partitioning: carve up your multicore system into multiple single-core systems, and then you can run that legacy software without modification. In fact, you can still win on hardware costs, because you probably save money on space, power, cooling, etc.

But virtualization makes sense on unicore systems too! Consolidating multiple legacy systems onto a single new processor can save you space, cooling, and maybe even still improve performance if the legacy hardware is a lot slower than the new stuff.

So consolidation can make sense, but there's another really interesting benefit too. For some embedded system developers, software development resources are actually a major limiting factor. These folks want to roll out a product line spanning 1-, 2-, and 4-core systems, but they don't have enough software developers to re-engineer their stack for each system.

Enter virtualization: time-share the single-core system to make it appear to have four cores. Is it fast? Hell no, but that was never the point: this is the low-end model. The most important thing is that you've managed to stretch your software investment to cover the whole line.

This may sound like something you can do with threads in a normal time-sharing kernel, but that doesn't work if the software is actually kernel-level code. For example, in an asymmetric multiprocessing (AMP) networking system, you may have an RTOS (or less) handling the data plane (pushing packets), and Linux as the control plane. Instead of implementing a data plane surrogate, virtualization allows you to preserve the same software layers and go on to solve more interesting problems.

embedded virtualization use case: hot upgrade

Mark Hermeling at Wind River has written about a valuable use case for virtualization in embedded systems: No Downtime Upgrade. I admit I often omit this from my presentations because it doesn't seem as sexy as other use cases, but I'm glad he wrote about it because to some systems it's absolutely critical.

Basically it goes like this: to upgrade the software running in some embedded device, you do the upgrade in a virtual machine clone of the active code. Then, when it's all patched and running, you switch over from the old code, which has been running the whole time. You can imagine how this applies to High Availability in general, outside the software upgrade case.

To me it doesn't sound that sexy, at least not compared to consolidating multiple hardware systems (e.g. networking control and data planes) onto a single system without impacting reliability. But consider this: some of these embedded devices handle lots of IO traffic all the time. Think about a RAID controller in a busy server, or a network backbone. If it takes you 10 seconds to upgrade your software, how many packets or IOps have you missed?

(Of course, unless you have built an extra core into your hardware design, this is something you can't really do if you're stuck with strict hardware isolation...)

thin vs thick hypervisors

I made a presentation at the 2008 Linux Plumbers Conference about "thin" vs "thick" hypervisors, a subject very important to KVM in embedded systems. It's true Linux doesn't fit everywhere, and many embedded systems people have been dismayed by Linux's increasing memory footprint (which also hurts performance through cache and TLB pressure). Kevin Lawton has also written an article about this issue: it would increase KVM's appeal in embedded systems if we could first slim it down.

However, there's an important issue here that is sometimes obscured by memory footprint: functionality. Many of the proprietary "embedded virtualization" solutions (offered by vendors like OK Labs, Trango/VMware, VirtualLogix, et al) are thin precisely because they don't do a whole lot. In many cases, they are strictly about isolating hardware. That sounds good, right?

Strictly hardware isolation is a double-edged sword, because while it allows you to minimize the virtualization layer (good for memory/flash footprint, security, etc), it doesn't let developers take advantage of most of virtualization's benefits. I'm not even talking about "frills" like live migration (which could still be considered critical for high-availability in network infrastructure); I'm talking about basic consolidation, which is virtualization's bread and butter (and yes, software consolidation still makes sense in embedded systems).

You have 2 cores? If you only have 1 network interface, with a thin hypervisor you can only have 1 partition on the network. But even if you have 2 network interfaces for your 2 cores, there are still pesky bits of hardware you must consider. How many interrupt controllers do you have? How many real-time clocks? How many UARTs, and can you isolate them with the MMU, or are they all in the same page? If your software stacks require nonvolatile storage, how many flash controllers do you have?

For very simple services like PIC control, you can get away with embedding that directly in the hypervisor itself. Note however that the PICs on some modern systems require rather complicated configuration code, and re-implementing that can actually be pretty tricky. (I'm thinking about x86 ACPI and PowerPC device tree parsing here.)

Regardless, the only alternative for these services with a "thin" hypervisor is a dom0-style service partition. ("dom0" is the Xen terminology for the all-powerful Linux partition that is allowed to muck with all hardware on the system; all other partitions depend on it for IO services.) Once you go that route, you still have a thin hypervisor, but you've completely lost the security and reliability benefits that may have brought you to virtualization in the first place.

So for very simple use cases on very low-resource systems, thin hypervisors can make sense. But once you start to need anything beyond strict hardware partitioning, you've already entered the world of "thick" hypervisors, and the larger footprint of KVM becomes much less of an obstacle.

Virtualization hardware support for embedded PowerPC

The Power Instruction Set Architecture (ISA) version 2.06 is now available from Power.org. I'm pretty excited about this because it finally includes hardware virtualization support for embedded PowerPC (in contrast to server PowerPC, which has had it for years).

So why does this matter? After all, VirtualLogix, Sysgo, and others have already implemented virtualization layers for embedded PowerPC. The problem is that these solutions require loss of isolation, loss of performance, dramatic modifications to the guest kernels, or some combination of all three.

In KVM for PowerPC (currently supporting PowerPC 440 and e500v2 cores), we chose to maximize isolation and require no changes to the guest kernel; we do this by running the guest kernel in user mode. As a consequence, we suffer reduced performance, since every privileged instruction in the guest kernel traps into the host, and we must emulate it. (Of course, the vast majority of instructions execute natively in hardware.)

Of course, the most important question is when we'll actually see hardware implementing this version of the architecture (and even then, virtualization is an optional feature). IBM hasn't made any announcements about that, but IBM hasn't delivered a new embedded core for a long time now. Freescale has already announced that their e500mc core (as found in their upcoming P4080 processor) will implement virtualization support, and they are even developing their own hypervisor for it.

(Updated Oct 2009 to fix KVM wiki URL.)