In our last post, we discussed some of the ways that virtualization is used both by individual consumers, and by enterprises. Recapping, the most prominent use is server consolidation, but there are plenty of other ways that virtualization can prove beneficial.

One particularly interesting topic is how the various uses of virtualization have influenced and impacted the design of modern servers. For the most part, the goal of virtualization and server consolidation is to minimize the power, heat and space foot print consumed by servers. As it turns out, this generally means that workloads are consolidated onto larger four socket servers which tend to maximize the capacity for guest operating systems, rather than the smaller single or dual socket servers.

Memory Capacity

Virtualization tends to heavily emphasize memory capacity, which is often the limiting factor in a given environment. As we discussed earlier, a regular operating system usually keeps the kernel in main memory to avoid frequent page faults. In a similar fashion, the entire VMM is usually kept in main memory, and is rarely paged to disk. In a virtualized environment, when a guest OS becomes active, large portions will be moved back to memory. This isn’t a problem when a server has one or two guest OSes, but when there are a dozen or so virtual machines on a server all contending for memory, the pressure becomes quite substantial. Each OS might eat up 0.5-1GB, plus another 0.5GB for the VMM. This can be a serious issue if several of the VMs are likely to be utilized at the same time – since contention for memory will result in a swap storm, with important data constantly being shuttled back and forth between memory and disk, and extremely poor responsiveness. Ultimately, memory capacity is often the gating factor in how many VMs can be hosted on a given server – which encourages server vendors to differentiate their offerings by providing more memory capacity. For instance, IBM’s current X4 chipset supports up to 1TB of memory in a single system – four times the capacity of most other x86 servers.

Reliability

Virtualization also substantially changes the equation for reliability and uptime. Isolation is the key to virtualization’s success – without it, IT departments would likely not be willing to deal with the problems associated with running dozens of applications together on the same server. Virtualization improves reliability and availability by insulating each guest OS from one another (and from any applications running above a guest OS). However, every guest OS is equally dependent on the VMM and hardware operating correctly; a fatal problem in either will cause the guest OS to become unavailable. With a dozen VMs on a single server, this means that the negative consequences of a hardware failure are a dozen times worse than before. The hardware (and the VMM) must become correspondingly more reliable in order to keep the same uptimes. This is one area that has rarely been mentioned and deserves extra emphasis – while virtualization does a lot to improve reliability, the hardware must be increasingly reliable as well to maintain the same level of service.

This is actually one way that IBM has historically differentiated their high-end server offerings. IBM actually uses a custom chipset for their 4-16 socket servers, with extra reliability and availability features designed into the silicon. Several years ago, I wrote an article about the IBM X3 chipset (http://www.realworldtech.com/page.cfm?ArticleID=RWT042405213553), which unlike most other servers at the time had memory scrubbing, hot-swap and bit-steering. IBM has since updated their line with the X4 chipset, which is discussed at a high level in this white paper (http://virtualizationconversation.com/wp-content/uploads/2008/12/ibm_x-architecture_blueprint.pdf).

I/O System

The last real impact of virtualization is on the I/O subsystem. As we discussed in our second post on virtualization (insert link), I/O is not really encompassed in current virtualization technologies – although it will be in the future. The only major changes required in the I/O system itself are the IOMMU, which is an element of the chipset. Currently, none of the standard chipsets incorporate this feature, although IBM has included an IOMMU in their server chipsets since the X3.

When it comes to the disk system, virtualization generally tends to increase the number of random seek requests initiated by the disk, simply because rather than sending a single stream of requests, the system is sending perhaps a dozen streams all intermingled. This can present a bit of a problem, in that random reads and writes are by far the most difficult disk pattern to deal with. This in turn tends could necessitate massive disk arrays or other exotic solutions. Given current trends though, it is more likely that solid-state drives will be adopted in situations with especially demanding I/O patterns, since they can service many times more random I/O requests.

The only other ingredient is the network interfaces, which are far simpler in many regards. The network I/O simply needs to have enough aggregate bandwidth to satisfy the various VMs residing on a given server. The only foreseeable impact here is that some enterprises may choose to more rapidly adopt 10 gigabit Ethernet, if current gigabit Ethernet is a bottleneck.