Andy Fox | 18 July 2012
For some time during most of the VMware courses that I teach, the question of whether the balloon memory driver should be disabled has been raised. I thought that it was something that warranted a blog, so here it is!
Although it is always desirable to have more than sufficient resources than required, we have to accept that the whole purpose of virtualisation is consolidation, and therefore to save money. Virtualisation is therefore "sold" as a concept to organisations with the premise that over commitment is possible. Unfortunately, many organisations do take this too far, and attempt to run far too many virtual machines with far too little resources, thus resulting in poor performance.
To allow for over commitment of memory resources, vSphere has a number of memory "reclamation" techniques (as VMware put it!).
Transparent Page Sharing
This is a background process running on each ESX/ESXi host that looks to see if any VMs have duplicate pages in RAM, the whole idea is to reduce the overall RAM requirements between VMs by only hold one copy and not multiple copies if they hold the same data. It's really a similar concept for RAM as disk deduplication in storage.
When the VMware tools are installed in the guest OS within the VM, one component is the balloon memory driver. It's job when instructed by the vmkernel is to "reclaim" the physical RAM that has been allocated to the VM but has not been passed back to the hypervisor.
A new feature in vSphere 4.1, this is a technique when memory is very scarce, the idea is to compress pages in RAM rather than use vmkernel swap files, i.e. use disk instead of RAM.
Each virtual machine has a vmkernel swap file (2 with vSphere 5), which is a failsafe, if all else fails and there simply is no more RAM, use disk instead.
From a performance perspective alone, they are listed in order, with Transparent Page Sharing not really having any effect, yet reducing overall memory requirements, to Host swapping being the option to avoid.
So now let's have to look at why memory over commitment is a problem.
On a physical PC, the OS is responsible for controlling memory resources, and will use memory for itself, and will allocate memory to processes. When these processes finish using the memory, it is either handed back to the OS, which then marks it as free or rather deallocated, or the process or rather application may hang on to the memory for future use. The important thing is that when memory is released back to the OS it is treated slightly differently to free memory that has never been used, in that when asked for memory the OS will favour this deallocated memory. This stems back to the good old days when memory was not that reliable, the fact that the memory is deallocated must mean that it has been used successfully by a process previously and therefore it is safe to use again. Free memory in implication has never been used and therefore it's integrity cannot be guaranteed.
So, given what happens on a physical PC, what's the story with a VM?
Well, the hypervisor will do the job of allocating physical RAM when the VM tries to use the memory it is configured with, and will continue to allocate physical RAM based of course on the virtual machine's settings (reservation and limit or configured amount whichever is the lower).
The problems will start either if there is a limit set on the amount of physical RAM the virtual machine is allowed to use being lower than the configured amount for the VM, or if the physical host is starting to run out of physical RAM to allocate. In either event, the hypervisor may be forced to issue memory from the vmkernel swap file (i.e. disk), if it either is not allowed to provide any more physical RAM (when a limit is set on the VM), or has no more physical RAM to allocate as the host has none free. Having the hypervisior (vmkernel) swap to disk is a failsafe, a get out of jail free! The VMs will continue to run if they are using disk instead of RAM, but it's fairly obvious that their performance will be dire or as VMware put it "Performance will be noticeably slow"!.
So how do I avoid Host swapping / use of the vmkernel swap file?
You have two options, ensure that the host or VM has sufficient physical RAM to meet requirements, or allow the balloon memory driver to claw back memory from VMs so over commitment is possible.
So what's the issue with the balloon memory driver anyway?
When the balloon memory driver is instructed to reclaim memory from the guest, it starts requesting memory from the guest OS, which in turn is given back to the hypervisor for it to use or pass on to other VMs requesting additional memory. As a consequence, the guest OS will typically start to use its own memory management processes as it will see less available memory due to the driver, the end result will typically be that the guest OS pages more. This obviously can affect the performance within the guest.
However, it is extremely important to understand that any performance degradation within the guest and application as a result of the guest paging more will be much lower than Host swapping. The reason for this is that the guest will choose which memory pages are paged to disk based upon their use, and will target pages that have not been used for some time. Host swapping does no such thing, the hypervisor is responding to a memory request by the VM, and has no idea of what the memory it then allocates will be used for.
One of the best blogs I have found on the subject is by Frank Dennerman Senior Architect in the Technical Marketing team at VMware, and co-author of VMware vSphere 5 Clustering technical deepdive and vSphere 4.1 HA and DRS technical deepdive.
Above all VMware do not recommend that the balloon memory driver should ever be disabled, and that the VMware Tools should be installed (and up to date) in every VM.