This post was originally published on September 9, 2015 and updated November 14, 2019.
One of the benefits of nested virtualization is running multiple hypervisors on the same host server. This includes ESXi hypervisors , which can be installed as a virtual machines. The solution, commonly referred to as Nested ESXi, opens the door to a lot of interesting use cases such as development and testing, virtual training, or virtual sales demos and POCs.
Although the concept of nested virtualization is not limited to VMware, it is by far the most popular hypervisor. For that reason I decided to focus my article on the VMware hypervisor.
In this article, I am going to answer some common questions, discuss performance and common use-cases, provide a short walkthrough on how to enable a Nested ESXi and provide a list of CloudShare’s preconfigured ESXi environments.
I bet that when you first encountered nested virtualization, some of the first questions to pop into your head were most likely along of the lines of:
– “Why would you ever want to do such a scary thing?”
– “Isn’t normal Virtualization good enough anymore?”
– “It’s going to be extremely slow, isn’t it?!”
Before we move forward, let me answer your [burning] questions:
– Great benefit in certain use cases!
– It’s great!! But not for all scenarios.
– Potentially it would be a little slower, but not much if you do it right, and I’ll explain more next.
Why should I use this solution? What use cases is nested virtualization good for?
Practice an Upgrade Procedure of your VSphere Infrastructure
Suppose you want to upgrade your entire production VSphere infrastructure. What’s the catch? You’ve never done this before. What better way of experimenting than doing this on a nested environment? Performing the upgrade on a nested environment could save you the trouble and downtime of the production infrastructure, especially if you don’t have ‘spare’ ESX hosts to play with. Plus, you can enjoy all the added features that come along with a cloud based virtual IT platform such as snapshotting current state and reverting back to previous state.
Suppose you or your company is building a new solution and you need a virtual IT lab to rapidly create and provision environments for build verification, test automation and/or manual testing. You can deploy Nested ESXi as a test/development platform without spending money on hardware and without changing any of your in-house tools or processes. By creating Nested ESXi templates you can easily deploy and tear down vSphere instances as needed during development and test cycles.
Virtual Sales Demos
What better way of selling your VMware vSphere solution than showing a live virtual demo of your product? Your sales staff can deploy a Nested ESXi demo template while on site with the customer and showcase its features, without the costs of expensive hardware/datacenter cabling and gear. For a better support experience, your support staff can deploy a Nested ESXi lab environment to reproduce customer issues or resolve bugs within your software.
Virtual Education and Training
Once your developers have finished building the product, you will need a way to educate your staff on the VSphere product Family and its features. You can provide hands on virtual training environments using the same Nested ESXi template used during development. The template can be used for virtual classroom activities or labs and deployed on a per student basis. Using CloudShare’s Virtual IT lab environments, you can deploy the virtual classroom only when needed and pay only for what you use thus eliminating high costs on infrastructure. If you convert the cloud-based training for external use, you can utilize public IP addresses to enable your customers to access the course anywhere in the world, while enjoying all the VMware Features (Students can Snapshot/ revert / clone ETC)
Let’s try it ourselves:ESXi 5.x Virtual machine inside a psychical ESXi 5.x.
There are two ways to enable nested virtualization support:
1. Via the nested machine’s .vmx file:
o VSphere 5.0 = “vhv.allow = true” (or in /etc/vmware/config)
o VSphere 5.x = “vhv.enable = true”.
2. Via VCenter Web Client (the recommended way):
If you like, Optional VMware Tools for nested ESXi 5.x could be downloaded from VMware ‘flings’:
What about Performance?
Suppose you run a ‘regular’ Virtual Machine. Technically, you are running a process in the hypervisor. If you have 10 VMs in the hypervisor you have 10 processes, and so on. The more processes you have, the more time you spend on context-switching and physical processor scheduling and ready time will increase, which will result in potentially less performance.
Now imagine that one of those VM processes ‘hides’ inside 10 mores VMs (nested). Will it be slower? Yes.
Much slower? Not necessarily. It all depends on your devices and configurations.
To prove my point, I ran a small test, and the nested machine got 10-15% less ‘overall performance’ score, compared to its parent machine. My conclusion? If you don’t plan on doing production on those nested machines, the benefits far exceed the minor performance price tag.
Pre-configured ESXi Environment templates
While each configuration is likely unique, most of the time and hassle is on the front end: meaning, setting up a base image with nested virtualization. We saved all this hassle for you and created a number of nested ESXi environments for development and testing, virtual training and virtual sales purposes. These templates are a huge time saver – the environment is set up once, and once it’s saved as a blueprint, you can run tests, and save each result as it spins up and down.
The available templates include:
- VMware ESXi server 6.0
Windows Server 2008 R2 x64 with vSphere 6.0 client Unlicensed.
- ESXi 5.5.0U2 Env
VMware ESXi server 5.5.0U2
Windows Server 2008 R2 x64 with vSphere 5.5.0U2 client Unlicensed.
- ESXi 5.1 Env
This ESXi 5.1 environment contains two machines: a VMware ESXi server 5.1 & a Windows Server 2008 R2 x64 with vSphere 5.1 client. vSphere is unlicensed.
- ESXi 5.5 vCenter
Full vCenter Environment, 2 ESXi 5.5 – Unlicensed. 1 vCenter and vSphere 5.5 Client – Unlicensed. 1 Windows Active Directory 2012
The system image includes nested ESXi and VCenter, storage communication, and a top of the line Linux box with bridge controls.
How do you access these environments?
If you are not yet a CloudShare user, simply select the ESXI 6.0Env from the VM template library page, select your desired plan and… voila, your environment will be ready in a few minutes.
Or, you can create your account and select the pre-configured VM template from within the CloudShare virtual IT lab platform.
So nested virtualization? VERY COOL!
Editor’s Note – This post was originally published in and has been completely revamped and updated for accuracy and comprehensiveness.