Install ESXi in a VirtualBox VM

Created on January 22, 2023 at 4:53 pm

Category: Programming

VirtualBox provides an effective and free solution for running ESXi in a lab environment. However, at this point, the support for nested virtualization doesn’t allow the installation of current versions of VMware’s hypervisor. As with Hyper-V VMs, a nested configuration is limited to ESXi 6.x. If you need to enable Microsoft’s hypervisor on a Windows PC, VirtualBox can run on top of it.

When I attempted to install ESXi 7.0 and ESXi 8.0, I received various purple screen of death (PSOD) messages.

AP 1 is incompatible with BSP; cannot start

In addition, when I did get the ESXi 7.0 installation to work, there were network issues with a lack of network adapter drivers.

I decided to install VMware ESXi 6.7 in VirtualBox for various testing purposes, as it does not appear to have the issues associated with the newer releases.

Create the ESXi-VM in VirtualBox

Once you have VirtualBox installed and configured on your host system, the first step is to create the virtual machine that will be used for the ESXi installation. Click the New button in the VirtualBox GUI.

This launches the Create Virtual Machine wizard. On the first screen, you need to name the virtual machine, choose the storage location for the VM files, and point it to an ISO file for installation. I downloaded the ESXi 6.7 installation ISO and configured the path in the ISO image configuration. Choose Linux 2.6 / 3.x / 4.x / 5.x (64-bit) for the version.

On the Hardware configuration screen, set the values for RAM and CPU. When configuring your virtual hard disk, you might factor in the storage space demand for VMs you run on top of ESXi.

Finally, you will see the Summary screen. Click Finish.

Changing the settings of the VirtualBox VM

After we create the virtual machine that will house the ESXi installation, we need to make a few changes to its settings. Under System > Motherboard, select the following:

  • Chipset: ICH9
  • Pointing Device: PS/2 Mouse

On the Processor tab, enable the Extended Features > Enable PAE/NX and Nested VT-x/AMD-V settings.

For the Network configuration, select the Host-only Adapter. For Promiscuous Mode, select Allow All.

Install ESXi

The final step is installing VMware ESXi. After updating the settings, we power on the ESXi VM in VirtualBox.

This will launch the familiar installer that guides you through all steps of setting up VMware’s hypervisor.

If you receive the warning below, you need to make sure you have set the Enable Nested VT-x/AMD-V setting in the properties of your VirtualBox VM. The installer will proceed with the warning. However, this may prevent you from running nested virtual machines on the ESXi installation.

Hyper-V-enabled VirtualBox hosts

VirtualBox virtual machines can run in parallel with Hyper-V with a few settings changes. Keep in mind that if you are running modern security technologies such as virtualization-based security (VBS), they use Hyper-V as the underlying virtualization technology.

To run VirtualBox with Hyper-V technologies enabled, you must add the Virtual Machine Platform optional component.

Then, in VirtualBox, you can select Hyper-V for the Paravirtualization Interface.

Wrapping up

VirtualBox has long been a popular enthusiast virtualization platform used for running various operating systems on top of desktop operating systems. It is free to download and provides various installation capabilities and configuration features.

As shown, it is a viable option to install and play around with VMware ESXi. However, there appear to be incompatibilities with ESXi 7 and higher, both from a CPU and network perspective. In my testing, installing these versions of ESXi ended with mixed results.

There may be a workaround for this issue or an ESXi kernel option that allows bypassing a check initiated with newer ESXi installations to prevent PSODs. It may also be possible to slipstream additional network drivers to work with VirtualBox.

Connecting to Connected... Page load complete