How to Reset ESXi Root Password on a Standalone Host

If you’re reading this, I’m sorry. You’re in some trouble, aren’t you? Don’t worry! It happens to all of us. Okay – maybe not all of us – just the unlucky ones… or idiots with lack of caution.

Chill! Read this guide – you’ll be back in your ESXi host in no time. Well, maybe a few hours – no more than a day – promise 😇.

The Problem

Just to state the obvious… The problem here is that we can’t get into the root account on our ESXi host. In addition:

• We have no other administrator accounts
• The ESXi host is a standalone and is not managed through vCenter
• We have no SSH key pair with which to access the server

N.B. The host in question had ESXi 7.0.3 installed – This probably works for other versions as well. Furthermore, it seems that earlier versions of ESXi were less secure, so the option mentioned in the What doesn’t work section below could work in earlier versions.

Broadcom, you’re not helpful

The official answer from Broadcom is that…

If the host is standalone and not managed by vCenter, then re-installing ESXi is the only option.

LIEESSSSSS!

What a lame reply. I’ll admit, what I’m about to show you is a very shady way to get back into your ESXi host, hackish, to say the least – but if it works, it works.

What doesn’t work ❌

Stop! 🤚 If you’re in a rush – which I assume you are – skip this section altogether. You’re not getting into your host by reading it. It’s just me venting 😤.

There are quite a few guides on the interwebs, claiming that just like with any other Linux distro, you can

1. Create a bootable Linux ISO – Ubuntu, Arch – you name it
2. Mount the ESXi OS partition in a temporary directory – sudo mount /dev/sda5 /mnt/sda5
3. Find state.tgz and extract it – tar -xzf /mnt/sda5/state.tgz -C /tempstate
4. From the extracted contents, extract local.tgz as well – tar -xzf /tempstate/local.tgz -C /templocal
5. Edit the shadow file from the extracted local.tgz to delete the hash for the root account – vi /templocal/etc/shadow
6. Pack the shadow file in the local.tgz and then into state.tgz, move the state.tgz back into /mnt/sda5 to replace the old one
7. Unmount the ESXi OS partition, reboot
8. Voila! You got the root account with a blank password

Right? Nope ❌

This may have worked for earlier versions of ESXi, but save your time and don’t even try for ESXi 7.0.3.

Why doesn’t it work?

For starters, trying to mount the OS partition on a bootable Linux ISO will fail. This isn’t any regular ext4 partition. No, no. This is vmfs.

Okay, no worries there’s the vmfs-tools package, which can mount vmfs partitions. One problem, it doesn’t come pre-installed on your typical Linux install media. And no, I didn’t have an internet connection in my case. So I couldn’t just install it from the repos.

Doesn’t matter – I created a Linux installation media (based on Linux Mint) with vmfs-tools preinstalled. I used Cubic. I won’t get into the details, but it’s a pretty easy to use tool. Anyways, around one (or two) hours later, I got my installation media with vmfs-tools preinstalled and to no surprise – vmfs: unsupported version 6. 😡😡You wouldn’t believe it! vmfs-tools currently supports up to vmfs version 5, so if you’re trying to mount a partition formatted with vmfs 6 (which is used by newer versions of VMware ESXi), it won’t work.

No worries, I thought (being overly optimistic and all), I’ll re-create the bootable Linux media with vmfs6-tools installed. Surely, that would do it. Oh how I hoped 🙏.

How wrong I was! Reading the vmstorage and datastore partitions worked with the vmfs6-tools installed, but there was no getting to the precious state.tgz. I kept getting the error fsinfo invalid magic number 0x2fabf15f. Not only that, I read about the vmfs6-tools package and found that “Only read access is available at the moment, but write access is under works. Multiple extents are supported.” So, even if somehow I managed to access the state.tgz from my bootable Linux media, I was surely not going to replace it with the modified one.

I’ll be honest, optimistic as I usually am, this was still a battle I had considered lost. Worst part of it all? I had strung along 5 friends on this wild good chase. After looking like an absolute fool getting locked out of the ESXi host in the first place, I had wasted 6 hours of their day trying to get back in, as they offered their ideas, research, solutions, and moral support.

I won’t even get into the part where I tried to insert the raided OS disks into another ESXi host 😆.

Let’s get on to the solution…

The Solution 💡

When all hope was almost lost, we came across this gold mine of an article on Mwyann’s Weblog. Damn you Google for not putting this in my face sooner!

In a nutshell, what this fellow did, and what we copied… and essentially, what I’m showing you here is:

1. Install ESXi on another server (or virtual machine).
2. Use the ESXi installation media to boot on the original host (the one you’re locked out of).
3. Go to the BOOTBANK volume and extract the state.tgz. This is encrypted.
4. Copy the encryption.info and the local.tgz.ve files to a USB.
5. Go to the new ESXi host and extract state.tgz from the BOOTBANK.
6. Decrypt state.tgz and replace the encryption.info file with the encryption.info file from the original host.
7. Repack the state.tgz and copy it to the BOOTBANK and reboot. This will trick the new host into using the encryption key from the original host.
8. Decrypt the local.tgz.ve of the original host from the new host. This is possible since it now has the original host’s encryption key.
9. Generate a public/private key pair to SSH into the locked out server.
10. Add the public key to the authorized_keys file under the /etc/ssh/keys-root folder (from local.tgz).
11. Add /etc/rc.local.d/local.sh containing a command to start the SSH service on ESXi boot.
12. Repack the local.tgz and state.tgz and transfer to the original host via USB.
13. Connect to the original host via SSH through the private key.
14. Reset the root password 🥲.

1. Install ESXi on another server

Just create a bootable USB for this – You can use Rufus, but I personally prefer Ventoy. Note that you may need to disable secure boot on your server (or enroll the MOK to the list of trusted keys).

I had a tough time finding the ESXi ISO from Broadcom’s site. So I’ll save you the hassle – God bless the Internet Archive 💖!

2. Access the locked host through the ESXi installation media

Boot your original host (the one you’re locked out of) using the ESXi installation media you created in the previous step. At the ESXi installation screen hit Alt + F1 to enter into a shell. Enter username root and hit enter when prompted for the password. It’s blank.

When you list the volumes (ls /vmfs/volumes) you’ll see your BOOTBANK1, BOOTBANK2, OSDATA_xxxxx, datastore1 – plus some additional volumes you may have created. That’s right you can view the volumes from the old host when you boot on it with an ESXi installation media.

In our case, we only had to work on BOOTBANK1, but you might need to repeat this procedure on both bootbanks if they were both being used in your case. For simplicity, I will continue the rest of the guide with BOOTBANK1.

3. Extract state.tgz from the locked host

mkdir /mylosthost
tar xzf /vmfs/volumes/BOOTBANK1/state.tgz -C /mylosthost
ls /mylosthost

You’ll see that when extracting the state.tgz from the BOOTBANK we don’t get the expected local.tgz, but rather an encrypted local.tgz.ve together with an encryption.info file.

You would typically decrypt the local.tgz.ve using the below command, but since we’re booted from the ESXi installation media, we won’t be able to decrypt it.

crypto-util envelope extract --aad ESXConfiguration local.tgz.ve local.tgz

So, the solution is to trick the new host to use the encryption.info from the locked host such that it’s able to decrypt the local.tgz.ve file.

4. Copy local.tgz.ve and encryption.info to USB

Okay, so insert a USB drive into the locked out host. Format it as FAT32 not NTFS – ESXi is more friendly with FAT. You should be able to see it in the list of volumes (ls /vmfs/volumes).

Copy the encryption.info and local.tgz.ve files to the USB drive. Remember we have them in a temporary directory we created /mylocalhost.

cp -r /mylosthost /vmfs/volumes/MYUSBDRIVE/mylosthost

5. Extract state.tgz from the new ESXi host

cp -r /vmfs/volumes/MYUSBDRIVE/mylosthost /vmfs/volumes/BOOTBANK1/mylosthost

Boot up the new ESXi host and hit F2 to customize the system. Enter the password you set during the installation and then go to Troubleshooting Options. From here, you can enable shell access.

Then, hit Alt + F1 to open up a shell. Logon with the root account and navigate to the bootbank via either /vmfs/volumes/BOOTBANK1 or /bootbank. We need to extract its state.tgz file.

mkdir /mynewhost
tar xzf /vmfs/volumes/BOOTBANK1/state.tgz -C /mynewhost
ls /mynewhost

You see the encryption.info and local.tgz.ve files. Differently from step 3, we can decrypt the local.tgz.ve file because we’re logged in to our host properly (not from the bootable installation media).

6. Decrypt state.tgz and replace encryption.info

Okay, now decrypt the local.tgz.ve file to get the local.tgz file. Then, delete the local.tgz.ve file and encryption.info files and copy the encryption.info file from the locked host (remember that we transferred it to this host via USB in the previous bonus step).

cd /mynewhost
crypto-util envelope extract --aad ESXConfiguration local.tgz.ve local.tgz
rm -f local.tgz.ve
rm -f encryption.info
cp /vmfs/volumes/BOOTBANK1/mylosthost/encryption.info /mynewhost/encryption.info
ls /mynewhost

In the /mynewhost directory you should now have a local.tgz file (from the new host) and the encryption.info file (from the old host).

7. Trick the new host to use the locked host’s encryption key

This part is very important, so please pay close attention to the next steps.

At this step of the process, we’re booted on the new host. The next step is to trick it to use the locked host’s encryption key, but we cannot do this while booted into it, we have to be booted via the ESXi installation media.

Trust me, don’t try to do it, because the state will be overwritten on reboot and it won’t work.

But if we shut off now, we won’t be able to access the /mynewhost directory from the ESXi installation media. So, let’s copy this folder to the BOOTBANK1 volume which will still be accessible while booted from the installation media.

cp -r /mynewhost /vmfs/volumes/BOOTBANK1/mynewhost

Okay, now we’ll restart the new host, boot up from the ESXi installation media, and open up a shell with Alt + F1. As before, we’ll login with username root and a blank password.

As you recall from the previous step, in the /vmfs/volumes/BOOTBANK1/mynewhost directory you should now have a local.tgz file (from the new host) and the encryption.info file (from the old host).

Let’s pack them back into state.tgz and replace the new host’s state.tgz with this new updated state (that uses the original host’s encryption.info file)

rm -f /vmfs/volumes/BOOTBANK1/state.tgz
cd /vmfs/volumes/BOOTBANK1/mynewhost
tar czf /vmfs/volumes/BOOTBANK1/state.tgz encryption.info local.tgz

Now that the state.tgz of the new host has been replaced, we can reboot into the new ESXi. Now the new host is using the original host’s encryption key so we’ll be able to decrypt the local.tgz.ve file that we copied over from the locked host via USB.

8. Decrypt the locked host’s local.tgz.ve

Back into the shell of the new host, we can navigate to the directory in which we copied the locked host’s local.tgz.ve and decrypt it. This time, it will work because we are using the old host’s encryption key on the new host.

cd /vmfs/volumes/BOOTBANK1/mylosthost
crypto-util envelope extract --aad ESXConfiguration local.tgz.ve local.tgz
tar xzf local.tgz
ls

Awesome! You can see that we extracted the etc directory (and some other directories) from the locked host’s local.tgz. Unfortunately, the shadow file is nowhere to be found in the etc directory, so this won’t be as simple as deleting the root account’s hash from the shadow file.

What we’ll do instead is create a public/private key pair and add the public key to the authorized_keys in /etc/ssh/keys-root. This will enable us to SSH into the locked host.

9. Generate public/private key pair for SSH

On a separate Linux machine generate the SSH key pair using ssh-keygen.

ssh-keygen -t rsa -b 4096

Optionally, set a passphrase on your private key… Make sure you don’t forget this one 😜.

I also renamed the public key from mypublickey.pub to authorized_keys. This made the next step easier for me, because I could copy the file directly to /etc/ssh/keys-root (that will be packed to the state of the locked host). So your authorized_keys file should look as follows with a single public key (the one you just generated). Make sure to update the user at the end of the public key to root@thenameofyourlockedhost.

10. Add the authorized_keys file to /etc/ssh/keys-root

Copy the authorized_keys file to your FAT32 formatted USB drive and then transfer it to the new host. You might need to boot from the ESXi installation media if the USB is not visible in /vmfs/volumes.

From the USB drive, we need to copy the authorized_keys to the /etc/ssh/keys-root directory. In my case I needed to create the keys-root directory because it was not present in the /etc/ssh folder.

mkdir /vmfs/volumes/BOOTBANK1/mylosthost/etc/ssh/keys-root
cp /vmfs/volumes/MYUSBDRIVE/authorized_keys /vmfs/volumes/BOOTBANK1/mylosthost/etc/ssh/keys-root/authorized_keys

11. Add /etc/rc.local.d/local.sh to start SSH service on ESXi boot

This step could be optional for you, but if you’re not sure, follow along because it’s a small step relative to the entire process.

For added security, we prefer to keep the SSH service switched off on our ESXi hosts. With SSH disabled, it won’t be possible to access the locked host via the SSH private key we generated earlier. So what we need to do is instruct the locked host to start the SSH service as it boots up. We do this by modifying the local.sh and packing it into the state.tgz that we’ll transfer to the locked host.

So, on the new host:

mkdir /vmfs/volumes/BOOTBANK1/mylosthost/etc/rc.local.d
vi /vmfs/volumes/BOOTBANK1/mylosthost/etc/rc.local.d/local.sh

Your local.sh should look as follows:

#!/bin/sh

/etc/init.d/SSH start

exit 0

12. Transfer the modified state to the locked host

Now, that we have our public key in the authorized_keys file and the /etc/init.d/SSH start command in the local.sh file, we can repack everything and transfer the modified state to the locked host.

In my case, in addition to /etc, I had another 2 directories that were extracted from local.tgz. I made sure to repack them as well in the final tgz file, although I’m not sure if it makes a difference.

cd /vmfs/volumes/BOOTBANK1/mylosthost
tar czf /vmfs/volumes/MYUSBDRIVE/local.tgz etc dir1 dir2
tar czf /vmfs/volumes/MYUSBDRIVE/state.tgz /vmfs/volumes/MYUSBDRIVE/local.tgz

With the updated state in the USB drive, we need to reboot the locked host one last time from the ESXi installation media. Enter a shell and transfer state.tgz to the BOOTBANK.

rm -f /vmfs/volumes/BOOTBANK1/state.tgz
cp /vmfs/volumes/MYUSBDRIVE/state.tgz /vmfs/volumes/BOOTBANK1/state.tgz

Okay, we updated the state of the locked host. Time for a reboot… Hold your breath!

Remember – you may need to repeat the above steps for both BOOTBANK1 & BOOTBANK2 if they were both being used by your locked ESXi host. If your BOOTBANK2 only contains a boot.cfg file, you’re in luck it’s not being used by your locked host.

13. SSH into the locked host using the private key

If all went well, your locked host should be up and running after the reboot. The SSH service should be started as well. So, the next step is to connect to it using the private key.

ssh -i myprivatekey root@mylockedhost

At long last, you should now be logged on to your ESXi host with the root account.

You may stop holding your breath 😁.

14. Reset root password

You’ve gotten this far. One final step is to reset the root password – don’t lose it this time!

passwd

That’s All Folks

I hope this guide has been helpful to get you back into your ESXi host. If you’re still clueless, or need help with some of the steps, feel free to reach out. I’ll be happy to help.