This article is written by Ernst Cozijnsen.
A few days back after applying some patches via "Update Manager" we came to notice that our ESXi 4.1 host bacame unresponsive after a reboot. Loggin into the command prompt via ILO was unsuccessful too because the rootpassword was not working anymore. That just sucks!.... but nothing to get all emotional about since is just linux ;-) (well a sort of...)
Looking for a nice how-to explaining this i stumbled across the following article: http://www.vm-help.com/esx/esx3i/Reset_root_password.php
After doing the endless tar commands described in the article I rebooted the machines and what the ....... still not working. (And this is where I got frustrated since this is the only decent way to reset a security feature like this).
VMware states that it is impossible to reset a root password of an ESXi server in a supported way. Well.... let's see if we can bend that opinion.
Since VMware was so nice to give us "host profiling" I thought this was the time to look into it and see if we can use it in an alternative way. Chaning a root password without root permissions would normally be a serious security leak. OK, so the assumption is that the host you are having issues with was connected to vCenter and the account you are using has admin rights.
Execute teh following steps:
- Take a "similair" server and create a host profile from this machine
- Edit the Host profile and change the "Administrator password" to a fixed one

- Now "Attach" and "Apply" the modified profile to the host that is troubling you
Host profiles were initially intended for embedded hosts to read their config at boot time since they were not able to store changes made to the ramiefied filesystem.
Not totally true because installing drivers / patches require commiting changes to disk.
Anyway.
- Since the "Host Profile" now temporarily changed the root password you are now able to login via the console or SSH (if configured). But this change is only temporary and if you would remove the host profile and/or the host itself from VC you would end up with the same problem as before.
HOW-To make these changes permanent:
- Execute a manual "passwd" command and set the root password to a known/desired one
- Execute the "/sbin/auto-backup.sh" script to permanently write the changes to the state.tgz file which is located in the primary "Bootbank"
- Reboot.... and enjoy the wonder ;-)
It's great, it works............ but is it logical?
If i look around on the "Console" I cannot find a user that has the rights to modify the root password in any way.

- There is no user in the group root
- There is no user that has uid 0 except root
- The "Set UID Root" bit is not set on the passwd command or on the busybox binary
- The vpxuser account shouldn't have enough right to do such a modification
From a UNIX perspective no unprivileged user should be able to change a root password. If its possible, its a security breach!
The weird thing is the "Sticky Bit" on the Shadow file. This is pretty exotic since the "Sticky Bit" on files is ignored by Linux security, See http://en.wikipedia.org/wiki/Sticky_bit for more info on what it does.
So what do we see here?