vSphere/ESXi: When good tasks go bad – Fix with ESXCLI via PowerCLI

Most of the time, tasks we start in vSphere just work. That’s why we love it. Queue ’em up, and move onto something else. Sometimes though, they go bad. They freeze, stick, die or just plain error out. This happened to me last week.

After changing the scratch location for hosts in a dev cluster, I planned to reboot them for the change to ‘take’.  This was late afternoon on a Thursday so I kicked it off before I went home for the day. Returning in the morning, I was surprised to see the ‘Enter Maintenance Mode’ task sitting @ 2 % with a sub-task migrating the final VM from the host stuck on 67%.

There were also some SCOM alerts showing this particular vm dropped comms not long  after the migration began.

Cancelling the task in client made no difference, nor did stop-task

17-03-2015 9-55-34 PM

 

 

 

 

 

 

17-03-2015 9-57-48 PM

This usually happy little vm was becoming rather obstreperous !

So how to fix ? First verify which host it’s on;

17-03-2015 10-58-12 PM

 

 

Check the VM.process state with esxcli via PowerCLI and it’s not there ?? Plot=thickening  **

Check another host, and it’s there. This was obviously the Target of the migration task. Plot= thicker still

17-03-2015 11-00-57 PM

I thought I’d try and kill it on the target host, but first need to find the WorldId, by matching it to the DisplayName in the process.list output.

…then kill that thang, firstly using the ‘soft’ parameter. (if this fails you escalate to hard, then force)

Did it work ? Check the task status, and then the VM status

17-03-2015 11-22-33 PM

 

 

Groovy, back in business and the vm can be powered on.

Neeeeeext.


** To use ESXCLI in a POWERCli context, the host(s) need to be stored in a variable(s). In my example i’m using $esxcli and $esxcli3

To declare; (substituting your own hosts fqdn)

18-03-2015 7-11-17 PM

to check what ESXCli commands/namespaces are available;

$esxcli

18-03-2015 7-11-01 PM

 

 

 

 

 

 

 

 

Leave a Reply

Your email address will not be published. Required fields are marked *

%d bloggers like this: