2009/05/12

XenServer VM won't shutdown


I had a windows x64 vm domU that was locked up. It would not shut down through XenCenter. It would not shut down through the command line with "xe vm-shutdown vm=" or "xe vm-shutdown --force vm=".

In the logs were these mesages:
VM.hard_shutdown R:82ca53505e13|xapi] VM.hard_shutdown locking failed: caught transient failure OTHER_OPERATION_IN_PROGRESS: [ VM; OpaqueRef:6c1f16b5-7a80-c3fa-eb07-6b605d1fa305 ]
[20090512 08:43:59.181|debug|vserve0|166 unix-RPC|VM.hard_shutdown R:82ca53505e13|xapi] Waiting for 12.745108 seconds before retrying...

I could not find any information on what "OTHER_OPERATION_IN_PROGRESS" meant (other than the obvious), or how to address it.

A host reboot also did not work: the xen host would not shut down gracefully, because it could not terminate this vm -- it would just hang at "terminating remaining VM's".

Finally, too late, I found the answer in the Citrix forums:

To know the cause of the rejection, you can try running xe task-list and see if anything is in pending state that might be related to the command failure.

If you do see anything that is likely to be in the way, try removing the task with xe task-cancel uuid= TASK-UUID, then try the shutdown operation again.


You may have to do a
xe-toolstack-restart.

5 comments:

dafella said...

We ran xe-toolstack-restart
The services stopped ok, but did not restart.
BANG!!

Error - deamon missing

We had to shut down all VM's and restart the Host

All up again, but the downtime was not welcomed. SLAP!!

DeBaan said...

I've since found that the most common reason my VM's won't shut down is that they're using a storage repository (such as an ISO library) that is offline.

In other words, if the virtual cd of a VM is pointing at an ISO that's on an offline storage repository, then that VM can't be shut down! (bug)

Lane said...

Have also found this problem also to be caused by...

The behavior was, I clicked “start”, and it went amber in the pool, but never actually started running on one of the servers; once I sent it a “start”, I could not send it any more signals (start, stop, force reboot, force shutdown); nor could I snapshot it nor clone it. Figured out that any disk-related operations would hang or fail.

Problem was, somehow the storage repository NFS share on the NetApp had been made non-writeable by root. Thus, all write requests from the XenServer were rejected.

Solution: export it read-write and root-modifiable (browse as domain admin to \\ausnafs04\vol0\etc\exports and modify it in wordpad), then ssh in to fs04 and “exportfs -a”

I suspect someone modified the run-time exports and forgot to save it out to the exports file. So, when I did "exportfs -a" to export another filesystem, the export parameters for this SR share were changed (to non-root r/w).

Anonymous said...

I have same issue, i find instructions on how to fix here on this tech site -sysdigg

Anonymous said...

Thanks, after a few tries with the tasks renewing itself I finally managed to cancel them and kill the VM