Well this is interesting. First of all, do not move any vhd or avhd files around, whether your guest VM is running or not.

I came back from a week’s vacation to find that my VMs were pretty much all broken. Awesomesauce. What happened was that the server that I run SCVMM on is also the Backup Exec server, and due to a mistake by some end-user, the size of the weekly backup jumped about 600gb and the backup2disk folder ran out of space and halted all backups. All the Virtual Machines paused themselves too because the host was out of hard drive space.

To alleviate the situation, a co-worker found 100gb or so of files in a “snapshot” folder under the VM’s folder and moved them elsewhere. What he didn’t know or realize was that these VM files have very specific ACLs that are tied to a username called NT Virtual Machine\{SID}.

When you move a file in Windows, if you’re copying on the same volume (say from My Pictures to My Pictures\vacation 2011) it will take it’s permissions with it. When you move a file to a different volume (to a D drive, or a flash drive or a network drive) it will inherit the permissions of it’s new home. Normally that’s a good thing, but for these snapshot files, it’s a bad thing. a very bad thing.

I discovered this when I found & moved the files back to where they were. The VM still would not start up and was giving all kinds of cryptic errors. unable to mount, unable to start virtual controller, things like that. I should have made a note of the exact errors and put them here for people to find, because figuring out what to do was a bit of a pain. Ultimately I found a KB article that described how to re-set the permissions and re-assign full control to the NT Virtual Machine\GUID user to the folder and then each of the avhd files directly using your favorite tool and mine: icacls.exe

This allowed the machine to re-start up and everything seemed to be OK so after 24 hours I thought I’d figure out how to get rid of those snapshot files and free up that space “the right way”. The first problem was that I did not have any snapshots of this VM, so how could I have snapshot files??

I found this article called “Hyper-V: What are these *.avhd files for? Snapshots? But I have no snapshots!” while Googling around and at first was stumped, because what he was displaying I could not see. I followed his directions to shut down the VM and power it off (the guest) and realized that yes it had been paused and rebooted, but it had never been shut down in nearly two years. I powered it off (it’s an MDT and WSUS server, so no “production” data on it) and looked around for the “merging 1%” to show up and it didn’t. I couldn’t figure it out! why couldn’t I see this happening in my SCVMM administrator’s console? On a whim, I decided to try the “local” Hyper-V MMC snap-in, so I fired up the Server Manager and drilled down to it. There it was, on the main screen under “Operations”: Merge in progress: 11%

I watched it for a few minutes and saw that one of the AVHD files disapeared! it was working! Awesome! so now it’s merging “the big file” which is where all the deployment images and WSUS download data was and is taking a while longer. As soon as the first AVHD file disapeared, I looked at the drives and saw that there was now 80GB free and the backup jobs resumed their steady march.

Once this is done, I’m going to have to do the same to the other Guest VM on this machine, which IS a production machine and probably has even more data in it, so that will have to wait for 5pm and run overnight.