Deleting Orphaned (AKA Zombie) VMDK Files
A problem that most Virtualization Administrators need to deal with is Orphaned VMDK files (or Zombie VMDK files, as some people call them). These are Virtual Machine hard drive files that are just hanging out, taking up storage, but are attached to no VMs. How can this happen? Well, the most common way is when an administrator needs to delete a VMDK but isn't quite ready to commit... so, after pressing "remove", they select "remove from virtual machine" rather than "remove and delete files". The intent is always good: "I want to be able to restore this if it turns out to be necessary... so I'll just come back and delete it next week if nothing breaks!" But, of course, next week something else happens and the file doesn't get deleted. After a little while, it turns into "which files did I rename, again?" and nothing seems to happen.
So, how do you deal with these orphaned files? Well, a few years ago I found a script to locate orphaned files, but I've been told it doesn't always work on newer platforms. Around the same time that I learnt of that problem, a friend introduced me to RVTools, which is basically a GUI for pulling all of the delicious data from vCenter that is hard to get at through the normal GUI. Ever wanted to know what the Max EVC vs. the Current EVC Setting is for every host in your environment? Just check the vHost tab. How about a nice easy way to check the Load Balancing Policy for every Distributed vSwitch Port Group in the environment? Check the dvPort tab! It also has an excellent "vHealth" tab, that highlights particular issues that you should probably be aware of... including orphaned VMDK files! Well, it calls them Zombie VMDK files, but that's a lot more fun, so I can't really begrudge them that. In fact, I think that the rest of this post is going to be about killing zombie VMDKs before they can overrun your SAN, cause why not?
So, all of that was a long winded way of saying that you know you've got a zombie problem, but you need an easy way to quantify it and identify those zombies, so that you can take care of them. As I said earlier, fire up RVTools and move over to the vHealth tab. You'll notice that there's probably a lot of stuff in there that you're going to want to deal with, but for now, stay focused. You've got a zombie problem after all. Go to File and select Export, then save the contents as a CSV file somewhere. Now, you know exactly where the zombies are lurking, but who wants to go toe-to-toe with a horde of zombies? You're gonna need a tool, and I've got just the thing: ThisIsMyBoomstick.ps1.
When dealing with Zombie VMDK files, I like to take a cautious approach. After all, it's a high stakes game and you don't want to come out as the loser. Fortunately, the ESXi host command line tool, vmkfstools, is really user friendly in this situation. I say that because it's just smart enough to stop you from doing something really stupid... i.e. it respects file locks. So, my process is to rename the zombie VMDK first, then delete it. I like this process because, if the VMDK is in use, vmkfstools will fail to rename the file. If the VMDK is associated with a powered off VM, it will rename the file but it won't "fix" the VMX to use that new filename, and so will break that VM. In this situation, that's a good thing, because it lets you know that that VMDK file is necessary (and if the VM was powered off anyway, you've probably got enough time for some basic troubleshooting when it does need to get powered on). So, after the VMDK files have been successfully renamed for a while (I usually leave them in place for a couple of weeks), you can then go through and delete them.
So, that's what this script does. It parses through the RVTools output, looking for 'zombie' lines. It then generates 2 sets of vmkfstools commands. The first renames all of those suspected zombie files by prefacing the filenames with "Unused-". The second set of vmkfstools commands deletes all of those renamed files. When I'm ready to use this, I SSH to an ESXi host and then fire off the (vetted) rename commands and I create a calendar reminder a few weeks out. I put the delete commands in my calendar reminder and, after I'm sure that none of those renamed files are required, I SSH back into an ESXi host and fire off the delete commands.
As with any scripts you find on the internet (especially ones related to deleting VMDK files!), read through them carefully and make sure that you understand them before executing them. Also, doublecheck the list of files that it wants to delete, just to make sure that there are no false positives.
So, how do you deal with these orphaned files? Well, a few years ago I found a script to locate orphaned files, but I've been told it doesn't always work on newer platforms. Around the same time that I learnt of that problem, a friend introduced me to RVTools, which is basically a GUI for pulling all of the delicious data from vCenter that is hard to get at through the normal GUI. Ever wanted to know what the Max EVC vs. the Current EVC Setting is for every host in your environment? Just check the vHost tab. How about a nice easy way to check the Load Balancing Policy for every Distributed vSwitch Port Group in the environment? Check the dvPort tab! It also has an excellent "vHealth" tab, that highlights particular issues that you should probably be aware of... including orphaned VMDK files! Well, it calls them Zombie VMDK files, but that's a lot more fun, so I can't really begrudge them that. In fact, I think that the rest of this post is going to be about killing zombie VMDKs before they can overrun your SAN, cause why not?
So, all of that was a long winded way of saying that you know you've got a zombie problem, but you need an easy way to quantify it and identify those zombies, so that you can take care of them. As I said earlier, fire up RVTools and move over to the vHealth tab. You'll notice that there's probably a lot of stuff in there that you're going to want to deal with, but for now, stay focused. You've got a zombie problem after all. Go to File and select Export, then save the contents as a CSV file somewhere. Now, you know exactly where the zombies are lurking, but who wants to go toe-to-toe with a horde of zombies? You're gonna need a tool, and I've got just the thing: ThisIsMyBoomstick.ps1.
When dealing with Zombie VMDK files, I like to take a cautious approach. After all, it's a high stakes game and you don't want to come out as the loser. Fortunately, the ESXi host command line tool, vmkfstools, is really user friendly in this situation. I say that because it's just smart enough to stop you from doing something really stupid... i.e. it respects file locks. So, my process is to rename the zombie VMDK first, then delete it. I like this process because, if the VMDK is in use, vmkfstools will fail to rename the file. If the VMDK is associated with a powered off VM, it will rename the file but it won't "fix" the VMX to use that new filename, and so will break that VM. In this situation, that's a good thing, because it lets you know that that VMDK file is necessary (and if the VM was powered off anyway, you've probably got enough time for some basic troubleshooting when it does need to get powered on). So, after the VMDK files have been successfully renamed for a while (I usually leave them in place for a couple of weeks), you can then go through and delete them.
So, that's what this script does. It parses through the RVTools output, looking for 'zombie' lines. It then generates 2 sets of vmkfstools commands. The first renames all of those suspected zombie files by prefacing the filenames with "Unused-". The second set of vmkfstools commands deletes all of those renamed files. When I'm ready to use this, I SSH to an ESXi host and then fire off the (vetted) rename commands and I create a calendar reminder a few weeks out. I put the delete commands in my calendar reminder and, after I'm sure that none of those renamed files are required, I SSH back into an ESXi host and fire off the delete commands.
As with any scripts you find on the internet (especially ones related to deleting VMDK files!), read through them carefully and make sure that you understand them before executing them. Also, doublecheck the list of files that it wants to delete, just to make sure that there are no false positives.
This comment has been removed by the author.
ReplyDeleteI have not got to use your script and so I wrote a version of zombie removal using csv file
ReplyDeleteExample csv:
DSname;Path
datastore;folder
My script:
$folders = Import-Csv -Delimiter ";" -Path C:\temp\ZombieVM.csv
$i=0
foreach ($ds in $folders.dsname)
{
$DSuid = Get-Datastore $ds
New-PSDrive -Location $DSuid -Name ds -PSProvider VimDatastore -Root ""
Set-Location ds:
Write-Host $folders.Path[$i]
#Remove-Item $folder.Path -Recurse
$i=$i+1
cd c:\
Remove-PSDrive ds
}
I commented out for the safety of the removal line
For use, need remove # before Remove-Item
Update:
Deleteuse this line
#Remove-Item $folders.Path[$i] -Recurse
#Convert the bracketed datastore syntax to a normal folder structure
ReplyDeleteCan you give me an example of this?
I got this error is why I'm asking.
DeleteCannot index into a null array.
At C:\Scripts\VMware orphaned files\ThisIsMyBoomstick.ps1:38 char:9
+ if ($_[ <<<< 0] -eq "[")
+ CategoryInfo : InvalidOperation: (0:Int32) [], RuntimeException
+ FullyQualifiedErrorId : NullArray
A NullArray error there might mean that $zombies is empty. Print it out before that line and take a look. Also, does your code actually have all of those "<<<<" inside the brackets?
DeleteNo and I just noticed that. The error is adding them. I'm going by the script which has
Deleteif ($_[0] -eq "[")
That comment isn't an instruction, it's a note about what that next bit of code is doing. That bit is doing some string manipulation to convert from [Datastore1] VM\vm.vmdk into a format that's easier to work with for vmfstools.
ReplyDelete