Invalid Configuration for Device '0' Error after Server Crash

One of my customers recently had a major issue; due to a fault in their cooling system, their server room overheated and many devices turned themselves off in order to avoid damage... including one of their SANs.  After resolving the cooling issue, the environment largely came back up without serious incident, however there was one lingering problem.

Several VMs in the environment could not connect to their distributed virtual switch.  Each VM was associated with its port group correctly, however the "connected" checkbox was cleared.  When we tried enabling that option, the task failed with an error that read: Invalid Configuration for Device '0'

Some quick googling revealed a VMware KB Article about this particular issue, with several different workarounds suggested.  They ranged from easy (move the VM to a different Port Group and back) to arduous and time consuming (remove the NIC from the VM, then add a new NIC and reassign its network identity to the new NIC).  We found that the easy options were generally not working; the option that the customer had settled on was NIC removal and re-adding.

In the middle of that KB article, it mentions another option - use the CLI on the ESXi host to force a VMX reload.  This customer has quite a few ESXi hosts though, and so this option also struck me as being more labor intensive than it needed to be.  Once again, I hit Google and found a way to do that same VM reload from PowerCLI, which connects to vCenter and will allow you to target every needed VM from a single interface.

That article discusses finding machines that were in a disconnected state; that wasn't the symptom that we'd experienced.  Instead, we had a list of VMs that were not pinging even though they were powered up, so I reduced his PowerCLI line to the following (for a server named "Server-Name"):

(get-view -viewtype VirtualMachine | ?{$_.name -like "Server-Name"}).reload()

While doing so, I played around with it a little bit and learned something interesting.  This command works because the VM object that is returned by the Get-View command has a method called .reload() which forces vCenter to reload the VM's vmx file.  The VM object that is returned by the normal Get-VM command does not have this method, so if you're ever working with a similar situation, make sure that you use the "Get-View -viewtype VirtualMachine" technique.

Once we ran that command on each VM, we were able to reconnect their network adapters successfully and services were restored.

Comments

Post a Comment

Sorry guys, I've been getting a lot of spam recently, so I've had to turn on comment moderation. I'll do my best to moderate them swiftly after they're submitted,

Popular posts from this blog

Clone a Standard vSwitch from one ESXi Host to Another

One of my customers is standardizing the configurations on their HP C7000 enclosures (they've been set up at various sites by various administrators with varying involvement from the architecture team).  As such, we need to stand up some temporary resources so that we can take down the main enclosure for reconfiguration.  That's fine, we can easily ship a smaller enclosure to each site for temporary compute resources.  Some of the sites are using Standard vSwitches, so we need to be able to quickly copy the networking configuration over from their existing blades to these new, slightly different blades.  As I see it, we had 2 good options: 1) Capture a Host Profile with the desired vSwitch configuration.  Delete every other component of the Host Profile so that only the networking section is applied; design the new ESXi hosts so that the vSwitches'll work with the old vmnic-to-vswitch configurations. 2) Write a script to clone the vSwitch from ones ESXi host to another.
Read more

PowerShell Sorting by Multiple Columns

PowerShell's Sort-Object cmdlet is super useful, especially when preparing output for human consumption.  A few people have found my blog while looking for more information about its use, specifically while looking for how to sort by multiple columns (well, properties, technically).  I've never done so (much less written about it), so I hope those folks found answers elsewhere.  But, it got me curious... and it turns out that it's really easy. The Technet article on Sort-Object has the answer directly spelled out: just use commas to create a list of properties to sort by (in order to precedence).  Let's look at some examples!  First, prepare a variable with some good sortable data: $myData = get-eventlog System -newest 25 And then we can get to sorting!  Say you want to sort primarily by EntryType (Warning, Information, Error) and then by Source.  Easy-peasy: $myData | sort EntryType,Source How about if you want it to be in descending order?  Yeah, there&
Read more

Deleting Orphaned (AKA Zombie) VMDK Files

A problem that most Virtualization Administrators need to deal with is Orphaned VMDK files (or Zombie VMDK files, as some people call them).  These are Virtual Machine hard drive files that are just hanging out, taking up storage, but are attached to no VMs.  How can this happen?  Well, the most common way is when an administrator needs to delete a VMDK but isn't quite ready to commit... so, after pressing "remove", they select "remove from virtual machine" rather than "remove and delete files".  The intent is always good: "I want to be able to restore this if it turns out to be necessary... so I'll just come back and delete it next week if nothing breaks!"  But, of course, next week something else happens and the file doesn't get deleted.  After a little while, it turns into "which files did I rename, again?" and nothing seems to happen. So, how do you deal with these orphaned files?  Well, a few years ago I found a scrip
Read more