Virtually Jason

Towards the start of the year, I made a post about my vCenter to vCenter migration process , which can be used to update from vCenter 5.5 to vCenter 6 (or for that matter, to downgrade, or whatever).  Since then, I've done a lot of these migrations and the scripts/procedure have matured.   Jeff and I have fixed many bugs, but the reason that I'm posting about this (again) is that I recently added a new feature and have polished our migration process. The latest versions of the migration scripts (the ones that get the settings from the source vCenter and then recreate those settings on the destination vCenter) now support DRS Rules!  The native support for DRS Rule manipulation through PowerCLI is a bit lacking... but fortunately, the community has solved that problem!   Matt Boren and Luc Dekens have created a module called DRSRule that is built to help with the reading and manipulation of DRS Rules (pretty self explanatory, right?).  I took advantage of t...

9/15/2016 Update:  I've updated these scripts with some new functionality.  I just posted a description of the new functionality as well as how to use the scripts in the context of a vCenter Upgrade . A couple years ago, I made a blog post about a process that I use to migrate from one vCenter to another , and then I later created some scripts  to facilitate that process.  Those scripts worked fine for me, but others have found some weaknesses.  Namely, they don't work if there are multiple folders in the inventory with the same name (such as VM\Project1\Production and VM\Project2\Production, where there are two folders named Production).  Also, I never thought about VMs in vApps, which don't technically have a parent folder. So, I've been working on updated scripts.  I'm a little hesitant to post these, as I haven't actually used them to perform a migration yet, but there seems to be a lot of people who need them.  I've put together two scrip...

One of my customers has many physical datacenters, each with its own vCenter server.  The servers are all connected via Linked Mode, which works fine in the GUI, but at the CLI I don't usually want to connect to every one of them.  Fortunately, connect-viserver doesn't do so unless you use the -AllLinked switch, so that's fine.  Unfortunately, I'm left in a position where I need to connect my PowerCLI session to any one of many vCenter servers, and I don't necessarily like typing that much or remembering that many server names. Fortunately, I can address both of those points in one fell swoop, through PowerShell Profiles .  When you launch PowerShell, it simply applies the appropriate PowerShell Profile (aka. a script that you can use to customize your work environment) to that session and you're good to go.  So, I put together a customized PowerShell Profile for myself that presents me with a "Connect to vCenter" option and a little text driven men...

We’ve all probably been in this situation: vCenter has experienced a catastrophic failure and isn’t even responding to ping, so you need to get console access in order to resolve the issue.  The challenge is that the VM could be running on any of 16 ESXi hosts in your cluster.  Generally, I like to create DRS Host Affinity rules for the vCenter and its SQL server so that I know which host those VMs should be running on during normal operations, but if HA has been busy they could still be anywhere.  That means that you need to find those VMs in order to troubleshoot, which could take a while.  Instead of opening 16 vSphere Client instances, each connected directly to an ESXi host in your environment, there’s a much easier way to find those VMs: PowerCLI! PowerCLI can query a whole array of ESXi servers fairly quickly, returning the host(s) that you need to log the vSphere Client into in order to fix things.  To do this, you’ll need to change your PowerCLI insta...

1/26/2016 Update:  We've learned that this script doesn't elegantly handle the situation where multiple VM folders have the same name (but different parent folders).  I've put together a basic script that should help recover from that (in the comments), but I haven't had a chance to thoroughly test it.  That script is built to move all VMs from the "Discovered Virtual Machine" Folder back into their correct locations.  I'm working on an updated pair of scripts that will handle this whole situation better. 2/9/2016 Update:  I've published an updated version of the vCenter Migration Scripts  - check it out instead of these! One of my customers wanted to export their VM Folder structure and the associated permissions from one vCenter to another, in preparation for a vCenter forklift.  Once the destination vCenter was fully prepared, the goal would be to migrate ESXi hosts with VMs into the new vCenter, then move the VMs into the appropriate folders....

One of my customers recently had a major issue; due to a fault in their cooling system, their server room overheated and many devices turned themselves off in order to avoid damage... including one of their SANs.  After resolving the cooling issue, the environment largely came back up without serious incident, however there was one lingering problem. Several VMs in the environment could not connect to their distributed virtual switch.  Each VM was associated with its port group correctly, however the "connected" checkbox was cleared.  When we tried enabling that option, the task failed with an error that read: Invalid Configuration for Device '0' Some quick googling revealed a VMware KB Article about this particular issue, with several different workarounds suggested.  They ranged from easy (move the VM to a different Port Group and back) to arduous and time consuming (remove the NIC from the VM, then add a new NIC and reassign its network identity to the new NI...

One of my customers was transitioning into a cloud service provider, and as such their ESXi infrastructure needed to be changed around to support this decision.  A fairly standard design element of a provider datacenter is to break up the environment into 2 vCenter environments; one for management elements and one for customer VMs.  As such, we stood up the second vCenter server... but then we had the challenge of how do we move the exiting customers into the new customer resource vCenter? It wasn't that hard, although it took a little planning to ensure that we didn't cause any service disruption to the VMs that were already running.  Adding to the complexity of the move was the fact that Distributed vSwitches were in use at this site.  Given that vCenter owns the Distributed vSwitch, we weren't sure how the system would behave if we simply took over ownership of the ESXi servers into the new vCenter, so we settled on the following procedure: Remove vShield com...

A customer recently asked me how to easily check their environment for high CPU Ready metrics.  Given that I don't know of an easy way to quickly look through the whole environment by the GUI, I decided to pull out PowerCLI and see what I could put together. For those who don't know, CPU Ready records the time that a VM spends (during a given interval) ready for CPU time, but no physical CPU is available.  Basically, it reports on CPU contention.  There will always be small amounts of CPU Ready - that just reflects the time that it takes a CPU to react to an OS's request to compute.  In general, CPU Ready times averaging below 5% of your polling interval should be completely unnoticeable.  Between 5% and 10%, delays become noticeable, and beyond 10% they are seriously problematic. Those rules for interpreting CPU ready are all based on percentage of a time interval, but the metric that is actually recorded is the number of milliseconds spent ...

I’ve been working with Unidesk lately, as well as with the vSphere PowerCLI .  One of my customers has also been having an intermittent issue where they cannot browse their datastores (it just sticks at “Searching Datastore…” until they lose patience and close the window).  All of this has gotten me thinking about the relationship between vCenter and the ESXi hosts.  When we send a command to vCenter, such as creating a new VM, we all know that vCenter is simply relaying that command to one of its ESXi hosts, which does the actual work.  Same thing with file copies through the Datastore Browser, and in fact even reading the datastores through that same tool. How do we, as administrators, know which ESXi Host vCenter is using as its “workhorse”?  In a clustered environment, it’s really very unclear.  It turns out that, if that workhorse ESXi host is having an issue, it may prevent a lot of those basic vCenter functions from working (such as the aforem...

I feel silly linking to the Lone Sysadmin, as if my blog might get readership that his doesn’t, but I recently had to configure an MSSQL 2008 server for VCenter  for one of my customers, and his article on that subject was amazing.  I am by no means a DBA, but I feel that, by following his process, I’ve configured a server that a real DBA might look at and not immediately break down with tears of frustration.  In case any of you have missed it, it’s an excellent read and seems to result in a very robust configuration.

Does your VCenter run on an Oracle backend?  Has that Oracle server ever gone down unexpectedly and, even after the DBA swears it’s back up (and your ODBC connection tests successfully), your VCenter process continues to refuse to start?  You too?  Here’s some details about the situation that I experienced and how we got around it.  First, the identifying information: If you look on your VCenter server in C:\ProgramData\VMware\VMware VirtualCenter\Logs, you will see a lot of log files.  The VPXD-### log files are the ones that I looked at (the most recent one will be a .log file, the others are compressed tar.gz files that can be opened in Windows with 7-Zip , amongst other tools).  If your VCenter service is refusing to start but your ODBC Test Connection is successful, take a look at those files.  I found the following interesting errors in mine: [Vdb::LockRepositoryHelper] SQLExecDirect failed… …Failed to Lock vCenter Repository!!... Unable t...