Virtually Jason

One of my customers has many physical datacenters, each with its own vCenter server.  The servers are all connected via Linked Mode, which works fine in the GUI, but at the CLI I don't usually want to connect to every one of them.  Fortunately, connect-viserver doesn't do so unless you use the -AllLinked switch, so that's fine.  Unfortunately, I'm left in a position where I need to connect my PowerCLI session to any one of many vCenter servers, and I don't necessarily like typing that much or remembering that many server names. Fortunately, I can address both of those points in one fell swoop, through PowerShell Profiles .  When you launch PowerShell, it simply applies the appropriate PowerShell Profile (aka. a script that you can use to customize your work environment) to that session and you're good to go.  So, I put together a customized PowerShell Profile for myself that presents me with a "Connect to vCenter" option and a little text driven men...

One of my customers set up a large number of ESXi hosts, installed on local storage.  Since they're not doing anything with that storage, they never bothered renaming those datastores and so their vCenter inventory was filled with instances of "datastore1 (#)".  This is just a housekeeping thing, but that's really messy.  Generally speaking, a bit of mess isn't a big deal, but those little messes always seem to cause problems when you're troubleshooting some other major issue.  And, fortunately, this is really easy to fix.  Rather than write my own script, I did some quick googling and found that VMNICK had done all the work already, with his own " rename local host storage " script. In his post, he notes that the script might explode if you use the string "datastore1" on any other datastores in the environment.  Excepting a very particular subset of situations, I'm not a fan of explosions, and so I figured that I could probably fix t...

Every now and then, as a virtualization engineer, you'll be going through a log entry or be examining a setting and you'll ask yourself, "I wonder where that file is?".  Why might you ask yourself that?  Probably because you're staring at a path that looks something like this: "/vmfs/volumes/5126bbe6-5478b245-773b-04519ad9e307/SacDC1/SacDC1.vmdk" and all of your datastores are named things like this: "sac-LUN1".  Needless to say, correlating the two can be a bit difficult. Translating LUN Names, just how do we do it?  Really quite simple, there's nothing much to it! (can you tell I have young kids at home?)  The only way that I know of to translate those UIDs to human friendly datastore names is to look at each datastore and see if the UID matches.  While that's an arduous and time consuming task for a human, it's a trivial one for a machine!  So, I put together the following PowerCLI one-liner to track down the datastore: get-...

A problem that most Virtualization Administrators need to deal with is Orphaned VMDK files (or Zombie VMDK files, as some people call them).  These are Virtual Machine hard drive files that are just hanging out, taking up storage, but are attached to no VMs.  How can this happen?  Well, the most common way is when an administrator needs to delete a VMDK but isn't quite ready to commit... so, after pressing "remove", they select "remove from virtual machine" rather than "remove and delete files".  The intent is always good: "I want to be able to restore this if it turns out to be necessary... so I'll just come back and delete it next week if nothing breaks!"  But, of course, next week something else happens and the file doesn't get deleted.  After a little while, it turns into "which files did I rename, again?" and nothing seems to happen. So, how do you deal with these orphaned files?  Well, a few years ago I found a scrip...

Have you ever been in a situation where you've needed to reload a VM's VMX file?  It's not very common, but it does happen occasionally.  VMware has a KB Article about how to do it, but it requires vim-cmd, which means that you need access to the host's CLI.  This isn't that big of a deal, unless you need to reload a lot of VMs that are scattered across many ESXi hosts.  Fortunately, Techazine has posted an article about a technique that reloads all invalid VMs via PowerCLI.  In my use case, I only needed to target a small set of VMs, so I just used that technique as follows: @("VM1","VM2","VM3") | % {(get-vm $_ | get-view).reload()}

One of my customers is running many vCenter servers at many different sites.  Each site has its own distributed vSwitch, which we want to be based on an enterprise standard configuration.  That said, each site has specific needs and so needs to be able to vary from that enterprise standard.  What's that?  A repetitive task where many distinct objects need to have precise configurations applied to them?  Let's write a script! I'm going to experiment with posting this script to GitHub rather than trying to put the thing in here as text.  It'll hopefully make it easier for people to use and will give me some version control and generally make it easier to access these things.  Before you use the script, make sure that the Distributed vSwitch exists and that your PowerCLI session is connected to vCenter.  So, how's this script work? Well, you call it like this: make-vdspg.ps1 -vdswitch site-vSwitch But it actually has more options than th...

One of my customers wanted to reset the root password on all of their ESXi servers.  This can be easily done through a Host Profile (which is particularly useful if you don’t know the root password), but that would require putting the hosts into maintenance mode and waiting for all of those vMotions to apply.  There’s no need to put a host into maintenance mode just to change the root password, and so we looked for a better solution. Jeff  found the PowerCLI cmdlet, set-vmhostaccount , and with that, we went to work.  The script is actually really simple – it just connects to the ESXi host, then uses the set-vmhostaccount cmdlet to change the password, and records the change. For flexibility, I included a few variables as parameters.  You may want to select only a subset of ESXi servers, and so there are both -hostSuffix and -hostPrefix parameters.  If the hosts that you want to target are all called Sac-ESX#.lab.local and the production hosts that you’...

One of my customers recently asked me to help them set up a stand-alone DR solution for their Unidesk + VMware View VDI.  We decided to try out the Unidesk LayerSync utility, to replicate their desktop configurations to the remote site, where we installed brand new View 6.1 connection servers.  There were a few small challenges to getting that running, but it's still in beta so that's perfectly understandable.  After we got the replication working, we stood up a few desktops as a test.  They booted up fine, all of the layer assembling as expected, and we were able to log into the consoles without trouble. We came across an issue though, Unidesk successfully added the desktops into the selected pool, but View reported them as being Agent Unreachable.  I went through the normal troubleshooting steps (testing name resolution, checking HKLM\Software\VMware, Inc.\VMware VDM\Agent\Configuration\ to make sure that the Broker setting is correct, verifying that  ...

One of my customers found that they had an IP conflict on their network, but the network team was unavailable.  They asked me if I could help them track down the address, while waiting for help from the network team.  I don’t have nearly as much visibility into the environment as a network admin, but vCenter is aware of the vast majority of the servers in the environment and it has data about what MAC Address each one is using.  Of course, that data is buried under “Edit Settings” on each VM and, when you have thousands of VMs to look through, doing that process by hand would be a staggering effort.  So, I scripted it. This is a really simple script that takes a single parameter –MacAddress.  That address should be specified in colon separated format (like 00:50:56:00:00:01) and it accepts wildcards (so *00:01 is a fine search query).  The script first searches all VMs for the specified MAC Address, then moves on and searches all of the VMKernel interface...

We’ve all probably been in this situation: vCenter has experienced a catastrophic failure and isn’t even responding to ping, so you need to get console access in order to resolve the issue.  The challenge is that the VM could be running on any of 16 ESXi hosts in your cluster.  Generally, I like to create DRS Host Affinity rules for the vCenter and its SQL server so that I know which host those VMs should be running on during normal operations, but if HA has been busy they could still be anywhere.  That means that you need to find those VMs in order to troubleshoot, which could take a while.  Instead of opening 16 vSphere Client instances, each connected directly to an ESXi host in your environment, there’s a much easier way to find those VMs: PowerCLI! PowerCLI can query a whole array of ESXi servers fairly quickly, returning the host(s) that you need to log the vSphere Client into in order to fix things.  To do this, you’ll need to change your PowerCLI insta...

One of my customers asked me to help out with the zoning configuration on their Brocade fiber switches.  I'm neither a SAN nor a network engineer, but I've created a few fiber zones in my time.  The best practice that I've always heard is to create 1:1 zones.  That means that each zone has 1 initiator (an ESXi host's VMHBA) and 1 target (the storage processor on the SAN).  This is a very simple process, but it's also really arduous and really boring, especially as environments scale up with redundant VMHBAs and multiple SANs with redundant storage processors. So, I spent a little bit of time reading about how to use the Brocade CLI , then moved forward.  I'm not quite confident enough in my scripting to actually enable any changes on the Brocade switches (after all, if you mess something up, storage could simply go away), so what I did was create a PowerCLI script that generates a bunch of Brocade CLI commands.  These commands (after you've read them over...

In this post, I'm going to discuss the details of the second half of the Windows Template Update workflow that we created in vRealize Orchestrator.  If you want to read an overview of the script, please check out  Part 1 !  If you want to read about the first half of the workflow, check out Part 2 ! The bulk of the work for this workflow happens in the "Download then Install Windows Updates until there are no more to install" section.  By the time the workflow has hit this section, it has a powered on VM that has Windows Updates enabled and all of the necessary scripts in place.  This section begins by executing the Download script.  This workflow step completes almost immediately, even though the script will take several minutes to complete, so it goes into a sleep (as Windows Updates can a while to download everything).  The final step of that script is to create a flag file - it either generates a "Done" file if there are no more updates to do...

In this post, I'm going to discuss the details of the first half of the Windows Template Update workflow that we created in vRealize Orchestrator.  If you want to read an overview of the script, please check out Part 1 ! There are two major sections before Windows Updates can actually be applied.  The first section ensures that the workflow has a powered on VM that is ready to be manipulated and the second section creates and transfers necessary resources to that VM.  How do they work? The Get Powered On VM section begins by checking if the supplied VM is a template or not.  If it is a template, it converts it to a VM and then waits 30 seconds before powering it on.  I included that sleep command because there is a note on the "Mark as VM" workflow that it can potentially complete before vCenter updates the VM's status, which can cause problems.  The "Start VM" workflow waits for VMTools to start; I went ahead and put a 30 second sleep in after that...

Jeff and I have been doing a lot of work with Orchestrator lately.  It's been a bit of an uphill struggle, but we've managed to create a few cool things!  This one is very simple, but it addresses a common pain point and so we're sharing it.  We've generated an Orchestrator workflow that automatically applies Windows Updates to templates. Admittedly, that's not a very difficult process to go through by hand... but it's arduous and needs to be done regularly... and, truth be told, it seems like most people just don't bother.  So, we decided to automate it!  It turns out that automating it was actually a fairly interesting process, as we had to deal with a lot of decisions that are very easy for a human to make but are not so straight-forward for a machine.  I'm going to try my hand at a 3 part series of posts, detailing this workflow. So, what does this Workflow look like?  Here you go: As you can see, the workflow has 4 major parts. ...

We've been working with Orchestrator recently and came across a subtle incompatibility issue.  Our vCenter is currently version 5.5, update 2.  We came across an authentication issue and one of the more popular work arounds on the forums was to upgrade Orchestrator to version 6.  We did a quick compatibility check on the good old VMware Interoperability Matrix and found that Orchestrator 6 has no problems with the vCenter 5 family, so we went ahead with the plan. We applied the update and everything seemed to be working just fine for several months.  We decided that we're happy with how Orchestrator is working and so want to roll it out to several of the other vCenter servers in the environment... and that's when we ran into a problem. We had some difficulty adding the second vCenter to the system, so we went ahead and removed the first one and then re-added them both.  They added just fine, so I logged into the Web Client to kick off a test workflow......

I’ve had the chance to work with vRealize Orchestrator recently.  It’s got a bit of a learning curve, but we’re making good progress and are managing to automate some of those basic tasks that just seem to eat up time otherwise.  One of the challenges that we’ve come across is related to running PowerShell commands in the Guest OS itself (this is different than the PowerShell integration, which is a whole different ball of wax) via VMTools.  We ended up trying a bunch of different techniques, with various drawbacks and advantages. The first thing that we tried was the “Run program in guest” workflow under vCenter -> Guest Operations -> Processes.  When you launch that workflow, you’ll need to provide several pieces of data: which VM you want to execute the application in, the credentials for that VM, the path to the program and any arguments that the program will need (as well as a few others that we didn’t use in this situation).  The VM and credentials...

One of my customers recently encountered an interesting problem.  They encountered an issue that caused their Unidesk desktop backups to fail to delete from the archive datastore.  As such, that datastore filled up and backups were unable to continue.  The customer was working directly with Unidesk and VMware to resolve the issue, but in the meantime needed an easy way to purge all of the stale backups from their system. A few months ago, I posted about setting up Persistent ESXi Scratch Space for hosts using SD cards for their installations.  In that post, we covered how to mount a VMFS volume into a PowerCLI session.  Once it's mounted, we created the per-host Scratch folders, but that same technique can be easily used here to help us find those stale backups.  First, we ran this command to mount the "VDI-Archive1" datastore as a drive named DS: New-PSDrive -Name "DS" -Root \ -PSProvider VimDatastore -Datastore (get-datastore VDI-Archive1) Next, we...

One of my customers is migrating a lot of ESXi hosts (with a strong naming convention) from one network onto a new network with a pre-defined IP Addressing scheme based around fully populated HP C7000 blade enclosures (containing 16 blades).  The host naming convention is <Site>-<Enclosure ID>-ESX<Blade Slot Number>.<FQDN>.  Based on this, I've been tasked with automating as much of this migration as possible, and part of that is automatically assigning the new IP Addresses to the blades. In order to do this, I'm using the new-vmhostnetworkadapter cmdlet.  That cmdlet will need several parameters in order to work: -virtualswitch -portgroup -IP -subnetmask Most of these are trivial, as they will be constant for every host in the environment.  The big exception to that statement is the IP address parameter.  As I said earlier though, due to the strong naming convention at this site we can derive the host's new IP address entirely f...

One of my customers has asked me to help them migrate from one vCenter server to another.  In order to do this without causing interruptions, we're using ESXi servers like boats to bring the VMs from the old environment into the new one; basically the same process that I used last time a customer asked me to do this. This time, we're completing this process at many different sites and each site has many VMs that need to traverse the vCenter servers.  Since this customer doesn't yet have vSphere 6 with its inter-vCenter vMotion capability, we're left doing it the hard way.  That means doing an arduous, detail oriented, highly repetitive task... or, writing a script to do it for me. The worst part of this process comes from the VM networking.  We have to create Standard vSwitches on each ESXi host with VLAN configurations that mirror the Distributed vSwitch that belongs in the vCenter, then move all of the VMs to the appropriate Port Groups on the newly create...