Virtually Jason

Hey everyone - one of my customers was doing some maintenance on the FC switches recently and wanted an easy way to ensure that all of their ESXi hosts had 4 Active links to their datastores before bringing a switch down for maintenance.  Sure, we can click through the GUI and manually examine each host... or we can automat it! So, I wrote a script to report the status of all paths to a given datastore (or datastores via wildcard matching) for all hosts that are connected to it!  The usage on this guy is pretty straight-forward: .\report-datastorePaths.ps1 -datastoreName <name of datastore to examine; multiple datastores may be specified with wildcards> -alertThreshold <if the number of active paths is less than this number, alert the user> That -alertTreshold parameter (which defaults to 4) is used to cause the script to write to the host if it ever finds a host with fewer than that number Active links.  So, if it doesn't spit out any red text, you're p...

 We were recently helping a customer with a domain consolidation where we had to copy a bunch of files from several different domains over to a new file server.  We decided to break the task up into a few steps, the first of which was to simply get the data onto the destination!  To do that, we decided to use Robocopy, but we were going to need to copy data from several different domains without trust relationships, meaning that we'd have to get a bit creative with gaining access to the source data. In the end, I put together a script!  This guy takes several important input parameters: foldersList, drivesList, and logFolder. FoldersList is the path to a CSV that lists all of the folders that are being migrated.  This script ultimately depends on two columns in that CSV: SourceDrive and DestinationDrive, although we put SourceUNC and DesintationUNC fields in that file to help with later steps.  At its core, the FoldersList CSV is used to make the robocopy c...

One of my customers wanted an easy way to generate a report of all of their NSX Firewall rules as a CSV.  So, simply use get-nsxfirewallrule | export-csv and call it a day, right?  Unfortunately, the NSX Firewall Rule objects are a bit too complex to directly export as a CSV, so I put together a short script to unpack them.  Nothing's too complex here, but sometimes it's nice to have an easy script that just gives you the information that you need! Check out the script on Github !

As you can tell by the kinds of posts that I've been making recently, I've had the chance to do a fair amount of work with NSX!  Within NSX, you can define IPSets to allow you to control the way non-NSX objects (such as physical devices) can interact with your VMs.  If you need to figure out which existing IPSet contains a given IP Address, you can run an easy VRNI query (ipset where ip address = <IP Address>)... but you don't always have access to VRNI!  So, I put together a PowerNSX script to check on it for me! Using this script is pretty simple: get-appliedNSXIPsets -ipAddr <IP Address in Question>.  It will return a table with three columns (NSXSecurityGroup, NSXIPSet, and IPAddress) and one row per unique Security Group / IPSet combination.  The script understands IPSets that list multiple objects, but only if they are whole subnets (defined with CIDR notation) or individual IP Addresses.  The script does not support ranges of IP Ad...

I've been working with one of my customers to implement the NSX Distribtued Firewall via Security Policies instead of hand-crafted rules.  For auditing and reporting purposes, we needed to be able to display all of the policies that have been created and the DFW rules within each one.  Ideally, we'd need to be able to generate this report on demand, with real-time data... and since the NSX GUI doesn't make that easily visible, I figured that I should put together a script to do it for me! Enter the summarize-NSXSecurityPolicy.ps1 script.  This script uses PowerNSX to get all of the defined Security Policies in the environment, then builds a table with one row per DFW Rule.  Each row contains several columns, including one for the policy that defines the rule, so we can easily filter the table to show what's going on with specific policies or we can easily search it to find a policy that involves specific traffic.  That later use case is almost certainly more im...

One of my customers is considering moving some of their infrastructure around and wanted to get an idea about how their WAN connection might be impacted by the move.  They didn't have vROPS and we didn't want to enable greater vCenter logging due to space constraints on the SQL server (that tells you that we're working with some older systems, doesn't it!).  So, I decided that our best course of action would be to write a script that could run on an interval, collecting and summarizing the real-time statistics that we actually needed.  Hence the creation of summarize-VMNetUsage.ps1 ! This is a pretty straightforward script.  If you run it without any parameters, it will find the highest 20 second Average Network Usage stats from all VMs in an environment, then return a summary of its findings: VM Count, sum, average, maximum, minimum, and a date-stamp.  Then, the script enters a holding pattern until 1 hour has passed and it starts the process again.  It ...

Well, that title sure is a mouthful!  But, it's also what this post is all about, so let's get to it!  One of my customers was experiencing an issue where it was taking longer than expected for an NSX firewall rule publish to propagate to all of their ESXi hosts. While troubleshooting the core issue, they needed a way to get better visibility into the process so that they'd know when their publishes had succeeded.  That data was not available in the GUI, but after asking a few friends at VMware, we learned that we could get to it through the API by a simple command: GET /api/4.0/firewall/globalroot-0/status .  Those are the facts that we collected, so here's what we did with them! First, I knew that one of my customers had done some work with the NSX API, so I asked him for some advice.  He pointed me at one of Mark Wahl's articles  and gave me an excellent framework to build on. I used that NSX API framework to send the GET command that we'd collected,...

One of my customers recently asked me to generate a report showing all of the VMs that they had created within the last 12 months (ideally, broken down by OS), and then another showing the same for 24-12 months ago.  I did a bunch of digging around and couldn't find any attribute on the VMs that showed their creation date.  Some research revealed that the standard solution to this problem is to get-vievents for all of the VMs, then look at the date of the first event. Unfortunately, this customer had performed a vCenter migration about a year ago, so our logs weren't intact for this purpose.  I was stumped, but one of the other admins came up with a good idea: look at the AD objects instead of the VM objects.  AD objects actually have a .whenCreated attribute, so we just need to grab them all and then find the ones for our desired timeframes. Of course, that approach grabs all AD computers, including desktops.  We just needed a list of servers (we knew that...

This is probably one of those articles that's only going to apply to a tiny percentage of people within an already miniscule niche subset of a small population... but I'm proud of my work and so am going to post it here anyway.  One of my customers needs to move a bunch of VMs off of one SAN onto another.  Storage vMotion for the win, end of story, right?  Yes* * 99% of the problem is absolutely solved with Storage vMotion, but I'm not in the business of leaving 1% unfinished.  In this case, that 1% was a bunch of older SQL Servers, set up in 2 node pairs using Microsoft Clustering Services via shared Physical Mode RDMs.  Yikes. In theory, this process isn't too bad.  Just record the vital statistics about the RDMs, then detach them from the VMs, move the VMs (during an outage window, obviously), create new RDMs using the recorded data, and power everything back up.  This process depends on the new-harddisk cmdlet... but given that I'm writing abou...

I recently wrote about a situation where I was creating a zoning configuration and had to figure out which fiber channel devices were active .  After we finished that project, we decided that we should go through and actually remove the inactive aliases and zones.  We had a list of active devices, so we were all ready to move forward and say "delete everything that isn't on this list!".  That'll work great, right? Of course not; we needed the opposite.  "Delete everything that is on this list" is a far better instruction that is way less likely to lead to painful mistakes.  Even better is "run these commands to delete all of these unnecessary objects" and I know one good way to generate such a list of commands: a script (I feel like I'm developing a battle-cry...). I put together a script that does a few basic things.  First, it uses the nsshow  command to get a list of all of the active WWNs on a given Brocade fiber switch.  Then, it compa...

One of my customers recently purchased a new SAN, with the goal of decommissioning the old one.  They used Storage vMotion to migrate all of their VMs over to the new SAN and adjusted all of the ESXi hosts to put their scratch space on a new LUN, and were ready to proceed. Many people, at this point, would just turn off the old SAN... and they might be ok.  Maybe.  At that point, the ESXi hosts are going to seriously freak out, because they just encountered an unexpected SAN failure... and we've all seen that sometimes, ESXi doesn't respond well to losing datastores unexpectedly. So, the more cautious people would right click on each datastore and unmount it, then turn off the old SAN.  While not as bad as just turning off the SAN, the ESXi hosts still expect those LUNs to be there (even if they're no longer mounted as datastores) and can still run into issues. Miss Manners insists that people follow the procedure detailed in KB 2004605 .  That article inc...

One of my customers had an issue where a bunch of VMs were not able to vMotion, despite the hosts being configured correctly in all regards (other VMs using the same VDS Port Groups, for example, could vMotion onto and off of the host where these VMs were running).  When DRS (or an administrator) attempted a vMotion, a generic "A general system error occurred: vim.fault.NotFound" error message would be displayed. When I took a look at these VMs, I noticed something interesting (besides the fact that they were all on the same host); their VDS Port numbers were universally high, like in the 5000s.  This was particularly interesting because when I looked at the VDS itself, the highest numbered port on it was 4378.  I supposed that these ephemeral ports had somehow been assigned invalid port numbers, which was causing vMotion to fail when the new destination was unable to reserve that invalid number on the VDS.  Interestingly, all of these VMs were communicating just...

Towards the start of the year, I made a post about my vCenter to vCenter migration process , which can be used to update from vCenter 5.5 to vCenter 6 (or for that matter, to downgrade, or whatever).  Since then, I've done a lot of these migrations and the scripts/procedure have matured.   Jeff and I have fixed many bugs, but the reason that I'm posting about this (again) is that I recently added a new feature and have polished our migration process. The latest versions of the migration scripts (the ones that get the settings from the source vCenter and then recreate those settings on the destination vCenter) now support DRS Rules!  The native support for DRS Rule manipulation through PowerCLI is a bit lacking... but fortunately, the community has solved that problem!   Matt Boren and Luc Dekens have created a module called DRSRule that is built to help with the reading and manipulation of DRS Rules (pretty self explanatory, right?).  I took advantage of t...

One of my customers uses DRS groups heavily.  The environment contains a set of specialized ESXi servers for a specific subset of machines and a set of general ESXi servers for everything else.  However, in order to maximize availability, all of these ESXi servers are part of one big cluster.  That means that every VM in their environment must be a member of at least one DRS group, placing the VM on either these specialized ESXi servers or the general ESXi servers.  Until we get an automated solution in place (probably through Orchestrator), we need to depend on the administrators to place the VM in the correct group (almost always the general group) upon machine creation.  As you might expect, that's not going too well. So, I put together a quick PowerCLI script to detect any VMs that are not grouped and (optionally) place them into the specified group.  This is what I came up with! The script is easy to use, but it requires a non-standard PowerCLI mod...

One of my customers recently had a catastrophic thermal event in one of their datacenters and so had to shut down all of their infrastructure at that site.  After the cooling issue was resolved, we were asked to help them to get their infrastructure back online.  Fortunately, we have included several small details as best practices in our vSphere designs, and one of those really paid off for us.  We always create a VM to Host affinity rule that keeps one Domain Controller, the vCenter server, its PSC and its Database (if external) on a known host in the management cluster. So, after the SAN was powered back on and we restarted the physical ESXi servers, we knew exactly what to do.  I fired up the vSphere client and logged into that ESXi server in the management cluster as root.  From there, I was able to easily find those core infrastructure VMs and powered them all on.  Once they were running, I logged into vCenter... and found that I had an interesting ...

One of my customers recently had a host throw a PSOD.  It's a large environment with appropriate spare capacity, so that wasn't a major issue.  I mean, we never want a host to go down, but we designed the environment to accommodate that situation and it largely responded well... except for 2 VMs.  They errored out, with a message saying that "vSphere HA unsuccessfully failed over this virtual machine... ...Reason: An error occured during host configuration".  We found a related event that said, "Operation failed, diagnostics report: Failed to open file /vmfs/volumes/<Datastore UUID>/.dvsData/<DVS UUID>/<Port Number> Status (bad0003)= Not found".  True to the error message, that file was not there. VMware has a KB Article about this issue, but it's related to vSphere 5.0, when Storage vMotion fails to move a file that HA requires for tracking the VM on the Distributed vSwitch.  This customer is on vSphere 5.5 (and has been for a signi...

9/15/2016 Update:  I've updated these scripts with some new functionality.  I just posted a description of the new functionality as well as how to use the scripts in the context of a vCenter Upgrade . A couple years ago, I made a blog post about a process that I use to migrate from one vCenter to another , and then I later created some scripts  to facilitate that process.  Those scripts worked fine for me, but others have found some weaknesses.  Namely, they don't work if there are multiple folders in the inventory with the same name (such as VM\Project1\Production and VM\Project2\Production, where there are two folders named Production).  Also, I never thought about VMs in vApps, which don't technically have a parent folder. So, I've been working on updated scripts.  I'm a little hesitant to post these, as I haven't actually used them to perform a migration yet, but there seems to be a lot of people who need them.  I've put together two scrip...

One of my customers set up a large number of ESXi hosts, installed on local storage.  Since they're not doing anything with that storage, they never bothered renaming those datastores and so their vCenter inventory was filled with instances of "datastore1 (#)".  This is just a housekeeping thing, but that's really messy.  Generally speaking, a bit of mess isn't a big deal, but those little messes always seem to cause problems when you're troubleshooting some other major issue.  And, fortunately, this is really easy to fix.  Rather than write my own script, I did some quick googling and found that VMNICK had done all the work already, with his own " rename local host storage " script. In his post, he notes that the script might explode if you use the string "datastore1" on any other datastores in the environment.  Excepting a very particular subset of situations, I'm not a fan of explosions, and so I figured that I could probably fix t...

One of my customers is running many vCenter servers at many different sites.  Each site has its own distributed vSwitch, which we want to be based on an enterprise standard configuration.  That said, each site has specific needs and so needs to be able to vary from that enterprise standard.  What's that?  A repetitive task where many distinct objects need to have precise configurations applied to them?  Let's write a script! I'm going to experiment with posting this script to GitHub rather than trying to put the thing in here as text.  It'll hopefully make it easier for people to use and will give me some version control and generally make it easier to access these things.  Before you use the script, make sure that the Distributed vSwitch exists and that your PowerCLI session is connected to vCenter.  So, how's this script work? Well, you call it like this: make-vdspg.ps1 -vdswitch site-vSwitch But it actually has more options than th...

One of my customers wanted to reset the root password on all of their ESXi servers.  This can be easily done through a Host Profile (which is particularly useful if you don’t know the root password), but that would require putting the hosts into maintenance mode and waiting for all of those vMotions to apply.  There’s no need to put a host into maintenance mode just to change the root password, and so we looked for a better solution. Jeff  found the PowerCLI cmdlet, set-vmhostaccount , and with that, we went to work.  The script is actually really simple – it just connects to the ESXi host, then uses the set-vmhostaccount cmdlet to change the password, and records the change. For flexibility, I included a few variables as parameters.  You may want to select only a subset of ESXi servers, and so there are both -hostSuffix and -hostPrefix parameters.  If the hosts that you want to target are all called Sac-ESX#.lab.local and the production hosts that you’...