Virtually Jason

One of my customers was experiencing some extreme IO latency in their environment: in the hundreds of ms.  Obviously, there was some pain associated with this issue, and so they asked for help.  This environment had 3 HP c7000 chassis and 2 different SANs; the issue was affecting every host accessing every LUN, so we decided that the issue must be in the fiber channel fabric somewhere. After poking around, we quickly realized that the firmware on the Brocade switches was from January 2013, so was quite old.  I pulled up top on each switch and saw no appreciable CPU or Memory usage.  Next, I looked at porterrshow to see if there were any problems on the switches; each one had a single port that had a ridiculous number of Enc Out errors.  I cleared the error stat counters by using portstatsclear -i 0-33 and then issued another porterrshow , and found that we had roughly 100,000 Enc Out errors on that port each second.  Coincidentally, each switch had 1 p...

One of my customers needed some help with making some zoning changes on their fiber switches after standing up a batch of new ESXi servers.  I already had a script to create 1:1 fiber channel zones on Brocade switches , so that part was easy, but zoning changes to an existing environment are a little scary.  As in, if you really mess it up, the storage is going to disappear and every VM is going to crash, scary.  Fortunately, you've got to really mess it up to cause an issue, and so this customer was willing to allow changes during business hours as long as we promised not to cause an outage ;) So, how can I enforce that promise?  Well, I've got my script to create accurate zones for the new hosts, but that's not really the dangerous part.  If that's messed up, it just means that the new hosts won't work... and since they're still being configured, they're obviously not in production yet.  The dangerous part is when you enable the new zones, in case you s...

One of my customers had a whole collection of ESXi 6 hosts that were all installed from the April 2016 HP ESXi ISO image... and they hadn't been patched since then.  Well, one day they called me because their Splunk server had started sending out alerts from an alarm that we'd set up to monitor the ESXi hosts for memory leaks. So, I logged into one of the affected hosts to try and figure out what was going on.  After poking around in a bunch of logs and a fair amount of google work, I came across this article about a memory leak over at CPU Ready.  It sure looked promising.  So, I followed their instructions to check the version of the broadcom driver in one of the affected ESXi hosts and, sure enough, it was an older version.  Fortunately, HP has a fix available, so I just needed to get it installed on all of the ESXi hosts (since full on ESXi patching wasn't necessarily available, unfortunately). I needed some way to figure out exactly which hosts needed ...

At one of my customers' sites today, I saw an error message that I've not seen before: VM MAC Conflict.  "Well, that's certainly not good," I thought, as I poked around at the error message.  To my chagrin, I could only find that error message for a single VM in the environment, and that error message wouldn't tell me with which other VM it was conflicting.  So, I could only think of one way to figure out what was going on with this conflict: look at the MAC Address assigned to every NIC on every VM in the environment, and figure out what was causing the conflict.  Easy! No, really, it was easy.  Had I done it by hand, I would certainly have driven myself crazy, but PowerCLI made it nice and easy.  I just used this command: (get-vm | get-networkadapter | ? {$_.MacAddress -eq "<offending MAC Address>").parent Lo-and-behold, it returned 2 VMs.  One was the known VM that had flagged the error and the other was a powered-off VM.  Maybe tha...

Recently, I've written several scripts that need to perform relatively simple operations on a large set of objects (such as moving a bunch of VMs onto a given Port Group or reconfiguring NTP for a bunch of ESXi hosts).  In general, I approach these challenges by generating a list of all of the objects that I want to manipulate, and then I ForEach my way through that list until I've finished all of my work. This approach obviously works just fine; it's the way that we'e written scripts for ages.  Just as you might expect from something that's been done the same way for a long time (particularly something IT related...), that's not really the best way to do it any more.  With PowerShell version 3, Microsoft introduced the concept of Parallel operations.  Starting with PowerCLI 6, VMware changed PowerCLI to make it much easier to use with PowerShell Parallel operations. So, what is a parallel operation?  Well, a simple (and very practical!) example is that For...

One of my customers recently had one of their vCenter 6 Server Appliances go offline.  The VM was still running and responding to pings, but the service wasn't working.  I established an SSH session to the server and went through the basics, and what do you know, "df -h" revealed that the /storage/seat partition was 100% full. Well, VMware has a fine KB Article about a full seat partition and how to solve it.  At least, mostly how to solve it.  The problem that I ran into is that the truncate commands (that free up space) were failing to run because there wasn't enough space on the partition.  When I tried to execute them, I got the following message: "ERROR: could not extend file ... No space left on device" "Hint: Check free disk space." I'll admit to chuckling when I saw the "hint" line.  So, I had to free up some disk space so that I could free up some disk space.  I did a bit of research into how to free up some space on ...

Here's a quick one-liner that I've been particularly pleased by.  I put this together when I was reconfiguring several c7000 chassis at a time and wanted to minimize the amount of typing that I was doing per host.  This is a big ugly one liner, but what it does is to create a new VMK interface on the specified Port Group, assign it a valid IP Address and enable vMotion for that interface.  The command performs those tasks on all ESXi hosts in the "unconfigured" folder. How does it get the valid IP Address?  Well, it takes the first 2 octets of the host's management IP address, uses a customized 3rd octet, then appends the last octet from the host's management IP address.  So, for example, if the host is managed at 192.168.1.101, the script will use 192.168.#.101 for the vMotion interface. In this example, it uses a port group called "vMotion" on a VDSwitch that ends with Intranet, and has a standard /24 network with 2 in the third octet for vMoti...