vCenter Server Appliance Crash due to Full /Storage/SEAT Partition

One of my customers recently had one of their vCenter 6 Server Appliances go offline.  The VM was still running and responding to pings, but the service wasn't working.  I established an SSH session to the server and went through the basics, and what do you know, "df -h" revealed that the /storage/seat partition was 100% full.

Well, VMware has a fine KB Article about a full seat partition and how to solve it.  At least, mostly how to solve it.  The problem that I ran into is that the truncate commands (that free up space) were failing to run because there wasn't enough space on the partition.  When I tried to execute them, I got the following message:

"ERROR: could not extend file ... No space left on device"
"Hint: Check free disk space."

I'll admit to chuckling when I saw the "hint" line.  So, I had to free up some disk space so that I could free up some disk space.  I did a bit of research into how to free up some space on there, but the general feeling I got was "don't delete any of those files!" and I wasn't feeling overly brave... especially since I had another option.  Rather than deleting files to make free space, I could simply expand the VMDK to accomplish the same goal.  And, VMware has another fine KB article about how to grow a VCSA partition!

Of course, in order to follow that procedure, I had to track down the VCSA VM so that I could modify its hardware.  That's not always easy without vCenter... but we know how to find vCenter when it's down (please forgive the shameless plug).  So, after I grew the appropriate VMDK and extended the partition with the vpxd_servicecfg storage lvm autogrow command, I was able to finish the procedure in the first KB article by executing the truncate table vpx_event cascade; and truncate table vpx_event_arg cascade; database commands.

Once those completed, I restarted my vCenter services and everything came back up.  That left me with diagnosing the actual problem.  From the process in that first KB article, I knew that one of my ESXi hosts was generating a ridiculous amount of SEAT data (over 35 million "vim.event.GeneralHostWarningEvent" events and another 35 million "com.vmware.vc.StatelessAlarmTriggeredEvent" events).

Poking around at that host revealed a *bunch* of entries in \var\log\vmkernel:
"ALERT: vmsyslog logger <syslog server> lost X log messages"

And it turns out that there's another VERY applicable VMware KB article about this situation.  That KB has one of those good news/bad news lines.  This excessive logging behavior is a known issue that is resolved in an update.  So, our next step is simply to update these servers!  In the meantime, in my SSH session on the appliance, I've used truncate -s 2G /storage/seat/DeleteMe to create a 2 GB junk file that I can easily delete in case this issue comes back before we are able to apply updates to these hosts.

Comments

  1. UnknownMay 31, 2017 at 5:51 AM

    Hi Jason,

    Thanks for documenting this. I had a similar issue but when I tried to expand the vmdk file in gui, it failed with the error - “invalid configuration for device 0”. I had to SSH to the hosts and use vmkfstools -X to expand the drive. Once expanded the remaining of the journey was uneventful.

    Regards
    Rahul Stephen

    ReplyDelete

Post a Comment

Sorry guys, I've been getting a lot of spam recently, so I've had to turn on comment moderation. I'll do my best to moderate them swiftly after they're submitted,

Popular posts from this blog

Clone a Standard vSwitch from one ESXi Host to Another

One of my customers is standardizing the configurations on their HP C7000 enclosures (they've been set up at various sites by various administrators with varying involvement from the architecture team).  As such, we need to stand up some temporary resources so that we can take down the main enclosure for reconfiguration.  That's fine, we can easily ship a smaller enclosure to each site for temporary compute resources.  Some of the sites are using Standard vSwitches, so we need to be able to quickly copy the networking configuration over from their existing blades to these new, slightly different blades.  As I see it, we had 2 good options: 1) Capture a Host Profile with the desired vSwitch configuration.  Delete every other component of the Host Profile so that only the networking section is applied; design the new ESXi hosts so that the vSwitches'll work with the old vmnic-to-vswitch configurations. 2) Write a script to clone the vSwitch from ones ESXi host to another.
Read more

PowerShell Sorting by Multiple Columns

PowerShell's Sort-Object cmdlet is super useful, especially when preparing output for human consumption.  A few people have found my blog while looking for more information about its use, specifically while looking for how to sort by multiple columns (well, properties, technically).  I've never done so (much less written about it), so I hope those folks found answers elsewhere.  But, it got me curious... and it turns out that it's really easy. The Technet article on Sort-Object has the answer directly spelled out: just use commas to create a list of properties to sort by (in order to precedence).  Let's look at some examples!  First, prepare a variable with some good sortable data: $myData = get-eventlog System -newest 25 And then we can get to sorting!  Say you want to sort primarily by EntryType (Warning, Information, Error) and then by Source.  Easy-peasy: $myData | sort EntryType,Source How about if you want it to be in descending order?  Yeah, there&
Read more

Deleting Orphaned (AKA Zombie) VMDK Files

A problem that most Virtualization Administrators need to deal with is Orphaned VMDK files (or Zombie VMDK files, as some people call them).  These are Virtual Machine hard drive files that are just hanging out, taking up storage, but are attached to no VMs.  How can this happen?  Well, the most common way is when an administrator needs to delete a VMDK but isn't quite ready to commit... so, after pressing "remove", they select "remove from virtual machine" rather than "remove and delete files".  The intent is always good: "I want to be able to restore this if it turns out to be necessary... so I'll just come back and delete it next week if nothing breaks!"  But, of course, next week something else happens and the file doesn't get deleted.  After a little while, it turns into "which files did I rename, again?" and nothing seems to happen. So, how do you deal with these orphaned files?  Well, a few years ago I found a scrip
Read more