VCenter Server Difficulties After Oracle Crash


Does your VCenter run on an Oracle backend?  Has that Oracle server ever gone down unexpectedly and, even after the DBA swears it’s back up (and your ODBC connection tests successfully), your VCenter process continues to refuse to start?  You too?  Here’s some details about the situation that I experienced and how we got around it.  First, the identifying information:

If you look on your VCenter server in C:\ProgramData\VMware\VMware VirtualCenter\Logs, you will see a lot of log files.  The VPXD-### log files are the ones that I looked at (the most recent one will be a .log file, the others are compressed tar.gz files that can be opened in Windows with 7-Zip, amongst other tools).  If your VCenter service is refusing to start but your ODBC Test Connection is successful, take a look at those files.  I found the following interesting errors in mine:

[Vdb::LockRepositoryHelper] SQLExecDirect failed… …Failed to Lock vCenter Repository!!...
Unable to get exclusive access to vCenter Repository.  Please check if another vCenter instance is running against the same database schema.

Well, I knew for a fact that there was no other vCenter instance running against this database, so I dug in a bit deeper.  These errors continued popping up (and the service continued failing) even after the DBA restarted the database instance (as the VMware KB article suggests).  On a whim, I decided to reboot both the VCenter server and the Oracle server (this server is exclusively used by VCenter, so it wasn’t a big deal)… and it came back up successfully!  Apparently, rebooting the database instance is not sufficient to clear up the stale sessions… rebooting the actual server, on the other hand, worked like a charm.

Hopefully you never need to do this.

Comments

Post a Comment

Sorry guys, I've been getting a lot of spam recently, so I've had to turn on comment moderation. I'll do my best to moderate them swiftly after they're submitted,

Popular posts from this blog

PowerShell Sorting by Multiple Columns

PowerShell's Sort-Object cmdlet is super useful, especially when preparing output for human consumption.  A few people have found my blog while looking for more information about its use, specifically while looking for how to sort by multiple columns (well, properties, technically).  I've never done so (much less written about it), so I hope those folks found answers elsewhere.  But, it got me curious... and it turns out that it's really easy. The Technet article on Sort-Object has the answer directly spelled out: just use commas to create a list of properties to sort by (in order to precedence).  Let's look at some examples!  First, prepare a variable with some good sortable data: $myData = get-eventlog System -newest 25 And then we can get to sorting!  Say you want to sort primarily by EntryType (Warning, Information, Error) and then by Source.  Easy-peasy: $myData | sort EntryType,Source How about if you want it to be in descending order?  Yeah, there&
Read more

Clone a Standard vSwitch from one ESXi Host to Another

One of my customers is standardizing the configurations on their HP C7000 enclosures (they've been set up at various sites by various administrators with varying involvement from the architecture team).  As such, we need to stand up some temporary resources so that we can take down the main enclosure for reconfiguration.  That's fine, we can easily ship a smaller enclosure to each site for temporary compute resources.  Some of the sites are using Standard vSwitches, so we need to be able to quickly copy the networking configuration over from their existing blades to these new, slightly different blades.  As I see it, we had 2 good options: 1) Capture a Host Profile with the desired vSwitch configuration.  Delete every other component of the Host Profile so that only the networking section is applied; design the new ESXi hosts so that the vSwitches'll work with the old vmnic-to-vswitch configurations. 2) Write a script to clone the vSwitch from ones ESXi host to another.
Read more

Deleting Orphaned (AKA Zombie) VMDK Files

A problem that most Virtualization Administrators need to deal with is Orphaned VMDK files (or Zombie VMDK files, as some people call them).  These are Virtual Machine hard drive files that are just hanging out, taking up storage, but are attached to no VMs.  How can this happen?  Well, the most common way is when an administrator needs to delete a VMDK but isn't quite ready to commit... so, after pressing "remove", they select "remove from virtual machine" rather than "remove and delete files".  The intent is always good: "I want to be able to restore this if it turns out to be necessary... so I'll just come back and delete it next week if nothing breaks!"  But, of course, next week something else happens and the file doesn't get deleted.  After a little while, it turns into "which files did I rename, again?" and nothing seems to happen. So, how do you deal with these orphaned files?  Well, a few years ago I found a scrip
Read more