Updating iLO on a BL460c ESXi Blade

Several of my customers use HP C7000 blade chassis for their ESXi hosts.  One of them asked me to help them update the firmware in their enclosures.  There are basically three different pieces of the blade chassis; each one needs updating.  The blades themselves need updating, the Interconnect Modules (basically, embedded switches) need updating and the Onboard Administrator Modules need updating.  There are different and varied steps for updating each of those components.  The blade is updated by bootstrapping with an SPP (Service Pack for Proliant) ISO.  The OA is updated through its own web interface by uploading a specific .bin file.  The Virtual Connect (which is the centralized management for all of the Interconnect Modules) is updated through its own special command line utility by uploading its own specialized .bin file.

All of that is a little awkward, but it’s not really that big of a deal.  We did run into one spot of difficulty, as one of the BL460c ESXi hosts was reporting that it had lost access to its SD card.  A bit of research pulled up an interesting article from HP about a blade firmware update that would resolve the issue.  Seeing as how we were doing an update cycle anyway, this seemed like it would be a nice easy fix!  Of course, since I’m writing about it, you can probably guess that it didn’t work out quite so easily.

We specifically had to update the iLO firmware on the malfunctioning blade (and eventually, all of the blades in order to head off this issue).  Applying blade level updates, including iLO, is easily done by bootstrapping the system with an HP SPP disk.  The latest SPP disk (at the time of writing) is from September 2014 and updates the iLO firmware to version 2.00; the version that we needed to apply is version 2.03.  So, when you click on the download link next to the 2.03 firmware update, what do you get?  You get an .scexe file (after navigating the system and selecting ESX 5 as the Operating System).  Drilling down into the installation instructions reveals them to be functionally useless (at least, if you’re trying to apply the update to a system that can’t boot into an OS).  When I read those instructions, I basically understood that I had to generate my own custom SPP ISO, with the iLO firmware .scexe baked into it.  Some more googling brought me to a slightly more verbose description of how to create a custom SPP, but I wasn’t able to get that working (and after waiting through the ISO creation process... and the bootstrapping process, only to have it fail without an error message a few times… well, I was pretty frustrated with the whole process).

So, I got to thinking.  If you log into the iLO of the blade (either directly or through the chassis’ OA proxy), there’s a firmware update section (under Administration -> Firmware).  This section has an upload button, but no clear indication of what file you’re actually supposed to upload (I tried the .scexe and it returned an error, stating that it was not a valid file).  I read through the installation instructions for some other OSes and saw that the .scexe file is really just a container file that has the needed .bin inside of it.  Of course, I had no idea about how to get into this particular file, as it was built to run on ESXi (and I didn’t want to test this theory on another, working host).  So, I went back to the HP download site and downloaded the Windows Server 2012 iLO firmware package (after all, firmware is firmware, regardless of the OS that’s running above it).  I was able to run this executable on my desktop (pressing “Extract” rather than “Install” when prompted) and, there in my directory was sitting the .bin file that I needed!

So, I went into the blade’s iLO, went to Administration > Firmware and then browsed to the .bin file that I had extracted.  I pressed the upload button and watched the system go through its error checks and… success!  The iLO system took a bit longer to reboot than I expected (it came back to the login prompt almost instantly, but things were sluggish and occasionally 404’ed for a couple of minutes after the update).  After I gave it a few minutes to complete its process, everything seemed to be nice and stable and, when I restarted my ESXi installation, the SD card was ready and available for me to install upon.

Comments

  1. E SearcySeptember 26, 2016 at 12:11 PM

    Jason have you had a problem with connecting to the irc using the OA

    ReplyDelete
  2. E SearcySeptember 26, 2016 at 1:08 PM

    second how do you update the ilo for the blades using the OA

    ReplyDelete
    Replies
    1. Jason ColemanSeptember 26, 2016 at 1:16 PM

      Under "Administration" click on the "Firmware" section. You'll need the correct .bin file for that to work though.

      Delete
  3. UnknownJanuary 3, 2017 at 9:36 AM

    Jason, I have C7000 OA at 3.21. Recently the OA encounters an SSL_ERROR_BAD_MAC_ALERT when trying to get to any of the 16 ILOs [I run wireshark to see the cert presented by the ILO]. This occurs for any browser, PC, and even access using direct to ILO [without OA]. Chatting with HP, they say to upgrade OA, but the upgrade won't take. Since this occurs with all 16 blades, it must have something to do with OA? The only clue I have is the date of 11-27-2016 in the OA cert admin -when this started. But what would this have to do with it? Can't seem to get anywhere.... thanks for any help. John

    ReplyDelete
    Replies
    1. Jason ColemanJanuary 3, 2017 at 10:59 AM

      Yeah, it sounds like an OA issue to me because of the scope, but I don't have any more insight than that. That said, even if the OA is down, I think that you should be able to connect to the iLO directly (although it'll still route that traffic through the OA's network adapter), so maybe not. Have you tried doing a hard reset on the iLO on one of the blades to see if that clears it up? I wish I had a better idea for you.

      Delete
  4. UnknownJanuary 3, 2017 at 1:45 PM

    Thanks for your response Jason! OA is not down, I can still access it just fine with any browser. Just the access to ILO via this [or directly vioa browsr] gives me the bad cert warning. I still did putty directly to OA, logged on and then did a "CONNECT SERVER 1 [e.g.]" and then cd /map1 and then "reset" the ILO. [can't remember where I got all that....] Nothing changed. Is that a hard reset on the ILO? The cmd did take and I was able to come back in. Don't see much there via help to get anywhere. JOhn

    ReplyDelete
    Replies
    1. Jason ColemanJanuary 4, 2017 at 7:33 AM

      Sorry that I haven't seen this one myself, but have you seen this article https://www.hahosting.info/2015/07/hp-ilo3-ssl_error_bad_mac_alert-error-when-connecting/? That makes sense to me as a potential source of the issue. Also, what did you mean when you said that the iLO upgrade won't take? I',m guessing that you can't you get into the Web Administration for the blade and attempt to update the firmware directly via the .bin file as I described; here's an article that describes performing that update via SSH: http://techmolecules.blogspot.com/2014/08/four-ways-to-update-hp-ilo-firmware.html

      Delete
  5. UnknownJanuary 4, 2017 at 10:34 AM

    Hmmm. Sounds possible. Thot I went there once-suspected something like that, start to forget all these steps. But my browser worked till Dec 2016. But I had not encountered that link. Thanks!! I will go back on that path for sure. I could not update OA 3.21 all the way to 460... I suspect I need an intermediate step [OA told me he couldn't update to that firmware]. Once I get OA updated, will try to update ILO.... too much fun. Thanks again!

    ReplyDelete
  6. UnknownJanuary 5, 2017 at 4:48 AM

    Jason,
    I managed to upgrade OA to 3.6 (same as my other C7000). Still no go tho accessing the ILO.
    I then managed to turn off TLS 1.2 in IE, and was able to "get" to the ILO. Once there I got a warning that the digital signature on "com.hp.ilo2.intgapp.intg.app" was signed with a trusted cert, but that it had expired..... So I guess an ILO update is in order...
    BUT If I understand correctly, I will have to reboot all my blades with SPP! Not sure I want to take all those servers down real soon! Lots of stuff running on them :-(
    Of course I have to use an outdated java to run all this.
    So I guess I am back in business. Add to your own knowledge base I guess. I appreciate your help!!!! Take care. John

    ReplyDelete
    Replies
    1. Jason ColemanJanuary 5, 2017 at 9:41 AM

      Glad that I could help. I believe that you can update iLO through the Web Administration page without impacting the actual server by uploading the iLO bin directly.

      Delete

Post a Comment

Sorry guys, I've been getting a lot of spam recently, so I've had to turn on comment moderation. I'll do my best to moderate them swiftly after they're submitted,

Popular posts from this blog

Clone a Standard vSwitch from one ESXi Host to Another

One of my customers is standardizing the configurations on their HP C7000 enclosures (they've been set up at various sites by various administrators with varying involvement from the architecture team).  As such, we need to stand up some temporary resources so that we can take down the main enclosure for reconfiguration.  That's fine, we can easily ship a smaller enclosure to each site for temporary compute resources.  Some of the sites are using Standard vSwitches, so we need to be able to quickly copy the networking configuration over from their existing blades to these new, slightly different blades.  As I see it, we had 2 good options: 1) Capture a Host Profile with the desired vSwitch configuration.  Delete every other component of the Host Profile so that only the networking section is applied; design the new ESXi hosts so that the vSwitches'll work with the old vmnic-to-vswitch configurations. 2) Write a script to clone the vSwitch from ones ESXi host...
Read more

PowerShell Sorting by Multiple Columns

PowerShell's Sort-Object cmdlet is super useful, especially when preparing output for human consumption.  A few people have found my blog while looking for more information about its use, specifically while looking for how to sort by multiple columns (well, properties, technically).  I've never done so (much less written about it), so I hope those folks found answers elsewhere.  But, it got me curious... and it turns out that it's really easy. The Technet article on Sort-Object has the answer directly spelled out: just use commas to create a list of properties to sort by (in order to precedence).  Let's look at some examples!  First, prepare a variable with some good sortable data: $myData = get-eventlog System -newest 25 And then we can get to sorting!  Say you want to sort primarily by EntryType (Warning, Information, Error) and then by Source.  Easy-peasy: $myData | sort EntryType,Source How about if you want it to be in descending or...
Read more

Deleting Orphaned (AKA Zombie) VMDK Files

A problem that most Virtualization Administrators need to deal with is Orphaned VMDK files (or Zombie VMDK files, as some people call them).  These are Virtual Machine hard drive files that are just hanging out, taking up storage, but are attached to no VMs.  How can this happen?  Well, the most common way is when an administrator needs to delete a VMDK but isn't quite ready to commit... so, after pressing "remove", they select "remove from virtual machine" rather than "remove and delete files".  The intent is always good: "I want to be able to restore this if it turns out to be necessary... so I'll just come back and delete it next week if nothing breaks!"  But, of course, next week something else happens and the file doesn't get deleted.  After a little while, it turns into "which files did I rename, again?" and nothing seems to happen. So, how do you deal with these orphaned files?  Well, a few years ago I found a scrip...
Read more