Setting up Persistent Scratch Space for an Arbitrary Number of ESXi Hosts

We've been configuring a large number of ESXi servers that don't have local hard drives.  As such, we need to set up each system's scratch location on persistent storage, as per that KB article from VMware.

That KB Article has a great section devoted to performing the process from PowerCLI and it basically takes you through the whole thing, soup to nuts, for a single ESXi host.  That's all well and good, but what if you've got 16 hosts?  Or 32?  Or 64?  Well, with a few minor changes to the procedure that they've outlined, you can very easily target every ESXi host in an environment.  Bear in mind that this process does require a host reboot, so make sure that you have your ducks in a row (and preferably do this before you place any VMs onto the cluster).

Instead of their step 6, I used this command to create a folder for all ESX hosts in the inventory:
get-vmhost | foreach {new-item ".$($_.name)/scratch" -itemtype directory}

Instead of their step 8, I used this command to change the setting on all ESX hosts in the inventory:
get-vmhost | Set-VMHostAdvancedConfiguration -name "ScratchConfig.ConfiguredScratchLocation" -Value "/vmfs/volumes/LUN-ID/.$($_.name)/scratch"

Make sure to change the path in there to whatever LUN you want to work with, as "LUN-ID" isn't likely to work in your environment.

If you're feeling masochistic (or just want to strip the domain suffix from the hostname), you can replace all of the
".$($_.name)/scratch" instances with ".$($_.name.split(".")[0])/scratch"

If you want to filter either command to hit only a subset of the hosts, you can change the "get-vmhost" command to something more specific like "get-vmhost esx01".

So, here's the assembled process in its entirety:

1) Launch PowerCLI
2) Connect to vCenter:
connect-viserver <vCenter name>
3) Get the list of datastores in the environment:
get-datastore
4) Mount the datastore through your PowerCLI session:
New-PSDrive -Name "DS" -Root \ -PSProvider VimDatastore -Datastore (Get-Datastore "<Datastore Name>")
5) Move into the mounted datastore:
cd DS:
6) Create a folder for each ESXi Host:
get-vmhost | foreach {new-item ".$($_.name)/scratch" -itemtype directory}
7) If desired, check the current scratch location (I'm mostly including this to keep my numbers in line with the original article's numbers):
get-vmhost | Get-VMHostAdvancedConfiguration -Name "ScratchConfig.ConfiguredScratchLocation"
8) Set the Scratch location for the hosts:
get-vmhost | Set-VMHostAdvancedConfiguration -name "ScratchConfig.ConfiguredScratchLocation" -Value "/vmfs/volumes/LUN-ID/.$($_.name)/scratch"
9) Reboot the ESXi hosts.

Update: Since set-vmhostadvancedconfiguration is deprecated (although still works, at least for now), here's the new way to do step 8 (although I haven't actually done it this way yet):
get-vmhost | get-advancedsetting -name "ScratchConfig.ConfiguredScratchLocation" | set-advancedsetting -value "/vmfs/volumes/LUN-ID/.$($_.entity)/scratch"

Update 2: I've been doing this a lot and have optimized the process a bit.  Here it is in 5 steps (after connecting the PowerCLI session to the vCenter server).  Bear in mind that each of these is its own, single line... some of them have a lot of string manipulation in it to build the correct path for the scratch config.

1) Get the target Datastore
$thisDataStore = Get-datastore LogDataStore
2) Mount the Datastore via PowerShell
New-PSDrive -Name "DS" -Root \ -PSProvider VimDatastore -Datastore $thisDataStore
3) Create the per-host scratch space folders on the datastore
get-vmhost | foreach {new-item "DS:\.$($_.name.split('.')[0])/scratch" -itemtype directory}
4) Reconfigure the ESXi hosts to use those folders.
get-vmhost | foreach {$_ | get-advancedsetting -name "ScratchConfig.ConfiguredScratchLocation" | set-advancedsetting -Value "$($thisDataStore.ExtensionData.info.url.replace('ds://','')).$($_.name.split('.')[0])/scratch" -confirm:$false}
5) Reboot the ESXi hosts.

Comments

  1. Hi,
    How this supposed to look like?
    ($thisDataStore.ExtensionData.info.url.replace('ds://',''))

    ReplyDelete
    Replies
    1. If you're ever unsure about what a section of code does, you can usually extract it and run it by yourself. If it's a command, VMware's really good about supporting the -WhatIf switch, which will make the command tell you what it would do but not actually do anything.

      In this case, that bit is just string manipulation. A Datastore object (in my script, the $thisDatastore variable) has a property called .ExtensionData.info.url which is the datastore's name, in a path format. Unfortunately, that path starts with ds:// which is not valid syntax for the ScratchConfig field that I want to put it into. So, I call the .replace() method on that string (that whole path, starting with the ds://), and replace those four characters with nothing.

      If you look back at the original steps that I posted, that bit of code generates the "/vmfs/volumes/LUN-ID/" section of the command in the original step 8 (although, it obviously populates the LUN-ID for you).

      Delete
    2. Thanks Jason,
      Something I am doing wrong :)
      PowerCLI ds:\scratch> $thisDataStore = Get-datastore EUC_NFS_LOG
      PowerCLI ds:\scratch> get-vmhost | foreach {$_ | get-advancedsetting -name "ScratchConfig.Configured
      ScratchLocation" | set-advancedsetting -Value "$($thisDataStore.ExtensionData.info.url.replace('ds:/
      /','')).$($_.name.split('.')[0])" -confirm:$false}
      set-advancedsetting : 28/10/2015 8:21:20 AM Set-AdvancedSetting A specified parameter
      was not correct.

      At line:1 char:98
      + get-vmhost | foreach {$_ | get-advancedsetting -name "ScratchConfig.ConfiguredSc ...
      + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      + CategoryInfo : NotSpecified: (:) [Set-AdvancedSetting], InvalidArgument
      + FullyQualifiedErrorId : Client20_SystemManagementServiceImpl_GetVmHostAdvancedConfiguration_
      ViError,VMware.VimAutomation.ViCore.Cmdlets.Commands.SetAdvancedSetting

      set-advancedsetting : 28/10/2015 8:21:22 AM Set-AdvancedSetting A specified parameter
      was not correct.

      Delete
    3. Try executing this line after you get your datastore:
      get-vmhost | foreach {echo "$($thisDataStore.ExtensionData.info.url.replace('ds://','')).$($_.name.split('.')[0])/scratch"}

      It should just echo out the /vmfs/volumes/... stuff. Let's make sure that it's working. It might be something simple, like PowerShell version. You can use use $psversiontable.psversion to see what version you have installed; I write everything using version 4+

      Delete
    4. Major Minor Build Revision
      ----- ----- ----- --------
      4 0 -1 -1

      Delete
  2. Hi Jason,
    I have tried this:
    PowerCLI C:\Program Files (x86)\VMware\Infrastructure\vSphere PowerCLI> get-vmhost | foreach {echo "
    $($thisDataStore.ExtensionData.info.url.replace('ds:/',''))/scratch/$($_.name.split('.')[0])"}
    //vmfs/volumes/285a2ce0-5072c921//scratch/ucs2bccr241
    //vmfs/volumes/285a2ce0-5072c921//scratch/ucs2bccr242
    //vmfs/volumes/285a2ce0-5072c921//scratch/ucs2bccr227
    //vmfs/volumes/285a2ce0-5072c921//scratch/ucs2bccr238
    //vmfs/volumes/285a2ce0-5072c921//scratch/ucs2bccr239
    //vmfs/volumes/285a2ce0-5072c921//scratch/ucs2bccr240
    //vmfs/volumes/285a2ce0-5072c921//scratch/ucs2bccr225
    //vmfs/volumes/285a2ce0-5072c921//scratch/ucs2bccr226
    //vmfs/volumes/285a2ce0-5072c921//scratch/ucs2bccr228
    //vmfs/volumes/285a2ce0-5072c921//scratch/ucs2bccr229
    //vmfs/volumes/285a2ce0-5072c921//scratch/ucs2bccr230
    //vmfs/volumes/285a2ce0-5072c921//scratch/ucs2bccr231
    //vmfs/volumes/285a2ce0-5072c921//scratch/ucs2bccr232
    //vmfs/volumes/285a2ce0-5072c921//scratch/ucs2bccr233
    //vmfs/volumes/285a2ce0-5072c921//scratch/ucs2bccr234
    //vmfs/volumes/285a2ce0-5072c921//scratch/ucs2bccr235
    //vmfs/volumes/285a2ce0-5072c921//scratch/ucs2bccr236
    //vmfs/volumes/285a2ce0-5072c921//scratch/ucs2bccr237

    Basically this is what I am looking for, as I have created those directories, but as you can see it has // before scratch subdir.

    ReplyDelete
    Replies
    1. This works better.

      PowerCLI C:\Program Files (x86)\VMware\Infrastructure\vSphere PowerCLI> get-vmhost | foreach {echo "
      $($thisDataStore.ExtensionData.info.url.replace('ds://',''))scratch/$($_.name.split('.')[0])"}
      /vmfs/volumes/285a2ce0-5072c921/scratch/ucs2bccr241
      /vmfs/volumes/285a2ce0-5072c921/scratch/ucs2bccr242
      /vmfs/volumes/285a2ce0-5072c921/scratch/ucs2bccr227
      /vmfs/volumes/285a2ce0-5072c921/scratch/ucs2bccr238
      /vmfs/volumes/285a2ce0-5072c921/scratch/ucs2bccr239
      /vmfs/volumes/285a2ce0-5072c921/scratch/ucs2bccr240
      /vmfs/volumes/285a2ce0-5072c921/scratch/ucs2bccr225
      /vmfs/volumes/285a2ce0-5072c921/scratch/ucs2bccr226
      /vmfs/volumes/285a2ce0-5072c921/scratch/ucs2bccr228
      /vmfs/volumes/285a2ce0-5072c921/scratch/ucs2bccr229
      /vmfs/volumes/285a2ce0-5072c921/scratch/ucs2bccr230
      /vmfs/volumes/285a2ce0-5072c921/scratch/ucs2bccr231
      /vmfs/volumes/285a2ce0-5072c921/scratch/ucs2bccr232
      /vmfs/volumes/285a2ce0-5072c921/scratch/ucs2bccr233
      /vmfs/volumes/285a2ce0-5072c921/scratch/ucs2bccr234
      /vmfs/volumes/285a2ce0-5072c921/scratch/ucs2bccr235
      /vmfs/volumes/285a2ce0-5072c921/scratch/ucs2bccr236
      /vmfs/volumes/285a2ce0-5072c921/scratch/ucs2bccr237
      /vmfs/volumes/285a2ce0-5072c921/scratch/ucs2bccr243

      Delete
    2. Yay, it works now!!!
      get-vmhost | foreach {$_ | get-advancedsetting -name "ScratchConfig.ConfiguredScratchLocation" | set-advancedsetting -Value "$($thisDataStore.ExtensionData.info.url.replace('ds://',''))scratch/$($_.name.split('.')[0])" -confirm:$false}

      Worked just fine.
      Thanks a lot, Jason for your help. Really appreciate it. :)

      Delete
    3. Great, I'm glad to hear that you worked it out and thanks for sharing your fix! I wonder why my .ExtensionData.info.url didn't have the trailing slash on it, where yours did. Well, tomorrow I'll go through and add a .trimend("/") to my string so that it will remove any trailing slash that might be there, so that it will work in either situation.

      Delete
    4. Oh, I see what happened... you're putting all of the ESXi host subdirectories into a single "scratch" folder, whereas I made a separate folder for each ESXi host and put "scratch" beneath it. So, my script was basically "$host/scratch" and you initially reordered it into "/scratch/$host". The extra backslash came from adjusting the order of the parameters and your fix was to just get rid of it. I feel better about that - we're both getting the same string syntax out of the .ExtensionData.info.url parameter after all :)

      Delete
    5. It is a common NSF store, and I did not wanted to have 230 sub-directories in root :)
      Thanks for posing this script, it did save me a lot of time.
      Everything else, I could find on internet, was not so useful and straightforward.

      Delete

Post a Comment

Sorry guys, I've been getting a lot of spam recently, so I've had to turn on comment moderation. I'll do my best to moderate them swiftly after they're submitted,

Popular posts from this blog

Orphaned VMDK Files

Migrating from one vCenter to Another, Improved

Copying VM Folders and Permissions from One vCenter to Another