Setting up Persistent Scratch Space for an Arbitrary Number of ESXi Hosts
We've been configuring a large number of ESXi servers that don't have local hard drives. As such, we need to set up each system's scratch location on persistent storage, as per that KB article from VMware.
That KB Article has a great section devoted to performing the process from PowerCLI and it basically takes you through the whole thing, soup to nuts, for a single ESXi host. That's all well and good, but what if you've got 16 hosts? Or 32? Or 64? Well, with a few minor changes to the procedure that they've outlined, you can very easily target every ESXi host in an environment. Bear in mind that this process does require a host reboot, so make sure that you have your ducks in a row (and preferably do this before you place any VMs onto the cluster).
Instead of their step 6, I used this command to create a folder for all ESX hosts in the inventory:
get-vmhost | foreach {new-item ".$($_.name)/scratch" -itemtype directory}
Instead of their step 8, I used this command to change the setting on all ESX hosts in the inventory:
get-vmhost | Set-VMHostAdvancedConfiguration -name "ScratchConfig.ConfiguredScratchLocation" -Value "/vmfs/volumes/LUN-ID/.$($_.name)/scratch"
Make sure to change the path in there to whatever LUN you want to work with, as "LUN-ID" isn't likely to work in your environment.
If you're feeling masochistic (or just want to strip the domain suffix from the hostname), you can replace all of the
".$($_.name)/scratch" instances with ".$($_.name.split(".")[0])/scratch"
If you want to filter either command to hit only a subset of the hosts, you can change the "get-vmhost" command to something more specific like "get-vmhost esx01".
So, here's the assembled process in its entirety:
1) Launch PowerCLI
2) Connect to vCenter:
3) Get the list of datastores in the environment:
4) Mount the datastore through your PowerCLI session:
5) Move into the mounted datastore:
6) Create a folder for each ESXi Host:
7) If desired, check the current scratch location (I'm mostly including this to keep my numbers in line with the original article's numbers):
8) Set the Scratch location for the hosts:
9) Reboot the ESXi hosts.
Update: Since set-vmhostadvancedconfiguration is deprecated (although still works, at least for now), here's the new way to do step 8 (although I haven't actually done it this way yet):
Update 2: I've been doing this a lot and have optimized the process a bit. Here it is in 5 steps (after connecting the PowerCLI session to the vCenter server). Bear in mind that each of these is its own, single line... some of them have a lot of string manipulation in it to build the correct path for the scratch config.
1) Get the target Datastore
2) Mount the Datastore via PowerShell
3) Create the per-host scratch space folders on the datastore
4) Reconfigure the ESXi hosts to use those folders.
That KB Article has a great section devoted to performing the process from PowerCLI and it basically takes you through the whole thing, soup to nuts, for a single ESXi host. That's all well and good, but what if you've got 16 hosts? Or 32? Or 64? Well, with a few minor changes to the procedure that they've outlined, you can very easily target every ESXi host in an environment. Bear in mind that this process does require a host reboot, so make sure that you have your ducks in a row (and preferably do this before you place any VMs onto the cluster).
Instead of their step 6, I used this command to create a folder for all ESX hosts in the inventory:
get-vmhost | foreach {new-item ".$($_.name)/scratch" -itemtype directory}
Instead of their step 8, I used this command to change the setting on all ESX hosts in the inventory:
get-vmhost | Set-VMHostAdvancedConfiguration -name "ScratchConfig.ConfiguredScratchLocation" -Value "/vmfs/volumes/LUN-ID/.$($_.name)/scratch"
Make sure to change the path in there to whatever LUN you want to work with, as "LUN-ID" isn't likely to work in your environment.
If you're feeling masochistic (or just want to strip the domain suffix from the hostname), you can replace all of the
".$($_.name)/scratch" instances with ".$($_.name.split(".")[0])/scratch"
If you want to filter either command to hit only a subset of the hosts, you can change the "get-vmhost" command to something more specific like "get-vmhost esx01".
So, here's the assembled process in its entirety:
1) Launch PowerCLI
2) Connect to vCenter:
connect-viserver <vCenter name>
3) Get the list of datastores in the environment:
get-datastore
4) Mount the datastore through your PowerCLI session:
New-PSDrive -Name "DS" -Root \ -PSProvider VimDatastore -Datastore (Get-Datastore "<Datastore Name>")
5) Move into the mounted datastore:
cd DS:
6) Create a folder for each ESXi Host:
get-vmhost | foreach {new-item ".$($_.name)/scratch" -itemtype directory}
7) If desired, check the current scratch location (I'm mostly including this to keep my numbers in line with the original article's numbers):
get-vmhost | Get-VMHostAdvancedConfiguration -Name "ScratchConfig.ConfiguredScratchLocation"
8) Set the Scratch location for the hosts:
get-vmhost | Set-VMHostAdvancedConfiguration -name "ScratchConfig.ConfiguredScratchLocation" -Value "/vmfs/volumes/LUN-ID/.$($_.name)/scratch"
9) Reboot the ESXi hosts.
Update: Since set-vmhostadvancedconfiguration is deprecated (although still works, at least for now), here's the new way to do step 8 (although I haven't actually done it this way yet):
get-vmhost | get-advancedsetting -name "ScratchConfig.ConfiguredScratchLocation" | set-advancedsetting -value "/vmfs/volumes/LUN-ID/.$($_.entity)/scratch"
Update 2: I've been doing this a lot and have optimized the process a bit. Here it is in 5 steps (after connecting the PowerCLI session to the vCenter server). Bear in mind that each of these is its own, single line... some of them have a lot of string manipulation in it to build the correct path for the scratch config.
1) Get the target Datastore
$thisDataStore = Get-datastore LogDataStore
2) Mount the Datastore via PowerShell
New-PSDrive -Name "DS" -Root \ -PSProvider VimDatastore -Datastore $thisDataStore
3) Create the per-host scratch space folders on the datastore
get-vmhost | foreach {new-item "DS:\.$($_.name.split('.')[0])/scratch" -itemtype directory}
4) Reconfigure the ESXi hosts to use those folders.
get-vmhost | foreach {$_ | get-advancedsetting -name "ScratchConfig.ConfiguredScratchLocation" | set-advancedsetting -Value "$($thisDataStore.ExtensionData.info.url.replace('ds://','')).$($_.name.split('.')[0])/scratch" -confirm:$false}
5) Reboot the ESXi hosts.
Hi,
ReplyDeleteHow this supposed to look like?
($thisDataStore.ExtensionData.info.url.replace('ds://',''))
If you're ever unsure about what a section of code does, you can usually extract it and run it by yourself. If it's a command, VMware's really good about supporting the -WhatIf switch, which will make the command tell you what it would do but not actually do anything.
DeleteIn this case, that bit is just string manipulation. A Datastore object (in my script, the $thisDatastore variable) has a property called .ExtensionData.info.url which is the datastore's name, in a path format. Unfortunately, that path starts with ds:// which is not valid syntax for the ScratchConfig field that I want to put it into. So, I call the .replace() method on that string (that whole path, starting with the ds://), and replace those four characters with nothing.
If you look back at the original steps that I posted, that bit of code generates the "/vmfs/volumes/LUN-ID/" section of the command in the original step 8 (although, it obviously populates the LUN-ID for you).
Thanks Jason,
DeleteSomething I am doing wrong :)
PowerCLI ds:\scratch> $thisDataStore = Get-datastore EUC_NFS_LOG
PowerCLI ds:\scratch> get-vmhost | foreach {$_ | get-advancedsetting -name "ScratchConfig.Configured
ScratchLocation" | set-advancedsetting -Value "$($thisDataStore.ExtensionData.info.url.replace('ds:/
/','')).$($_.name.split('.')[0])" -confirm:$false}
set-advancedsetting : 28/10/2015 8:21:20 AM Set-AdvancedSetting A specified parameter
was not correct.
At line:1 char:98
+ get-vmhost | foreach {$_ | get-advancedsetting -name "ScratchConfig.ConfiguredSc ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Set-AdvancedSetting], InvalidArgument
+ FullyQualifiedErrorId : Client20_SystemManagementServiceImpl_GetVmHostAdvancedConfiguration_
ViError,VMware.VimAutomation.ViCore.Cmdlets.Commands.SetAdvancedSetting
set-advancedsetting : 28/10/2015 8:21:22 AM Set-AdvancedSetting A specified parameter
was not correct.
Try executing this line after you get your datastore:
Deleteget-vmhost | foreach {echo "$($thisDataStore.ExtensionData.info.url.replace('ds://','')).$($_.name.split('.')[0])/scratch"}
It should just echo out the /vmfs/volumes/... stuff. Let's make sure that it's working. It might be something simple, like PowerShell version. You can use use $psversiontable.psversion to see what version you have installed; I write everything using version 4+
Major Minor Build Revision
Delete----- ----- ----- --------
4 0 -1 -1
Hi Jason,
ReplyDeleteI have tried this:
PowerCLI C:\Program Files (x86)\VMware\Infrastructure\vSphere PowerCLI> get-vmhost | foreach {echo "
$($thisDataStore.ExtensionData.info.url.replace('ds:/',''))/scratch/$($_.name.split('.')[0])"}
//vmfs/volumes/285a2ce0-5072c921//scratch/ucs2bccr241
//vmfs/volumes/285a2ce0-5072c921//scratch/ucs2bccr242
//vmfs/volumes/285a2ce0-5072c921//scratch/ucs2bccr227
//vmfs/volumes/285a2ce0-5072c921//scratch/ucs2bccr238
//vmfs/volumes/285a2ce0-5072c921//scratch/ucs2bccr239
//vmfs/volumes/285a2ce0-5072c921//scratch/ucs2bccr240
//vmfs/volumes/285a2ce0-5072c921//scratch/ucs2bccr225
//vmfs/volumes/285a2ce0-5072c921//scratch/ucs2bccr226
//vmfs/volumes/285a2ce0-5072c921//scratch/ucs2bccr228
//vmfs/volumes/285a2ce0-5072c921//scratch/ucs2bccr229
//vmfs/volumes/285a2ce0-5072c921//scratch/ucs2bccr230
//vmfs/volumes/285a2ce0-5072c921//scratch/ucs2bccr231
//vmfs/volumes/285a2ce0-5072c921//scratch/ucs2bccr232
//vmfs/volumes/285a2ce0-5072c921//scratch/ucs2bccr233
//vmfs/volumes/285a2ce0-5072c921//scratch/ucs2bccr234
//vmfs/volumes/285a2ce0-5072c921//scratch/ucs2bccr235
//vmfs/volumes/285a2ce0-5072c921//scratch/ucs2bccr236
//vmfs/volumes/285a2ce0-5072c921//scratch/ucs2bccr237
Basically this is what I am looking for, as I have created those directories, but as you can see it has // before scratch subdir.
This works better.
DeletePowerCLI C:\Program Files (x86)\VMware\Infrastructure\vSphere PowerCLI> get-vmhost | foreach {echo "
$($thisDataStore.ExtensionData.info.url.replace('ds://',''))scratch/$($_.name.split('.')[0])"}
/vmfs/volumes/285a2ce0-5072c921/scratch/ucs2bccr241
/vmfs/volumes/285a2ce0-5072c921/scratch/ucs2bccr242
/vmfs/volumes/285a2ce0-5072c921/scratch/ucs2bccr227
/vmfs/volumes/285a2ce0-5072c921/scratch/ucs2bccr238
/vmfs/volumes/285a2ce0-5072c921/scratch/ucs2bccr239
/vmfs/volumes/285a2ce0-5072c921/scratch/ucs2bccr240
/vmfs/volumes/285a2ce0-5072c921/scratch/ucs2bccr225
/vmfs/volumes/285a2ce0-5072c921/scratch/ucs2bccr226
/vmfs/volumes/285a2ce0-5072c921/scratch/ucs2bccr228
/vmfs/volumes/285a2ce0-5072c921/scratch/ucs2bccr229
/vmfs/volumes/285a2ce0-5072c921/scratch/ucs2bccr230
/vmfs/volumes/285a2ce0-5072c921/scratch/ucs2bccr231
/vmfs/volumes/285a2ce0-5072c921/scratch/ucs2bccr232
/vmfs/volumes/285a2ce0-5072c921/scratch/ucs2bccr233
/vmfs/volumes/285a2ce0-5072c921/scratch/ucs2bccr234
/vmfs/volumes/285a2ce0-5072c921/scratch/ucs2bccr235
/vmfs/volumes/285a2ce0-5072c921/scratch/ucs2bccr236
/vmfs/volumes/285a2ce0-5072c921/scratch/ucs2bccr237
/vmfs/volumes/285a2ce0-5072c921/scratch/ucs2bccr243
Yay, it works now!!!
Deleteget-vmhost | foreach {$_ | get-advancedsetting -name "ScratchConfig.ConfiguredScratchLocation" | set-advancedsetting -Value "$($thisDataStore.ExtensionData.info.url.replace('ds://',''))scratch/$($_.name.split('.')[0])" -confirm:$false}
Worked just fine.
Thanks a lot, Jason for your help. Really appreciate it. :)
Great, I'm glad to hear that you worked it out and thanks for sharing your fix! I wonder why my .ExtensionData.info.url didn't have the trailing slash on it, where yours did. Well, tomorrow I'll go through and add a .trimend("/") to my string so that it will remove any trailing slash that might be there, so that it will work in either situation.
DeleteOh, I see what happened... you're putting all of the ESXi host subdirectories into a single "scratch" folder, whereas I made a separate folder for each ESXi host and put "scratch" beneath it. So, my script was basically "$host/scratch" and you initially reordered it into "/scratch/$host". The extra backslash came from adjusting the order of the parameters and your fix was to just get rid of it. I feel better about that - we're both getting the same string syntax out of the .ExtensionData.info.url parameter after all :)
DeleteIt is a common NSF store, and I did not wanted to have 230 sub-directories in root :)
DeleteThanks for posing this script, it did save me a lot of time.
Everything else, I could find on internet, was not so useful and straightforward.