Powershell Script to start a lot of Robocopy Jobs

 We were recently helping a customer with a domain consolidation where we had to copy a bunch of files from several different domains over to a new file server.  We decided to break the task up into a few steps, the first of which was to simply get the data onto the destination!  To do that, we decided to use Robocopy, but we were going to need to copy data from several different domains without trust relationships, meaning that we'd have to get a bit creative with gaining access to the source data.

In the end, I put together a script!  This guy takes several important input parameters: foldersList, drivesList, and logFolder.

FoldersList is the path to a CSV that lists all of the folders that are being migrated.  This script ultimately depends on two columns in that CSV: SourceDrive and DestinationDrive, although we put SourceUNC and DesintationUNC fields in that file to help with later steps.  At its core, the FoldersList CSV is used to make the robocopy commands, defining the source and the destination for each command.  The interesting challenge here was figuring out how to deal with getting access to the various sources.

We did that with the drivesList csv file.  It needs three columns: DriveLetter, UNCPath, and Credentials.  We decided that the best way for us to give Robocopy the permissions that it needed was to map those source shares, so that we could specify appropriate accounts for each one.  For each drive share, that file needs a letter to use when mapping it, the UNC path of the share to map, and a path to a secured powershell credential object (as an XML file).  You probably don't have that last thing already built the first time you run the script, so if it can't find the file where it's been specified, it will prompt the operator for credentials and then save them to that file.

Once it has all of the drives mapped, it then goes through and kicks off a robocopy command for each folder specified in the foldersList csv!  We set the script to store robocopy logs in the folder specified by the logFolder parameter, but we quickly realized that those log files were difficult to use.  So, I built a little log parser at the bottom that goes through that file and gives the operator a set of grand totals for all of the robocopy jobs, highlighting errors in that summary file.  If there's something mysterious there, that should make it easier to find the exact error in the main log file.

And that's it!  We put together a few other scripts to read all of the NTFS permissions on the source side (and the groups that are used in those permissions) and then recreate those structures in the destination... but those are each quite complicated and so are topics for another time!

Check out the script on Github!

Comments

Post a Comment

Sorry guys, I've been getting a lot of spam recently, so I've had to turn on comment moderation. I'll do my best to moderate them swiftly after they're submitted,

Popular posts from this blog

Clone a Standard vSwitch from one ESXi Host to Another

One of my customers is standardizing the configurations on their HP C7000 enclosures (they've been set up at various sites by various administrators with varying involvement from the architecture team).  As such, we need to stand up some temporary resources so that we can take down the main enclosure for reconfiguration.  That's fine, we can easily ship a smaller enclosure to each site for temporary compute resources.  Some of the sites are using Standard vSwitches, so we need to be able to quickly copy the networking configuration over from their existing blades to these new, slightly different blades.  As I see it, we had 2 good options: 1) Capture a Host Profile with the desired vSwitch configuration.  Delete every other component of the Host Profile so that only the networking section is applied; design the new ESXi hosts so that the vSwitches'll work with the old vmnic-to-vswitch configurations. 2) Write a script to clone the vSwitch from ones ESXi host to another.
Read more

PowerShell Sorting by Multiple Columns

PowerShell's Sort-Object cmdlet is super useful, especially when preparing output for human consumption.  A few people have found my blog while looking for more information about its use, specifically while looking for how to sort by multiple columns (well, properties, technically).  I've never done so (much less written about it), so I hope those folks found answers elsewhere.  But, it got me curious... and it turns out that it's really easy. The Technet article on Sort-Object has the answer directly spelled out: just use commas to create a list of properties to sort by (in order to precedence).  Let's look at some examples!  First, prepare a variable with some good sortable data: $myData = get-eventlog System -newest 25 And then we can get to sorting!  Say you want to sort primarily by EntryType (Warning, Information, Error) and then by Source.  Easy-peasy: $myData | sort EntryType,Source How about if you want it to be in descending order?  Yeah, there&
Read more

Deleting Orphaned (AKA Zombie) VMDK Files

A problem that most Virtualization Administrators need to deal with is Orphaned VMDK files (or Zombie VMDK files, as some people call them).  These are Virtual Machine hard drive files that are just hanging out, taking up storage, but are attached to no VMs.  How can this happen?  Well, the most common way is when an administrator needs to delete a VMDK but isn't quite ready to commit... so, after pressing "remove", they select "remove from virtual machine" rather than "remove and delete files".  The intent is always good: "I want to be able to restore this if it turns out to be necessary... so I'll just come back and delete it next week if nothing breaks!"  But, of course, next week something else happens and the file doesn't get deleted.  After a little while, it turns into "which files did I rename, again?" and nothing seems to happen. So, how do you deal with these orphaned files?  Well, a few years ago I found a scrip
Read more