在 Powershell 中使用 Start-ThreadJob 复制项目

Question

关闭此线程的背面：我有以下内容：

@mklement0 的方法（从复制并由修改）有效，但因为它为每个文件创建一个线程非常慢，并且在我的测试系统上使用 ~14,000 个文件消耗 > 4GB 内存:

# This works but is INCREDIBLY SLOW because it creates a thread per file
 Create sample CSV file with 10 rows.
 $FileList = Join-Path ([IO.Path]::GetTempPath()) "tmp.$PID.csv"
 @'
 Foo,SrcFileName,DestFileName,Bar
 1,c:\tmp\a,\server\share\a,baz
 2,c:\tmp\b,\server\share\b,baz
 3,c:\tmp\c,\server\share\c,baz
 4,c:\tmp\d,\server\share\d,baz
 5,c:\tmp\e,\server\share\e,baz
 6,c:\tmp\f,\server\share\f,baz
 7,c:\tmp\g,\server\share\g,baz
 8,c:\tmp\h,\server\share\h,baz
 9,c:\tmp\i,\server\share\i,baz
 10,c:\tmp\j,\server\share\j,baz
 '@ | Set-Content $FileList

# How many threads at most to run concurrently.
 $NumCopyThreads = 8

Write-Host 'Creating jobs...'
$dtStart = [datetime]::UtcNow

# Import the CSV data and transform it to [pscustomobject] instances
# with only .SrcFileName and .DestFileName properties - they take
# the place of your original [fileToCopy] instances.
$jobs = Import-Csv $FileList | Select-Object SrcFileName, DestFileName | 
  ForEach-Object {
    # Start the thread job for the file pair at hand.
    Start-ThreadJob -ThrottleLimit $NumCopyThreads -ArgumentList $_ { 
        param($f) 
        [System.IO.Fileinfo]$DestinationFilePath = $f.DestFileName
        [String]$DestinationDir = $DestinationFilePath.DirectoryName
        if (-not (Test-path([Management.Automation.WildcardPattern]::Escape($DestinationDir)))) {
            new-item -Path $DestinationDir -ItemType Directory #-Verbose
        }
        copy-item -path $f.srcFileName -Destination $f.destFilename
        "Copied $($f.SrcFileName) to $($f.DestFileName)"
    }
  }

Write-Host "Waiting for $($jobs.Count) jobs to complete..."

# Synchronously wait for all jobs (threads) to finish and output their results
# *as they become available*, then remove the jobs.
# NOTE: Output will typically NOT be in input order.
Receive-Job -Job $jobs -Wait -AutoRemoveJob
Write-Host "Total time lapsed: $([datetime]::UtcNow - $dtStart)"

# Clean up the temp. file
Remove-Item $FileList

This article（尤其是 PowerShell 作业部分）给了我将完整列表分成 1000 个文件的批次的想法，当它在我的测试用例中运行时，我得到 15 个线程（因为我有~14,500 个文件），但线程只处理每个 "chunk" 中的第一个文件，然后停止：

<#
.SYNOPSIS
<Brief description>
For examples type:
Get-Help .\<filename>.ps1 -examples
.DESCRIPTION
Copys files from one path to another
.PARAMETER FileList
e.g. C:\path\to\list\of\files\to\copy.txt
.PARAMETER NumCopyThreads
default is 8 (but can be 100 if you want to stress the machine to maximum!)
.PARAMETER LogName
default is output.csv located in the same path as the Filelist
.EXAMPLE
to run using defaults just call this file:
.\CopyFilesToBackup
to run using anything else use this syntax:
.\CopyFilesToBackup -filelist C:\path\to\list\of\files\to\copy.txt -NumCopyThreads 20 -LogName C:\temp\backup.log -CopyMethod Runspace
.\CopyFilesToBackup -FileList .\copytest.csv -NumCopyThreads 30 -Verbose
.NOTES
#>

[CmdletBinding()] 
Param( 
    [String] $FileList = "C:\temp\copytest.csv", 
    [int] $NumCopyThreads = 8,
    [String] $LogName
) 

$filesPerBatch = 1000

$files = Import-Csv $FileList | Select-Object SrcFileName, DestFileName

$i = 0
$j = $filesPerBatch - 1
$batch = 1

Write-Host 'Creating jobs...'
$dtStart = [datetime]::UtcNow

$jobs = while ($i -lt $files.Count) {
    $fileBatch = $files[$i..$j]

    $jobName = "Batch$batch"
    Start-ThreadJob -Name $jobName -ThrottleLimit $NumCopyThreads -ArgumentList ($fileBatch) -ScriptBlock {
        param($filesInBatch)
        foreach ($f in $filesInBatch) {
            [System.IO.Fileinfo]$DestinationFilePath = $f.DestFileName
            [String]$DestinationDir = $DestinationFilePath.DirectoryName
            if (-not (Test-path([Management.Automation.WildcardPattern]::Escape($DestinationDir)))) {
                new-item -Path $DestinationDir -ItemType Directory -Verbose
            }
            copy-item -path $f.srcFileName -Destination $f.DestFileName -Verbose
        }
    } 

    $batch += 1
    $i = $j + 1
    $j += $filesPerBatch

    if ($i -gt $files.Count) {$i = $files.Count}
    if ($j -gt $files.Count) {$j = $files.Count}
}

Write-Host "Waiting for $($jobs.Count) jobs to complete..."

Receive-Job -Job $jobs -Wait -AutoRemoveJob
Write-Host "Total time lapsed: $([datetime]::UtcNow - $dtStart)"

我觉得我错过了一些明显的东西，但我不知道是什么。

有人能帮忙吗？

Answer 1

变化：

Start-ThreadJob -Name $jobName -ThrottleLimit $NumCopyThreads -ArgumentList ($fileBatch) -ScriptBlock {

至

Start-ThreadJob -Name $jobName -ThrottleLimit $NumCopyThreads -ArgumentList (,$fileBatch) -ScriptBlock {

注意参数列表中 $fileBatch 之前的逗号。

之所以修复它是因为 ArgumentList 需要一个数组并将每个元素提供给参数。您试图将整个数组传递给第一个参数，这意味着您必须将数组放在数组中。

显然（这对我来说是个新闻），Powershell 会很乐意将您的字符串视为 foreach 循环中的单个项目数组，这就是每批处理第一项的原因。

Answer 2

因此，经过一周的反复试验才达到这一点，总的来说，我对结果非常满意。我将在下面分享的脚本负责处理我正在处理的文件的大约 3 个步骤：

创建文件夹
将文件复制到新文件夹
验证文件复制无误

在 Excel（使用 FileSystemObject 复制文件）

中执行步骤 1) 和 2) 所花费的时间不到 1/3

.SYNOPSIS
<Brief description>
For examples type:
Get-Help .\<filename>.ps1 -examples
.DESCRIPTION
Copys files from one path to another
.PARAMETER FileList
e.g. C:\path\to\list\of\files\to\copy.txt
.PARAMETER NumCopyThreads
default is 8 (but can be 100 if you want to stress the machine to maximum!)
.PARAMETER FilesPerBatch
default is 1000 this can be tweaked if performance becomes an issue because the Threading will HAMMER any network you run it on.
.PARAMETER LogName
Desired log file output. Must include full or relative (.\blah) path. If blank, location of FileList is used.
.PARAMETER DryRun
Boolean value denoting whether we're testing this thing or not. (Default is $false)
.PARAMETER DryRunNum
The number of files to Dry Run. (Default is 100)
.EXAMPLE
to run using defaults just call this file:
.\CopyFilesToBackup
to run using anything else use this syntax:
.\CopyFilesToBackup -filelist C:\path\to\list\of\files\to\copy.txt -NumCopyThreads 20 -LogName C:\temp\backup.log -CopyMethod Runspace
.\CopyFilesToBackup -FileList .\copytest.csv -NumCopyThreads 30 -Verbose
.NOTES
#>

[CmdletBinding()] 
Param( 
    [String] $FileList = "C:\temp\copytest.csv", 
    [int] $NumCopyThreads =75,
    [String] $JobName,
    [int] $FilesPerBatch = 1000,
    [String] $LogName,
    [Boolean] $DryRun = $false, #$true,
    [int] $DryRunNum = 100
) 



Write-Host 'Creating log file if it does not exist...'

function CreateFile([string]$filepath) {
    if (-not (Test-path([Management.Automation.WildcardPattern]::Escape($filepath)))) {
        new-item -Path $filepath -ItemType File
    }
    if (-not (Test-path([Management.Automation.WildcardPattern]::Escape($filepath)))) {
        return $false
    } else {
        return $true
    }
}

$dtStart = [datetime]::UtcNow

if ($LogName -eq "") {
    [System.IO.Fileinfo]$CsvPath = $FileList
    [String]$LogDirectory = $CsvPath.DirectoryName
    [string]$LognameBaseName = $CsvPath.BaseName
    $LogName = $LogDirectory + "\" + $LognameBaseName + ".log"
    if (-not (CreateFile($LogName)) ) { 
        write-host "Unable to create log, exiting now!"
        Break
    }
}
else {
    if (-not (CreateFile($LogName)) ) { 
        write-host "Unable to create log, exiting now!"
        Break
    }
}

Add-Content -Path $LogName -Value "[INFO],[Src Filename],[Src Hash],[Dest Filename],[Dest Hash]"

Write-Host 'Loading CSV data into memory...'

$files = Import-Csv $FileList | Select-Object SrcFileName, DestFileName

Write-Host 'CSV Data loaded...'

Write-Host 'Collecting unique Directory Names...'

$allFolders = New-Object "System.Collections.Generic.List[PSCustomObject]"

ForEach ($f in $files) {
    [System.IO.Fileinfo]$DestinationFilePath = $f.DestFileName
    [String]$DestinationDir = $DestinationFilePath.DirectoryName
    $allFolders.add($DestinationDir)
}

$folders = $allFolders | get-unique

Write-Host 'Creating Directories...'
foreach($DestinationDir in $folders) {
    if (-not (Test-path([Management.Automation.WildcardPattern]::Escape($DestinationDir)))) {
        new-item -Path $DestinationDir -ItemType Directory | Out-Null #-Verbose
    }
}
Write-Host 'Finished Creating Directories...'
$scriptBlock = {
    param(
        [PSCustomObject]$filesInBatch, 
        [String]$LogFileName)
        function ProcessFileAndHashToLog {
            param( [String]$LogFileName, [PSCustomObject]$FileColl)
            foreach ($f in $FileColl) {
                $mutex = New-object -typename 'Threading.Mutex' -ArgumentList $false, 'MyInterProcMutex'
                # [System.IO.Fileinfo]$DestinationFilePath = $f.DestFileName
                # [String]$DestinationDir = $DestinationFilePath.DirectoryName
                # if (-not (Test-path([Management.Automation.WildcardPattern]::Escape($DestinationDir)))) {
                #     new-item -Path $DestinationDir -ItemType Directory | Out-Null #-Verbose
                # }
                copy-item -path $f.srcFileName -Destination $f.DestFileName | Out-Null #-Verbose

                $srcHash = (Get-FileHash -Path $f.srcFileName -Algorithm SHA1).Hash #| Out-Null #could also use MD5 here but it needs testing
                if (Test-path([Management.Automation.WildcardPattern]::Escape($f.destFileName))) {
                    $destHash = (Get-FileHash -Path $f.destFileName -Algorithm SHA1).Hash #| Out-Null #could also use MD5 here but it needs testing
                } else {
                    $destHash = $f.destFileName + " not found at location."
                }
                if (-not ($null -eq $destHash) -and -not ($null -eq $srcHash)) {
                    $info = $f.srcFileName + "," + $srcHash + "," + $f.destFileName + "," + $destHash
                }
                $mutex.WaitOne() | Out-Null
                $DateTime = Get-date -Format "yyyy-MM-dd HH:mm:ss:fff"
                if ($DryRun) { Write-Host 'Writing to log file: '$LogFileName'...' }
                Add-Content -Path $LogFileName -Value "$DateTime,$Info"
                $mutex.ReleaseMutex() | Out-Null
            }
        }
        ProcessFileAndHashToLog -LogFileName $LogFileName -FileColl $filesInBatch
}

$i = 0
$j = $filesPerBatch - 1
$batch = 1
Write-Host 'Creating jobs...'
if (-not ($DryRun)) {
    $jobs = while ($i -lt $files.Count) {
        $fileBatch = $files[$i..$j]
        Start-ThreadJob -Name $jobName -ArgumentList $fileBatch, $LogName -ScriptBlock $scriptBlock #-ThrottleLimit $NumCopyThreads -ArgumentList $fileBatch, $LogName -ScriptBlock $scriptBlock
        $batch += 1
        $i = $j + 1
        $j += $filesPerBatch
        if ($i -gt $files.Count) {$i = $files.Count}
        if ($j -gt $files.Count) {$j = $files.Count}
    }
    Write-Host "Waiting for $($jobs.Count) jobs to complete..."
    Receive-Job -Job $jobs -Wait -AutoRemoveJob
} else {
    Write-Host 'Going in Dry...'
    $DummyFileBatch = $files[$i..$DryRunNum]
    & $scriptBlock -filesInBatch $DummyFileBatch -LogFileName $LogName
    Write-Host 'That wasn''t so bad was it..?'
}

Write-Host "Total time lapsed: $([datetime]::UtcNow - $dtStart)"

（我很乐意接受改进上述解决方案的建议。）

在 Powershell 中使用 Start-ThreadJob 复制项目

Copy-item using Start-ThreadJob in Powershell

powershell

multithreading

copy-item