通过 HTTP 上传大文件

Question

我正在尝试使用 PowerShell 将非常大的 VM 映像（5-15 Gb 大小）上传到 HTTP 服务器。

我尝试使用那几种方法（这里链接到script with net.WebClient.UploadFile and script with Invoke-webRequest）

它适用于小于 2GB 的文件，但不适用于大于 2GB 的文件。

我正在尝试直接使用 httpWebRequest，但无法将 FileStream 放入其中。

所以我的问题是：如何将文件流放入webrequest？

或者更一般地说：如何使用 PowerShell 通过 http 上传大文件？

$Timeout=10000000;
$fileName = "0.iso";
$data = "C:\$fileName";
$url = "http://nexus.lab.local:8081/nexus/content/sites/myproj/$fileName";
#$buffer = [System.IO.File]::Open("$data",[System.IO.Filemode]::Open, [System.IO.FileAccess]::Read) #Err Cannot convert argument "buffer", with value: "System.IO.FileStream", for "Write" to type "System.Byte[]": 
#$buffer = gc -en byte $data # too much space in memory 
$buffer = [System.IO.File]::ReadAllBytes($data) #Limit 2gb
[System.Net.HttpWebRequest] $webRequest = [System.Net.WebRequest]::Create($url)
$webRequest.Timeout = $timeout
$webRequest.Method = "POST"
$webRequest.ContentType = "application/data"
#$webRequest.ContentLength = $buffer.Length;
$webRequest.Credentials = New-Object System.Net.NetworkCredential("admin", "admin123");

$requestStream = $webRequest.GetRequestStream()
$requestStream.Write($buffer, 0, $buffer.Length)
$requestStream.Flush()
$requestStream.Close()

[System.Net.HttpWebResponse] $webResponse = $webRequest.GetResponse()
$streamReader = New-Object System.IO.StreamReader($webResponse.GetResponseStream())
$result = $streamReader.ReadToEnd()
return $result
$stream.Close()

Answer 1

默认情况下，HttpWebRequest 在内存中缓冲数据。只需将 HttpWebRequest.AllowWriteStreamBuffering 属性设置为 false，您就可以上传几乎任何大小的文件。在 msdn

查看更多详细信息

Answer 2

谢谢@Stoune，这是帮助我们获得最终工作解决方案的最后一件事。

还有一个，需要组织流文件读写webrequest buffer。它可能与那段代码有关：

$requestStream = $webRequest.GetRequestStream()
$fileStream = [System.IO.File]::OpenRead($file)
$chunk = New-Object byte[] $bufSize
  while( $bytesRead = $fileStream.Read($chunk,0,$bufsize) )
  {
    $requestStream.write($chunk, 0, $bytesRead)
    $requestStream.Flush()
  }

最终脚本如下所示：

$user = "admin"
$pass = "admin123"
$dir = "C:\Virtual Hard Disks"
$fileName = "win2012r2std.vhdx"
$file = "$dir/$fileName"
$url = "http://nexus.lab.local:8081/nexus/content/sites/myproj/$fileName"
$Timeout=10000000
$bufSize=10000

$cred = New-Object System.Net.NetworkCredential($user, $pass)

$webRequest = [System.Net.HttpWebRequest]::Create($url)
$webRequest.Timeout = $timeout
$webRequest.Method = "POST"
$webRequest.ContentType = "application/data"
$webRequest.AllowWriteStreamBuffering=$false
$webRequest.SendChunked=$true # needed by previous line
$webRequest.Credentials = $cred

$requestStream = $webRequest.GetRequestStream()
$fileStream = [System.IO.File]::OpenRead($file)
$chunk = New-Object byte[] $bufSize
  while( $bytesRead = $fileStream.Read($chunk,0,$bufsize) )
  {
    $requestStream.write($chunk, 0, $bytesRead)
    $requestStream.Flush()
  }

$responceStream = $webRequest.getresponse()
#$status = $webRequest.statuscode

$FileStream.Close()
$requestStream.Close()
$responceStream.Close()

$responceStream
$responceStream.GetResponseHeader("Content-Length") 
$responceStream.StatusCode
#$status

Answer 3

为了上传到 Sonatype Nexus3，我使用了下面的代码。我花了一些时间弄清楚，处理来自 Nexus3 的上传和响应以及上传和下载大文件（大于 2GB）。我们在 Nexus3 前面有 Apache 负责 https 连接。当 Apache 在 Nexus 前面时，使用基本身份验证可确保 Nexus 正确响应。通过 HEAD 发送预验证并对大文件使用分块上传修复了上传大文件没有过早结束的问题。

通过 Invoke-WebRequest 下载大文件也会失败并出现不同的错误。现在我通过 .Net 使用 WebRequest 并且它有效，请参阅 Download-File。

并且当代码运行通过自动化过程（来自 System Center Orchestrator）时，https 将失败。因此，当检测到方案 https 时，我们强制使用 TLS 1.2。

function New-HttpWebRequest
{
    <#
    .SYNOPSIS
    Creates a new [System.Net.HttpWebRequest] ready for file transmission.

    .DESCRIPTION
    Creates a new [System.Net.HttpWebRequest] ready for file transmission.
    The method will be Put. If the filesize is larger than the buffersize,
    the HttpWebRequest will be configured for chunked transfer.

    .PARAMETER Url
    Url to connect to.

    .PARAMETER Credential
    Credential for authentication at the Url resource.

    .EXAMPLE
    An example
    #>
    param(
        [Parameter(Mandatory=$true)]
        [ValidateNotNullOrEmpty()]
        [string]$Url,

        [Parameter(Mandatory=$true)]
        [ValidateNotNullOrEmpty()]
        [pscredential]$Credential,

        [Parameter(Mandatory=$true)]
        [long]$FileSize,

        [Parameter(Mandatory=$true)]
        [long]$BufferSize
    )

    $webRequest = [System.Net.HttpWebRequest]::Create($Url)
    $webRequest.Timeout = 600 * 1000;
    $webRequest.ReadWriteTimeout = 600 * 1000;
    $webRequest.ProtocolVersion = [System.Net.HttpVersion]::Version11;
    $webRequest.Method = "PUT";
    $webRequest.ContentType = "application/octet-stream";
    $webRequest.KeepAlive = $true;
    $webRequest.UserAgent = "<I use a specific UserAgent>";
    #$webRequest.UserAgent = 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)';
    $webRequest.PreAuthenticate = $true;
    $auth = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($Credential.UserName + ":" + $Credential.GetNetworkCredential().Password));
    $webRequest.Headers["Authorization"] = "Basic $auth"

    if (Get-UseChunkedUpload -FileSize $FileSize -BufferSize $BufferSize)
    {
        Write-Verbose "FileSize is greater than BufferSize, using chunked transfer.";
        $webRequest.AllowWriteStreamBuffering = $false;
        $webRequest.SendChunked = $true;
    }
    else
    {
        # Filesize is equal to or smaller than the BufferSize. The file will be transferred in one write.
        # Chunked cannot be used in this case.
        $webRequest.AllowWriteStreamBuffering = $true;
        $webRequest.SendChunked = $false;
        $webRequest.ContentLength = $FileSize;
    }

    return $webRequest;
}

function Get-BufferSize
{
    <#
    .SYNOPSIS
    Returns a buffer size that is 1% of ByteLength, rounded in whole MB's or at least AtLeast size.

    .DESCRIPTION
    Returns a buffer size that is 1% of ByteLength, rounded to whole MB's or if 1% is smaller than AtLeast, then AtLeast size is returned which is 1MB by default.

    .PARAMETER ByteLength
    Length of the bytes for which to calculate a valid buffer size.

    .PARAMETER AtLeast
    The minimum required buffer size, default 1MB.

    .EXAMPLE
    Get-BufferSize 4283304773

    Returns 42991616 which is 41MB.

    .EXAMPLE
    Get-BufferSize 4283304

    Returns 1048576 which is 1MB.

    .EXAMPLE
    Get-BufferSize 4283304 5MB

    Returns 5242880 which is 5MB.
    #>
    param(
        [Parameter(Mandatory=$true)]
        [long]$ByteLength,

        [long]$AtLeast = 1MB
    )

    [long]$size = $ByteLength / 100;
    if ($size -lt $AtLeast)
    {
        $size = $AtLeast;
    }
    else
    {
        $size = [Math]::Round($size / 1MB) * 1MB;
    }

    return $size;
}

function Get-UseChunkedUpload
{
    param(
        [Parameter(Mandatory=$true)]
        [long]$FileSize,

        [Parameter(Mandatory=$true)]
        [long]$BufferSize
    )

    return $FileSize -gt $BufferSize;
}

function Configure-Tls
{
    param(
        [Parameter(Mandatory=$true)]
        [ValidateNotNullOrEmpty()]
        [string]$Url
    )

    [System.Uri]$uri = $Url;
    if ($uri.Scheme -eq "https")
    {
        Write-Verbose "Using TLS 1.2";
        [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]::Tls12;
    }
}

function Send-PreAuthenticate
{
    param(
        [Parameter(Mandatory=$true)]
        [ValidateNotNullOrEmpty()]
        [string]$Url,

        [Parameter(Mandatory=$true)]
        [ValidateNotNullOrEmpty()]
        [pscredential]$Credential
    )

    $response = $null;
    try
    {
        [System.Uri]$uri = $Url;
        $repositoryAuthority = (($uri.GetLeftPart([System.UriPartial]::Authority)).TrimEnd('/') + '/');
        Write-Verbose "Send-PreAuthenticate - Sending HEAD to $repositoryAuthority";
        $wr = [System.Net.WebRequest]::Create($repositoryAuthority);
        $wr.Method = "HEAD";
        $wr.PreAuthenticate = $true;
        $auth = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($Credential.UserName + ":" + $Credential.GetNetworkCredential().Password));
        $wr.Headers["Authorization"] = "Basic $auth"
        $response = $wr.GetResponse();
    }
    finally
    {
        if ($response)
        {
            $response.Close();
            $response.Dispose();
            $response = $null;
        }
    }
}

function Upload-File
{
    <#
    .SYNOPSIS
    Uploads a file to the Nexus repository.

    .DESCRIPTION
    Uploads a file to the Nexus repository.
    If the file was uploaded successfully, the url via which the resource can be downloaded is returned.

    .PARAMETER Url
    The Url where the resource should be created.
    Please note that underscores and dots should be encoded, otherwise the Nexus repository does not accept the upload.

    .PARAMETER File
    The file that should be uploaded.

    .PARAMETER Credential
    Credential used for authentication at the Nexus repository.

    .EXAMPLE
    Upload-File -Url https://nexusrepo.domain.com/repository/repo-name/myfolder/myfile%2Eexe -File (Get-ChildItem .\myfile.exe) -Credential (Get-Credential)

    .OUTPUTS
    If the file was uploaded successfully, the url via which the resource can be downloaded.
    #>
    param(
        [Parameter(Mandatory=$true)]
        [ValidateNotNullOrEmpty()]
        [string]$Url,

        [Parameter(Mandatory=$true)]
        [ValidateNotNullOrEmpty()]
        [System.IO.FileInfo]$File,

        [Parameter(Mandatory=$true)]
        [ValidateNotNullOrEmpty()]
        [pscredential]$Credential
    )

    Write-Verbose "Upload-File Url:$Url"

    Configure-Tls -Url $Url;

    $fileSizeBytes = $File.Length;
    #$bufSize = Get-BufferSize $fileSizeBytes;
    $bufSize = 4 * 1MB;
    Write-Verbose ("FileSize is {0} bytes ({1:N0}MB). BufferSize is {2} bytes ({3:N0}MB)" -f $fileSizeBytes,($fileSizeBytes/1MB),$bufSize,($bufSize/1MB));
    if (Get-UseChunkedUpload -FileSize $fileSizeBytes -BufferSize $bufSize)
    {
        Write-Verbose "Using chunked upload. Send pre-auth first.";
        Send-PreAuthenticate -Url $Url -Credential $Credential;
    }

    $progressActivityMessage = ("Sending file {0} - {1} bytes" -f $File.Name, $File.Length);
    $webRequest = New-HttpWebRequest -Url $Url -Credential $Credential -FileSize $fileSizeBytes -BufferSize $bufSize;
    $chunk = New-Object byte[] $bufSize;
    $bytesWritten = 0;
    $fileStream = [System.IO.File]::OpenRead($File.FullName);
    $requestStream = $WebRequest.GetRequestStream();
    try
    {
        while($bytesRead = $fileStream.Read($chunk,0,$bufSize))
        {
            $requestStream.Write($chunk, 0, $bytesRead);
            $requestStream.Flush();
            $bytesWritten += $bytesRead;
            $progressStatusMessage = ("Sent {0} bytes - {1:N0}MB" -f $bytesWritten, ($bytesWritten / 1MB));
            Write-Progress -Activity $progressActivityMessage -Status $progressStatusMessage -PercentComplete ($bytesWritten/$fileSizeBytes*100);
        }
    }
    catch
    {
        throw;
    }
    finally
    {
        if ($fileStream)
        {
            $fileStream.Close();
        }
        if ($requestStream)
        {
            $requestStream.Close();
            $requestStream.Dispose();
            $requestStream = $null;
        }
        Write-Progress -Activity $progressActivityMessage -Completed;
    }

    # Read the response.
    $response = $null;
    try
    {
        $response = $webRequest.GetResponse();
        Write-Verbose ("{0} responded with {1} at {2}" -f $response.Server,$response.StatusCode,$response.ResponseUri);
        return $response.ResponseUri;
    }
    catch
    {
        if ($_.Exception.InnerException -and ($_.Exception.InnerException -like "*bad request*"))
        {
            throw ("ERROR: " + $_.Exception.InnerException.Message + " Possibly the file already exists or the content type of the file does not match the file extension. In that case, disable MIME type validation on the server.")
        }

        throw;
    }
    finally
    {
        if ($response)
        {
            $response.Close();
            $response.Dispose();
            $response = $null;
        }
        if ($webRequest)
        {
            $webRequest = $null;
        }
    }
}

function Download-File
{
    param(
        [Parameter(Mandatory=$true)]
        [ValidateNotNullOrEmpty()]
        [string]$Url,

        [Parameter(Mandatory=$true)]
        [ValidateNotNullOrEmpty()]
        [string]$FileName
    )

    $SDXDownloadType = @"
    using System.IO;
    using System.Net;

    public class SDXDownload
    {
        static public void DownloadFile(string Uri, string Filename)
        {
            HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(Uri);
            webRequest.Method = "GET";
            using (HttpWebResponse myHttpWebResponse = (HttpWebResponse)webRequest.GetResponse())
            using (Stream fileStream = File.OpenWrite(Filename))
            using (Stream streamResponse = myHttpWebResponse.GetResponseStream())
            {
                int bufSize = 64 * 1024;
                byte[] readBuff = new byte[bufSize];
                int bytesRead = streamResponse.Read(readBuff, 0, bufSize);
                while (bytesRead > 0)
                {
                    fileStream.Write(readBuff, 0, bytesRead);
                    bytesRead = streamResponse.Read(readBuff, 0, 256);
                }
            }
        }
    }
"@

    Configure-Tls -Url $Url;

    Add-Type -TypeDefinition $SDXDownloadType;
    [SDXDownload]::DownloadFile($Url, $FileName);
}

通过 HTTP 上传大文件

Upload BIG files via HTTP

powershell

file-upload

httpwebrequest