Post 在内容块上使用压缩时内容流得到缓冲

Question

首先是什么有效：（进一步描述我的问题）

客户端应用程序向服务器 WebAPI 发送一个长数据流。请求被分块 (request.Headers.TransferEncodingChunked = True) 并且数据被写入流 "on the fly".

数据由序列化对象组成，每个对象代表自定义 class Chunk。

一切正常，服务器可以在客户端完成写入之前开始读取流（为控制器禁用服务器输入缓冲）。

开始请求：

Public Async Function Transfer(authCookie As Cookie) As Task

    Using handler As New HttpClientHandler()
        handler.CookieContainer = New Net.CookieContainer
        handler.CookieContainer.Add(authCookie)

        'Client instance
        Using client As New HttpClient(handler), content As New Content(AddressOf WriteToStream)

            'Send stream asynchronously
            Dim request As New HttpRequestMessage(HttpMethod.Post, "https://myserver.net/api/datastream")
            request.Content = content
            request.Headers.TransferEncodingChunked = True

            Await client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead).ContinueWith(
                Async Function(requestTask)
                    Dim response As HttpResponseMessage = Nothing
                    Try
                        response = Await requestTask
                    Catch ex As Exception
                        'Exception writing the stream
                        Exit Function
                    End Try
                    Try
                        response.EnsureSuccessStatusCode()
                    Catch ex As Exception
                        'Exception on server ocurred
                    End Try

                    response.RequestMessage.Dispose()
                    response.Dispose()
                End Function)

        End Using
    End Using

End Function

Public Sub WriteToStream(str As IO.Stream)

    'Head-chunk
    head.attachedChunks.CompleteAdding()

    'Write head-chunk to stream
    Task.Run(
        Async Function()
            Await head.WriteToStream(str)
        End Function).Wait()

End Sub

HttpContent Class:

Public Class Content
    Inherits Net.Http.HttpContent

    Protected ReadOnly actionOfStream As Action(Of IO.Stream)

    Public Sub New(action As Action(Of IO.Stream))
        If action Is Nothing Then Throw New ArgumentNullException("action")
        actionOfStream = action
    End Sub

    Protected Overrides Function SerializeToStreamAsync(stream As IO.Stream, context As Net.TransportContext) As Task

        Return Task.Factory.StartNew(
            Sub(obj)
                Dim target As IO.Stream = DirectCast(obj, IO.Stream)
                actionOfStream(target)
            End Sub,
            stream)

    End Function

    Protected Overrides Function TryComputeLength(ByRef length As Long) As Boolean
        length = -1
        Return False
    End Function

End Class

方法 WriteToStream() 调用 Head-chunks WriteToStream()，它为其子块调用相同的方法：

Public Class Chunk

    'More properties here

    Public Property attachedChunks As BlockingCollection(Of Chunk)

    Public Overridable Async Function WriteToStream(str As IO.Stream) As Task

        Using serializationStream As New IO.MemoryStream

            'Serialize
            Serializer.Serialize(Of Chunk)(serializationStream, Me)

            'Write length
            Dim cnt As Integer = CInt(serializationStream.Length)
            Dim cntBuffer() As Byte = BitConverter.GetBytes(cnt)
            Await str.WriteAsync(cntBuffer, 0, cntBuffer.Length)

            'Write chunk
            serializationStream.Seek(0, IO.SeekOrigin.Begin)
            Await serializationStream.CopyToAsync(str)

        End Using

        Await str.FlushAsync()

        'Clearing and disposing stuff 
        '...

        'Write sub-chunks
        If attachedChunks IsNot Nothing Then
            For Each chunk As Chunk In attachedChunks.GetConsumingEnumerable
                Await chunk.WriteToStream(str)
            Next

            attachedChunks.Dispose()
        End If

        'Write ending mark
        If attachedChunksIndefiniteCount Then
            Await str.WriteAsync({0, 0, 0, 0}, 0, 4)
        End If

    End Function

End Class

到目前为止一切顺利。

当我开始使用像 GZipStream 这样的压缩来压缩我的数据块时，问题出现了：请求在客户端被缓冲，所以它接缝。

这是 Chunk-class 的 WriteToStream() 方法：

Public Class Chunk

    'More properties here

    Public Property attachedChunks As BlockingCollection(Of Chunk)

    Public Overridable Async Function WriteToStream(str As IO.Stream) As Task

        Using serializationStream As New IO.MemoryStream

            'Serialization and compression
            Using gZipStream As New IO.Compression.GZipStream(serializationStream, IO.Compression.CompressionMode.Compress, True)
                Using bufferStream As New IO.BufferedStream(gZipStream, 64 * 1024)

                    'Serialize
                    Serializer.Serialize(Of Chunk)(bufferStream, Me)

                End Using
            End Using

            'Write length
            Dim cnt As Integer = CInt(serializationStream.Length)
            Dim cntBuffer() As Byte = BitConverter.GetBytes(cnt)
            Await str.WriteAsync(cntBuffer, 0, cntBuffer.Length)

            'Write chunk
            serializationStream.Seek(0, IO.SeekOrigin.Begin)
            Await serializationStream.CopyToAsync(str)

        End Using

        Await str.FlushAsync()

        'Clearing and disposing stuff 
        '...

        'Write sub-chunks
        If attachedChunks IsNot Nothing Then
            For Each chunk As Chunk In attachedChunks.GetConsumingEnumerable
                Await chunk.WriteToStream(str)
            Next

            attachedChunks.Dispose()
        End If

        'Write ending mark
        If attachedChunksIndefiniteCount Then
            Await str.WriteAsync({0, 0, 0, 0}, 0, 4)
        End If

    End Function

End Class

直到所有数据写入后才发送请求。请求是否缓冲流？ GZipStream 是否对某种底层上下文有影响？我不知道是什么问题...

Answer 1

我发现了困扰我的问题：使用大量数据进行测试，请求在几个块之后开始发送数据。

请求有一定的缓冲区（您不能设置使用 HttpClient）。未压缩的内容很快就达到了这个限制，压缩的内容显然更晚。

看起来整个请求都被缓冲了，因为我的测试数据太短了。

Post 在内容块上使用压缩时内容流得到缓冲

Post content stream gets bufferend when using compression on content-block

.net

vb.net

compression

streaming

sendasync