Content-Length 从 SslStream 读取响应时无法信任？

Question

在 .NET Core 2.2 上使用 TcpClient 和 NetworkStream。
尝试从 https://www.google.com/

获取内容

在继续之前，我想明确表示我不想使用 WebClient、HttpWebRequest 或 HttpClient 类。有很多问题是人们在使用 TcpClient 时遇到了一些问题，响应者或评论者建议使用其他东西来完成这项任务，所以请不要这样做。

假设我们有一个从 TcpClient 的 NetworkStream 获得并经过正确验证的 SslStream 实例。

假设还有一个 StreamWriter 用于将 HTTP 消息写入该流，还有一个 StreamReader 用于从响应中读取 HTTP 消息 header：

var tcpClient = new TcpClient("google.com", 443);
var stream = tcpClient.GetStream();
var sslStream = new SslStream(stream, false);
sslStream.AuthenticateAsClient("google.com");
var streamWriter = new StreamWriter(sslStream);
var streamReader = new StreamReader(sslStream);

假设我们发送请求的方式与 Firefox 浏览器发送请求的方式相同：

GET / HTTP/1.1
Host: www.google.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: sr,sr-RS;q=0.8,sr-CS;q=0.6,en-US;q=0.4,en;q=0.2
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Upgrade-Insecure-Requests: 1
Cache-Control: max-age=0

这会导致发送以下响应：

HTTP/1.1 200 OK
Date: Sun, 28 Apr 2019 17:28:27 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=UTF-8
Strict-Transport-Security: max-age=31536000
P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
Content-Encoding: br
Server: gws
Content-Length: 55786
... etc

现在，在使用 streamReader.ReadLine() 读取所有响应 header 并解析响应 header 中找到的内容长度后，让我们将响应内容读入缓冲区：

var totalBytesRead = 0;
int bytesRead;
var buffer = new byte[contentLength];
do
{
    bytesRead = sslStream.Read(buffer,
        totalBytesRead,
        contentLength - totalBytesRead);
    totalBytesRead += bytesRead;
} while (totalBytesRead < contentLength && bytesRead > 0);

然而，这个 do..while 循环只会在远程服务器关闭连接后退出，这意味着对 Read 的最后一次调用将挂起。这意味着我们已经读取了整个响应内容，并且服务器已经在该流上侦听另一条 HTTP 消息。 contentLength 不正确吗？调用 ReadLine 时 streamReader 是否读取过多，因此是否弄乱了 SslStream 位置，从而导致读取无效数据？

什么给了？有没有人有这方面的经验？

P.S。这是一个示例控制台应用程序代码，省略了所有安全检查，证明了这一点：

private static void Main(string[] args)
{
    using (var tcpClient = new TcpClient("google.com", 443))
    {
        var stream = tcpClient.GetStream();
        using (var sslStream = new SslStream(stream, false))
        {
            sslStream.AuthenticateAsClient("google.com");
            using (var streamReader = new StreamReader(sslStream))
            using (var streamWriter = new StreamWriter(sslStream))
            {
                streamWriter.WriteLine("GET / HTTP/1.1");
                streamWriter.WriteLine("Host: www.google.com");
                streamWriter.WriteLine("User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0");
                streamWriter.WriteLine("Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
                streamWriter.WriteLine("Accept-Language: sr,sr-RS;q=0.8,sr-CS;q=0.6,en-US;q=0.4,en;q=0.2");
                streamWriter.WriteLine("Accept-Encoding: gzip, deflate, br");
                streamWriter.WriteLine("Connection: keep-alive");
                streamWriter.WriteLine("Upgrade-Insecure-Requests: 1");
                streamWriter.WriteLine("Cache-Control: max-age=0");
                streamWriter.WriteLine();
                streamWriter.Flush();

                var lines = new List<string>();
                var line = streamReader.ReadLine();
                var contentLength = 0;
                while (!string.IsNullOrWhiteSpace(line))
                {
                    var split = line.Split(": ");
                    if (split.First() == "Content-Length")
                    {
                        contentLength = int.Parse(split[1]);
                    }

                    lines.Add(line);
                    line = streamReader.ReadLine();
                }

                var totalBytesRead = 0;
                int bytesRead;
                var buffer = new byte[contentLength];
                do
                {
                    bytesRead = sslStream.Read(buffer,
                        totalBytesRead,
                        contentLength - totalBytesRead);
                    totalBytesRead += bytesRead;
                    Console.WriteLine(
                        $"Bytes read: {totalBytesRead} of {contentLength} (last chunk: {bytesRead} bytes)");
                } while (totalBytesRead < contentLength && bytesRead > 0);

                Console.WriteLine(
                    "--------------------");
            }
        }
    }

    Console.ReadLine();
}

编辑

这总是在我提交问题后发生。我已经摸不着头脑好几天了，一直没能找到问题的原因，但是一提交，我就知道是 StreamReader 在尝试时把事情搞砸了读一行。

因此，如果我停止使用 StreamReader 并将对 ReadLine 的调用替换为 byte-by-byte，一切似乎都很好。替换代码可以这样写：

private static IEnumerable<string> ReadHeader(Stream sslStream)
{
    // One-byte buffer for reading bytes from the stream
    var buffer = new byte[1];

    // Initialize a four-character string to keep the last four bytes of the message
    var check = new StringBuilder("....");
    int bytes;
    var responseBuilder = new StringBuilder();
    do
    {
        // Read the next byte from the stream and write in into the buffer
        bytes = sslStream.Read(buffer, 0, 1);
        if (bytes == 0)
        {
            // If nothing was read, break the loop
            break;
        }

        // Add the received byte to the response builder.
        // We expect the header to be ASCII encoded so it's OK to just cast to char and append
        responseBuilder.Append((char) buffer[0]);

        // Always remove the first char from the string and append the latest received one
        check.Remove(0, 1);
        check.Append((char) buffer[0]);

        // \r\n\r\n marks the end of the message header, so break here
        if (check.ToString() == "\r\n\r\n")
        {
            break;
        }
    } while (bytes > 0);

    var headerText = responseBuilder.ToString();
    return headerText.Split("\r\n", StringSplitOptions.RemoveEmptyEntries);
}

...这将使我们的示例控制台应用程序看起来像这样：

private static void Main(string[] args)
{
    using (var tcpClient = new TcpClient("google.com", 443))
    {
        var stream = tcpClient.GetStream();
        using (var sslStream = new SslStream(stream, false))
        {
            sslStream.AuthenticateAsClient("google.com");
            using (var streamWriter = new StreamWriter(sslStream))
            {
                streamWriter.WriteLine("GET / HTTP/1.1");
                streamWriter.WriteLine("Host: www.google.com");
                streamWriter.WriteLine("User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0");
                streamWriter.WriteLine("Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
                streamWriter.WriteLine("Accept-Language: sr,sr-RS;q=0.8,sr-CS;q=0.6,en-US;q=0.4,en;q=0.2");
                streamWriter.WriteLine("Accept-Encoding: gzip, deflate, br");
                streamWriter.WriteLine("Connection: keep-alive");
                streamWriter.WriteLine("Upgrade-Insecure-Requests: 1");
                streamWriter.WriteLine("Cache-Control: max-age=0");
                streamWriter.WriteLine();
                streamWriter.Flush();

                var lines = ReadHeader(sslStream);
                var contentLengthLine = lines.First(x => x.StartsWith("Content-Length"));
                var split = contentLengthLine.Split(": ");
                var contentLength = int.Parse(split[1]);

                var totalBytesRead = 0;
                int bytesRead;
                var buffer = new byte[contentLength];
                do
                {
                    bytesRead = sslStream.Read(buffer,
                        totalBytesRead,
                        contentLength - totalBytesRead);
                    totalBytesRead += bytesRead;
                    Console.WriteLine(
                        $"Bytes read: {totalBytesRead} of {contentLength} (last chunk: {bytesRead} bytes)");
                } while (totalBytesRead < contentLength && bytesRead > 0);

                Console.WriteLine(
                    "--------------------");
            }
        }
    }

    Console.ReadLine();
}

private static IEnumerable<string> ReadHeader(Stream sslStream)
{
    // One-byte buffer for reading bytes from the stream
    var buffer = new byte[1];

    // Initialize a four-character string to keep the last four bytes of the message
    var check = new StringBuilder("....");
    int bytes;
    var responseBuilder = new StringBuilder();
    do
    {
        // Read the next byte from the stream and write in into the buffer
        bytes = sslStream.Read(buffer, 0, 1);
        if (bytes == 0)
        {
            // If nothing was read, break the loop
            break;
        }

        // Add the received byte to the response builder.
        // We expect the header to be ASCII encoded so it's OK to just cast to char and append
        responseBuilder.Append((char)buffer[0]);

        // Always remove the first char from the string and append the latest received one
        check.Remove(0, 1);
        check.Append((char)buffer[0]);

        // \r\n\r\n marks the end of the message header, so break here
        if (check.ToString() == "\r\n\r\n")
        {
            break;
        }
    } while (bytes > 0);

    var headerText = responseBuilder.ToString();
    return headerText.Split("\r\n", StringSplitOptions.RemoveEmptyEntries);
}

Answer 1

标题问题的答案是是。
它是可以信任的，只要您正确阅读消息header，即不要使用StreamReader.ReadLine。

这里有一个实用方法可以完成这项工作：

private static string ReadStreamUntil(Stream stream, string boundary)
{
    // One-byte buffer for reading bytes from the stream
    var buffer = new byte[1];

    // Initialize a string builder with some placeholder chars of the length as the boundary
    var boundaryPlaceholder = string.Join(string.Empty, boundary.Select(x => "."));
    var check = new StringBuilder(boundaryPlaceholder);
    var responseBuilder = new StringBuilder();
    do
    {
        // Read the next byte from the stream and write in into the buffer
        var byteCount = stream.Read(buffer, 0, 1);
        if (byteCount == 0)
        {
            // If nothing was read, break the loop
            break;
        }

        // Add the received byte to the response builder.
        responseBuilder.Append((char)buffer[0]);

        // Always remove the first char from the string and append the latest received one
        check.Remove(0, 1);
        check.Append((char)buffer[0]);

        // boundary marks the end of the message, so break here
    } while (check.ToString() != boundary);

    return responseBuilder.ToString();
}

然后，要读取header，我们只需调用ReadStreamUntil(sslStream, "\r\n\r\n")。

这里的关键是逐字节读取流，直到遇到已知的字节序列（在本例中\r\n\r\n）。

使用此方法读取后，流将位于正确的位置，以便正确读取响应内容。

如果有任何好处，可以通过调用 await ReadAsync 而不是 Read 轻松地将此方法转换为异步变体。

值得注意的是，上述方法只有在文本为ASCII 编码时才有效。

Content-Length 从 SslStream 读取响应时无法信任？

Content-Length can't be trusted when reading a response from SslStream?

c#

https

.net-core

tcpclient

sslstream