获取 HttpRequestExceptions：响应过早结束
Getting HttpRequestExceptions: The response ended prematurely
出于某种原因，我收到 HttpRequestException 消息“响应过早结束。我正在创建大约 500 个任务，这些任务使用我的 RateLimitedHttpClient 向网站发出请求，以便它可以抓取它。
从第 return await response.Content.ReadAsStringAsync(); 行抛出异常。
有 500 个任务，每个任务有大约 20 个要下载和解析的页面（总共大约 11000 个），我是否超出了 .Net 的 HttpClient 的能力？
public class SECScraper
{
public event EventHandler<ProgressChangedEventArgs> ProgressChangedEvent;
public SECScraper(EPSDownloader downloader, FinanceContext financeContext)
{
_downloader = downloader;
_financeContext = financeContext;
}
public void Download()
{
_numDownloaded = 0;
var companies = _financeContext.Companies.OrderBy(c => c.Name);
_interval = companies.Count() / 100;
var tasks = companies.Select(c => ScrapeSEC(c.CIK) ).ToList();
Task.WhenAll(tasks);
}
}
public class RateLimitedHttpClient : IHttpClient
{
public RateLimitedHttpClient(System.Net.Http.HttpClient client)
{
_client = client;
_client.Timeout = TimeSpan.FromMinutes(30);
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls;
}
public async Task<string> ReadAsync(string url)
{
if (!_sw.IsRunning)
_sw.Start();
await Delay();
using var response = await _client.GetAsync(url);
return await response.Content.ReadAsStringAsync();
}
private async Task Delay()
{
var totalElapsed = GetTimeElapsedSinceLastRequest();
while (totalElapsed < MinTimeBetweenRequests)
{
await Task.Delay(MinTimeBetweenRequests - totalElapsed);
totalElapsed = GetTimeElapsedSinceLastRequest();
};
_timeElapsedOfLastHttpRequest = (int)_sw.Elapsed.TotalMilliseconds;
}
private int GetTimeElapsedSinceLastRequest()
{
return (int)_sw.Elapsed.TotalMilliseconds - _timeElapsedOfLastHttpRequest;
}
private readonly System.Net.Http.HttpClient _client;
private readonly Stopwatch _sw = new Stopwatch();
private int _timeElapsedOfLastHttpRequest;
private const int MinTimeBetweenRequests = 100;
}
看来我在这里遇到了一些 HttpRequestExceptions。
System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: The response ended prematurely.
at System.Net.Http.HttpConnection.FillAsync()
at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at POLib.Http.RateLimitedHttpClient.ReadAsync(String url) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\Http\RateLimitedHttpClient.cs:line 23
at POLib.SECScraper.EPS.EPSDownloader.GetReportLinks(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 65
at POLib.SECScraper.EPS.EPSDownloader.GetEPSData(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 19
at POLib.SECScraper.SECScraper.ScrapeSEC(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\SECScraper.cs:line 40
System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine.. ---> System.Net.Sockets.SocketException (10053): An established connection was aborted by the software in your host machine.
--- End of inner exception stack trace ---
at System.Net.Security.SslStream.<FillBufferAsync>g__InternalFillBufferAsync|215_0[TReadAdapter](TReadAdapter adap, ValueTask`1 task, Int32 min, Int32 initial)
at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
at System.Net.Http.HttpConnection.FillAsync()
at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at POLib.Http.RateLimitedHttpClient.ReadAsync(String url) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\Http\RateLimitedHttpClient.cs:line 23
at POLib.SECScraper.EPS.EPSDownloader.GetReportLinks(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 65
at POLib.SECScraper.EPS.EPSDownloader.GetEPSData(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 19
at POLib.SECScraper.SECScraper.ScrapeSEC(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\SECScraper.cs:line 40
System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine.. ---> System.Net.Sockets.SocketException (10053): An established connection was aborted by the software in your host machine.
--- End of inner exception stack trace ---
at System.Net.Security.SslStream.<WriteSingleChunk>g__CompleteAsync|210_1[TWriteAdapter](ValueTask writeTask, Byte[] bufferToReturn)
at System.Net.Security.SslStream.WriteAsyncInternal[TWriteAdapter](TWriteAdapter writeAdapter, ReadOnlyMemory`1 buffer)
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at POLib.Http.RateLimitedHttpClient.ReadAsync(String url) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\Http\RateLimitedHttpClient.cs:line 23
at POLib.SECScraper.EPS.EPSDownloader.GetReportLinks(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 65
at POLib.SECScraper.EPS.EPSDownloader.GetEPSData(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 19
at POLib.SECScraper.SECScraper.ScrapeSEC(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\SECScraper.cs:line 40
System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: The response ended prematurely.
at System.Net.Http.HttpConnection.FillAsync()
at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at POLib.Http.RateLimitedHttpClient.ReadAsync(String url) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\Http\RateLimitedHttpClient.cs:line 23
at POLib.SECScraper.EPS.EPSDownloader.GetReportLinks(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 65
at POLib.SECScraper.EPS.EPSDownloader.GetEPSData(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 19
at POLib.SECScraper.SECScraper.ScrapeSEC(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\SECScraper.cs:line 40
System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: The response ended prematurely.
> at System.Net.Http.HttpConnection.FillAsync()
> at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)
> at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
> --- End of inner exception stack trace ---
> at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
> at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
> at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
> at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
> at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
> at POLib.Http.RateLimitedHttpClient.ReadAsync(String url) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\Http\RateLimitedHttpClient.cs:line 23
> at POLib.SECScraper.EPS.EPSDownloader.GetReportLinks(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 65
> at POLib.SECScraper.EPS.EPSDownloader.GetEPSData(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 19
> at POLib.SECScraper.SECScraper.ScrapeSEC(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\SECScraper.cs:line 40
> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine.. ---> System.Net.Sockets.SocketException (10053): An established connection was aborted by the software in your host machine.
> --- End of inner exception stack trace ---
> at System.Net.Security.SslStream.<FillBufferAsync>g__InternalFillBufferAsync|215_0[TReadAdapter](TReadAdapter adap, ValueTask`1 task, Int32 min, Int32 initial)
> at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
> at System.Net.Http.HttpConnection.FillAsync()
> at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)
> at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
> --- End of inner exception stack trace ---
> at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
> at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
> at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
> at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
> at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
> at POLib.Http.RateLimitedHttpClient.ReadAsync(String url) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\Http\RateLimitedHttpClient.cs:line 23
> at POLib.SECScraper.EPS.EPSDownloader.GetReportLinks(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 65
> at POLib.SECScraper.EPS.EPSDownloader.GetEPSData(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 19
> at POLib.SECScraper.SECScraper.ScrapeSEC(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\SECScraper.cs:line 40
> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine.. ---> System.Net.Sockets.SocketException (10053): An established connection was aborted by the software in your host machine.
> --- End of inner exception stack trace ---
> at System.Net.Security.SslStream.<WriteSingleChunk>g__CompleteAsync|210_1[TWriteAdapter](ValueTask writeTask, Byte[] bufferToReturn)
> at System.Net.Security.SslStream.WriteAsyncInternal[TWriteAdapter](TWriteAdapter writeAdapter, ReadOnlyMemory`1 buffer)
System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.IO.IOException: Authentication failed because the remote party has closed the transport stream.
at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.ProcessAuthentication(LazyAsyncResult lazyResult, CancellationToken cancellationToken)
at System.Net.Security.SslStream.BeginAuthenticateAsClient(SslClientAuthenticationOptions sslClientAuthenticationOptions, CancellationToken cancellationToken, AsyncCallback asyncCallback, Object asyncState)
at System.Net.Security.SslStream.<>c.<AuthenticateAsClientAsync>b__65_0(SslClientAuthenticationOptions arg1, CancellationToken arg2, AsyncCallback callback, Object state)
at System.Threading.Tasks.TaskFactory`1.FromAsyncImpl[TArg1,TArg2](Func`5 beginMethod, Func`2 endFunction, Action`1 endAction, TArg1 arg1, TArg2 arg2, Object state, TaskCreationOptions creationOptions)
at System.Threading.Tasks.TaskFactory.FromAsync[TArg1,TArg2](Func`5 beginMethod, Action`1 endMethod, TArg1 arg1, TArg2 arg2, Object state, TaskCreationOptions creationOptions)
at System.Threading.Tasks.TaskFactory.FromAsync[TArg1,TArg2](Func`5 beginMethod, Action`1 endMethod, TArg1 arg1, TArg2 arg2, Object state)
at System.Net.Security.SslStream.AuthenticateAsClientAsync(SslClientAuthenticationOptions sslClientAuthenticationOptions, CancellationToken cancellationToken)
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at POLib.Http.RateLimitedHttpClient.ReadAsync(String url) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\Http\RateLimitedHttpClient.cs:line 23
at POLib.SECScraper.EPS.EPSDownloader.GetReportLinks(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 65
at POLib.SECScraper.EPS.EPSDownloader.GetEPSData(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 19
at POLib.SECScraper.SECScraper.ScrapeSEC(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\SECScraper.cs:line 40
> at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.IO.IOException: Authentication failed because the remote party has closed the transport stream.
at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.ProcessAuthentication(LazyAsyncResult lazyResult, CancellationToken cancellationToken)
at System.Net.Security.SslStream.BeginAuthenticateAsClient(SslClientAuthenticationOptions sslClientAuthenticationOptions, CancellationToken cancellationToken, AsyncCallback asyncCallback, Object asyncState)
at System.Net.Security.SslStream.<>c.<AuthenticateAsClientAsync>b__65_0(SslClientAuthenticationOptions arg1, CancellationToken arg2, AsyncCallback callback, Object state)
at System.Threading.Tasks.TaskFactory`1.FromAsyncImpl[TArg1,TArg2](Func`5 beginMethod, Func`2 endFunction, Action`1 endAction, TArg1 arg1, TArg2 arg2, Object state, TaskCreationOptions creationOptions)
at System.Threading.Tasks.TaskFactory.FromAsync[TArg1,TArg2](Func`5 beginMethod, Action`1 endMethod, TArg1 arg1, TArg2 arg2, Object state, TaskCreationOptions creationOptions)
at System.Threading.Tasks.TaskFactory.FromAsync[TArg1,TArg2](Func`5 beginMethod, Action`1 endMethod, TArg1 arg1, TArg2 arg2, Object state)
at System.Net.Security.SslStream.AuthenticateAsClientAsync(SslClientAuthenticationOptions sslClientAuthenticationOptions, CancellationToken cancellationToken)
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at POLib.Http.RateLimitedHttpClient.ReadAsync(String url) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\Http\RateLimitedHttpClient.cs:line 23
at POLib.SECScraper.EPS.EPSDownloader.GetReportLinks(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 65
at POLib.SECScraper.EPS.EPSDownloader.GetEPSData(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 19
at POLib.SECScraper.SECScraper.ScrapeSEC(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\SECScraper.cs:line 40
> --- End of inner exception stack trace ---
System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.IO.IOException: Authentication failed because the remote party has closed the transport stream.
at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.ProcessAuthentication(LazyAsyncResult lazyResult, CancellationToken cancellationToken)
at System.Net.Security.SslStream.BeginAuthenticateAsClient(SslClientAuthenticationOptions sslClientAuthenticationOptions, CancellationToken cancellationToken, AsyncCallback asyncCallback, Object asyncState)
at System.Net.Security.SslStream.<>c.<AuthenticateAsClientAsync>b__65_0(SslClientAuthenticationOptions arg1, CancellationToken arg2, AsyncCallback callback, Object state)
at System.Threading.Tasks.TaskFactory`1.FromAsyncImpl[TArg1,TArg2](Func`5 beginMethod, Func`2 endFunction, Action`1 endAction, TArg1 arg1, TArg2 arg2, Object state, TaskCreationOptions creationOptions)
at System.Threading.Tasks.TaskFactory.FromAsync[TArg1,TArg2](Func`5 beginMethod, Action`1 endMethod, TArg1 arg1, TArg2 arg2, Object state, TaskCreationOptions creationOptions)
at System.Threading.Tasks.TaskFactory.FromAsync[TArg1,TArg2](Func`5 beginMethod, Action`1 endMethod, TArg1 arg1, TArg2 arg2, Object state)
at System.Net.Security.SslStream.AuthenticateAsClientAsync(SslClientAuthenticationOptions sslClientAuthenticationOptions, CancellationToken cancellationToken)
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at POLib.Http.RateLimitedHttpClient.ReadAsync(String url) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\Http\RateLimitedHttpClient.cs:line 23
at POLib.SECScraper.EPS.EPSDownloader.GetReportLinks(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 65
at POLib.SECScraper.EPS.EPSDownloader.GetEPSData(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 19
at POLib.SECScraper.SECScraper.ScrapeSEC(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\SECScraper.cs:line 40
你只需要继续挖掘。异常 "The response ended prematurely" 不是根本原因。继续挖掘内部异常，直到找到最后一个异常。你会发现：
System.IO.IOException: Authentication failed because the remote party has closed the transport stream.
所以这与您的代码无关。您正在访问的服务器似乎无法处理负载，或者因为您访问太重而故意放弃您的请求。
就我而言，问题出在 HTTP 版本中。当我切换到 HTTP/2.0 时，异常消失并且请求成功完成。
var message = new HttpRequestMessage(HttpMethod.Get, url) { Version = new Version(2, 0) };
var request = await httpClient.SendAsync(message);
我将主机和代理添加到 header，问题已解决。
var request = (HttpWebRequest)WebRequest.Create(url);
request.Host = request.RequestUri.Host; //"example.com"
request.UserAgent = "Dotnet";
出于某种原因，我收到 HttpRequestException 消息“响应过早结束。我正在创建大约 500 个任务，这些任务使用我的 RateLimitedHttpClient 向网站发出请求，以便它可以抓取它。
从第 return await response.Content.ReadAsStringAsync(); 行抛出异常。
有 500 个任务，每个任务有大约 20 个要下载和解析的页面（总共大约 11000 个），我是否超出了 .Net 的 HttpClient 的能力？
public class SECScraper
{
public event EventHandler<ProgressChangedEventArgs> ProgressChangedEvent;
public SECScraper(EPSDownloader downloader, FinanceContext financeContext)
{
_downloader = downloader;
_financeContext = financeContext;
}
public void Download()
{
_numDownloaded = 0;
var companies = _financeContext.Companies.OrderBy(c => c.Name);
_interval = companies.Count() / 100;
var tasks = companies.Select(c => ScrapeSEC(c.CIK) ).ToList();
Task.WhenAll(tasks);
}
}
public class RateLimitedHttpClient : IHttpClient
{
public RateLimitedHttpClient(System.Net.Http.HttpClient client)
{
_client = client;
_client.Timeout = TimeSpan.FromMinutes(30);
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls;
}
public async Task<string> ReadAsync(string url)
{
if (!_sw.IsRunning)
_sw.Start();
await Delay();
using var response = await _client.GetAsync(url);
return await response.Content.ReadAsStringAsync();
}
private async Task Delay()
{
var totalElapsed = GetTimeElapsedSinceLastRequest();
while (totalElapsed < MinTimeBetweenRequests)
{
await Task.Delay(MinTimeBetweenRequests - totalElapsed);
totalElapsed = GetTimeElapsedSinceLastRequest();
};
_timeElapsedOfLastHttpRequest = (int)_sw.Elapsed.TotalMilliseconds;
}
private int GetTimeElapsedSinceLastRequest()
{
return (int)_sw.Elapsed.TotalMilliseconds - _timeElapsedOfLastHttpRequest;
}
private readonly System.Net.Http.HttpClient _client;
private readonly Stopwatch _sw = new Stopwatch();
private int _timeElapsedOfLastHttpRequest;
private const int MinTimeBetweenRequests = 100;
}
看来我在这里遇到了一些 HttpRequestExceptions。
System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: The response ended prematurely.
at System.Net.Http.HttpConnection.FillAsync()
at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at POLib.Http.RateLimitedHttpClient.ReadAsync(String url) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\Http\RateLimitedHttpClient.cs:line 23
at POLib.SECScraper.EPS.EPSDownloader.GetReportLinks(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 65
at POLib.SECScraper.EPS.EPSDownloader.GetEPSData(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 19
at POLib.SECScraper.SECScraper.ScrapeSEC(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\SECScraper.cs:line 40
System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine.. ---> System.Net.Sockets.SocketException (10053): An established connection was aborted by the software in your host machine.
--- End of inner exception stack trace ---
at System.Net.Security.SslStream.<FillBufferAsync>g__InternalFillBufferAsync|215_0[TReadAdapter](TReadAdapter adap, ValueTask`1 task, Int32 min, Int32 initial)
at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
at System.Net.Http.HttpConnection.FillAsync()
at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at POLib.Http.RateLimitedHttpClient.ReadAsync(String url) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\Http\RateLimitedHttpClient.cs:line 23
at POLib.SECScraper.EPS.EPSDownloader.GetReportLinks(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 65
at POLib.SECScraper.EPS.EPSDownloader.GetEPSData(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 19
at POLib.SECScraper.SECScraper.ScrapeSEC(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\SECScraper.cs:line 40
System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine.. ---> System.Net.Sockets.SocketException (10053): An established connection was aborted by the software in your host machine.
--- End of inner exception stack trace ---
at System.Net.Security.SslStream.<WriteSingleChunk>g__CompleteAsync|210_1[TWriteAdapter](ValueTask writeTask, Byte[] bufferToReturn)
at System.Net.Security.SslStream.WriteAsyncInternal[TWriteAdapter](TWriteAdapter writeAdapter, ReadOnlyMemory`1 buffer)
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at POLib.Http.RateLimitedHttpClient.ReadAsync(String url) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\Http\RateLimitedHttpClient.cs:line 23
at POLib.SECScraper.EPS.EPSDownloader.GetReportLinks(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 65
at POLib.SECScraper.EPS.EPSDownloader.GetEPSData(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 19
at POLib.SECScraper.SECScraper.ScrapeSEC(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\SECScraper.cs:line 40
System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: The response ended prematurely.
at System.Net.Http.HttpConnection.FillAsync()
at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at POLib.Http.RateLimitedHttpClient.ReadAsync(String url) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\Http\RateLimitedHttpClient.cs:line 23
at POLib.SECScraper.EPS.EPSDownloader.GetReportLinks(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 65
at POLib.SECScraper.EPS.EPSDownloader.GetEPSData(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 19
at POLib.SECScraper.SECScraper.ScrapeSEC(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\SECScraper.cs:line 40
System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: The response ended prematurely.
> at System.Net.Http.HttpConnection.FillAsync()
> at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)
> at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
> --- End of inner exception stack trace ---
> at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
> at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
> at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
> at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
> at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
> at POLib.Http.RateLimitedHttpClient.ReadAsync(String url) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\Http\RateLimitedHttpClient.cs:line 23
> at POLib.SECScraper.EPS.EPSDownloader.GetReportLinks(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 65
> at POLib.SECScraper.EPS.EPSDownloader.GetEPSData(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 19
> at POLib.SECScraper.SECScraper.ScrapeSEC(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\SECScraper.cs:line 40
> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine.. ---> System.Net.Sockets.SocketException (10053): An established connection was aborted by the software in your host machine.
> --- End of inner exception stack trace ---
> at System.Net.Security.SslStream.<FillBufferAsync>g__InternalFillBufferAsync|215_0[TReadAdapter](TReadAdapter adap, ValueTask`1 task, Int32 min, Int32 initial)
> at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
> at System.Net.Http.HttpConnection.FillAsync()
> at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)
> at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
> --- End of inner exception stack trace ---
> at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
> at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
> at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
> at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
> at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
> at POLib.Http.RateLimitedHttpClient.ReadAsync(String url) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\Http\RateLimitedHttpClient.cs:line 23
> at POLib.SECScraper.EPS.EPSDownloader.GetReportLinks(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 65
> at POLib.SECScraper.EPS.EPSDownloader.GetEPSData(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 19
> at POLib.SECScraper.SECScraper.ScrapeSEC(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\SECScraper.cs:line 40
> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine.. ---> System.Net.Sockets.SocketException (10053): An established connection was aborted by the software in your host machine.
> --- End of inner exception stack trace ---
> at System.Net.Security.SslStream.<WriteSingleChunk>g__CompleteAsync|210_1[TWriteAdapter](ValueTask writeTask, Byte[] bufferToReturn)
> at System.Net.Security.SslStream.WriteAsyncInternal[TWriteAdapter](TWriteAdapter writeAdapter, ReadOnlyMemory`1 buffer)
System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.IO.IOException: Authentication failed because the remote party has closed the transport stream.
at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.ProcessAuthentication(LazyAsyncResult lazyResult, CancellationToken cancellationToken)
at System.Net.Security.SslStream.BeginAuthenticateAsClient(SslClientAuthenticationOptions sslClientAuthenticationOptions, CancellationToken cancellationToken, AsyncCallback asyncCallback, Object asyncState)
at System.Net.Security.SslStream.<>c.<AuthenticateAsClientAsync>b__65_0(SslClientAuthenticationOptions arg1, CancellationToken arg2, AsyncCallback callback, Object state)
at System.Threading.Tasks.TaskFactory`1.FromAsyncImpl[TArg1,TArg2](Func`5 beginMethod, Func`2 endFunction, Action`1 endAction, TArg1 arg1, TArg2 arg2, Object state, TaskCreationOptions creationOptions)
at System.Threading.Tasks.TaskFactory.FromAsync[TArg1,TArg2](Func`5 beginMethod, Action`1 endMethod, TArg1 arg1, TArg2 arg2, Object state, TaskCreationOptions creationOptions)
at System.Threading.Tasks.TaskFactory.FromAsync[TArg1,TArg2](Func`5 beginMethod, Action`1 endMethod, TArg1 arg1, TArg2 arg2, Object state)
at System.Net.Security.SslStream.AuthenticateAsClientAsync(SslClientAuthenticationOptions sslClientAuthenticationOptions, CancellationToken cancellationToken)
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at POLib.Http.RateLimitedHttpClient.ReadAsync(String url) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\Http\RateLimitedHttpClient.cs:line 23
at POLib.SECScraper.EPS.EPSDownloader.GetReportLinks(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 65
at POLib.SECScraper.EPS.EPSDownloader.GetEPSData(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 19
at POLib.SECScraper.SECScraper.ScrapeSEC(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\SECScraper.cs:line 40
> at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.IO.IOException: Authentication failed because the remote party has closed the transport stream.
at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.ProcessAuthentication(LazyAsyncResult lazyResult, CancellationToken cancellationToken)
at System.Net.Security.SslStream.BeginAuthenticateAsClient(SslClientAuthenticationOptions sslClientAuthenticationOptions, CancellationToken cancellationToken, AsyncCallback asyncCallback, Object asyncState)
at System.Net.Security.SslStream.<>c.<AuthenticateAsClientAsync>b__65_0(SslClientAuthenticationOptions arg1, CancellationToken arg2, AsyncCallback callback, Object state)
at System.Threading.Tasks.TaskFactory`1.FromAsyncImpl[TArg1,TArg2](Func`5 beginMethod, Func`2 endFunction, Action`1 endAction, TArg1 arg1, TArg2 arg2, Object state, TaskCreationOptions creationOptions)
at System.Threading.Tasks.TaskFactory.FromAsync[TArg1,TArg2](Func`5 beginMethod, Action`1 endMethod, TArg1 arg1, TArg2 arg2, Object state, TaskCreationOptions creationOptions)
at System.Threading.Tasks.TaskFactory.FromAsync[TArg1,TArg2](Func`5 beginMethod, Action`1 endMethod, TArg1 arg1, TArg2 arg2, Object state)
at System.Net.Security.SslStream.AuthenticateAsClientAsync(SslClientAuthenticationOptions sslClientAuthenticationOptions, CancellationToken cancellationToken)
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at POLib.Http.RateLimitedHttpClient.ReadAsync(String url) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\Http\RateLimitedHttpClient.cs:line 23
at POLib.SECScraper.EPS.EPSDownloader.GetReportLinks(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 65
at POLib.SECScraper.EPS.EPSDownloader.GetEPSData(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 19
at POLib.SECScraper.SECScraper.ScrapeSEC(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\SECScraper.cs:line 40
> --- End of inner exception stack trace ---
System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.IO.IOException: Authentication failed because the remote party has closed the transport stream.
at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.ProcessAuthentication(LazyAsyncResult lazyResult, CancellationToken cancellationToken)
at System.Net.Security.SslStream.BeginAuthenticateAsClient(SslClientAuthenticationOptions sslClientAuthenticationOptions, CancellationToken cancellationToken, AsyncCallback asyncCallback, Object asyncState)
at System.Net.Security.SslStream.<>c.<AuthenticateAsClientAsync>b__65_0(SslClientAuthenticationOptions arg1, CancellationToken arg2, AsyncCallback callback, Object state)
at System.Threading.Tasks.TaskFactory`1.FromAsyncImpl[TArg1,TArg2](Func`5 beginMethod, Func`2 endFunction, Action`1 endAction, TArg1 arg1, TArg2 arg2, Object state, TaskCreationOptions creationOptions)
at System.Threading.Tasks.TaskFactory.FromAsync[TArg1,TArg2](Func`5 beginMethod, Action`1 endMethod, TArg1 arg1, TArg2 arg2, Object state, TaskCreationOptions creationOptions)
at System.Threading.Tasks.TaskFactory.FromAsync[TArg1,TArg2](Func`5 beginMethod, Action`1 endMethod, TArg1 arg1, TArg2 arg2, Object state)
at System.Net.Security.SslStream.AuthenticateAsClientAsync(SslClientAuthenticationOptions sslClientAuthenticationOptions, CancellationToken cancellationToken)
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at POLib.Http.RateLimitedHttpClient.ReadAsync(String url) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\Http\RateLimitedHttpClient.cs:line 23
at POLib.SECScraper.EPS.EPSDownloader.GetReportLinks(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 65
at POLib.SECScraper.EPS.EPSDownloader.GetEPSData(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 19
at POLib.SECScraper.SECScraper.ScrapeSEC(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\SECScraper.cs:line 40
你只需要继续挖掘。异常 "The response ended prematurely" 不是根本原因。继续挖掘内部异常，直到找到最后一个异常。你会发现：
System.IO.IOException: Authentication failed because the remote party has closed the transport stream.
所以这与您的代码无关。您正在访问的服务器似乎无法处理负载，或者因为您访问太重而故意放弃您的请求。
就我而言，问题出在 HTTP 版本中。当我切换到 HTTP/2.0 时，异常消失并且请求成功完成。
var message = new HttpRequestMessage(HttpMethod.Get, url) { Version = new Version(2, 0) };
var request = await httpClient.SendAsync(message);
我将主机和代理添加到 header，问题已解决。
var request = (HttpWebRequest)WebRequest.Create(url);
request.Host = request.RequestUri.Host; //"example.com"
request.UserAgent = "Dotnet";