如何更快且异步地正确下载多个文件？

Question

我尝试使用来自

的脚本

http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.begingetresponse%28v=vs.110%29.aspx

但是当我尝试下载多个文件时，下载一个站点（同一站点）所花费的时间会根据请求增加，结果是：

retrive time: 1500 ms
retrive time: 1500 ms
retrive time: 2721 ms
retrive time: 4089 ms
retrive time: 5255 ms
retrive time: 5708 ms
retrive time: 6485 ms
retrive time: 6916 ms
retrive time: 7421 ms
retrive time: 8139 ms

这是我的代码（注意：httpwebrequest keepalive 已关闭）：

     static void Main()
    {

        for (int i = 0; i < 10; i++)
        {
            HttpWebRequest_BeginGetResponse req = new HttpWebRequest_BeginGetResponse();
            new Thread(new ThreadStart(req.start))
            {
                IsBackground=true

            }.Start();


        }
        Console.ReadLine();   
    }

这是我放秒表的地方

            IAsyncResult result =
              (IAsyncResult)myHttpWebRequest.BeginGetResponse(new AsyncCallback(RespCallback), myRequestState);
            ThreadPool.RegisterWaitForSingleObject(result.AsyncWaitHandle, new WaitOrTimerCallback(TimeoutCallback), myHttpWebRequest, DefaultTimeout, true);
            Stopwatch wtch = new Stopwatch();
            wtch.Start();
            allDone.WaitOne();
            wtch.Stop();
            Debug.WriteLine("retrive time: "+wtch.ElapsedMilliseconds+" ms");

执行此操作的正确方法是什么？

Answer 1

如果所有请求都命中同一个目标（URL 中的同一主机），那么您将命中 ServicePointManager.DefaultConnectionLimit，默认情况下为 2。

The default number of persistent connections (2) allowed on a ServicePoint object connected to an HTTP/1.1 or later server.

这意味着当您向同一主机启动 100 个 WebRequest 时，有 2 个被发布，98 个被排队。您可能想尝试增加该值。阅读 Understanding MaxServicePointIdleTime and DefaultConnectionLimit.

您的时间安排证实了这一点：只有前两个请求是真正并行的。然后后续请求被序列化并仅在先前请求完成时才开始，从而导致您看到的执行时间。

附带说明一下，默认值 (2) 是符合 RFC2616 的值：

Clients that use persistent connections SHOULD limit the number of simultaneous connections that they maintain to a given server. A single-user client SHOULD NOT maintain more than 2 connections with any server or proxy. A proxy SHOULD use up to 2*N connections to another server or proxy, where N is the number of simultaneously active users. These guidelines are intended to improve HTTP response times and avoid congestion.

如何更快且异步地正确下载多个文件？

how to correctly download multiple file faster and asynchronous?

c#

.net

asynchronous

httpwebrequest