下载网页es源最快的方法是什么？

Question

我需要下载很多网页的源代码。所以我需要尽快做到这一点。这是我的代码。

  private static async Task<string> downloadsource(string link)
  {                
     ServicePointManager.Expect100Continue = false;
     WebRequest req = WebRequest.Create(link);
     req.Proxy = null;
     req.Method = "GET";
     WebResponse res = await siteyeBaglantiTalebi.GetResponseAsync();
     StreamReader read = new StreamReader(res.GetResponseStream());
     return read.ReadToEnd();           
  }

  List<string> links = new List<string>(){... including some web page links};

  private static List<string> source_list(List<string> links)
  {
      List<string> sources = new List<string>();

      for (int i = 0; i < links.Count; i++)
      {
          Task<string> _task = downloadsource(links[i]);
          Console.WriteLine("Downloaded : " + i);
          sources.Add(_task.Result);
      }            

          return sources;
  }

我想知道这段代码是最快的方法还是可以改进。你能帮我吗？

Answer 1

您正在每个循环中执行 _task.Result 调用。如果你这样编码，你的代码将运行和你一个接一个地下载每一页一样快。

试试这个：

private async static Task<List<string>> source_list(List<string> links)
{
    List<Task<string>> sources = new List<Task<string>>();

    for (int i = 0; i < links.Count; i++)
    {
        Task<string> _task = downloadsource(links[i]);
        Console.WriteLine("Downloading : " + i);
        sources.Add(_task);
    }

    return (await Task.WhenAll(sources)).ToList();
}

这样就更好了：

private async static Task<string[]> source_list(List<string> links)
{
    return await Task.WhenAll(links.Select(l => downloadsource(l)));
}

我还清理了你的 downloadsource 方法：

private static async Task<string> downloadsource(string link)
{
    ServicePointManager.Expect100Continue = false;
    WebRequest req = WebRequest.Create(link);
    req.Proxy = null;
    req.Method = "GET";
    using (WebResponse res = await req.GetResponseAsync())
    {
        using (StreamReader read = new StreamReader(res.GetResponseStream()))
        {
            return read.ReadToEnd();
        }
    }
}

下载网页es源最快的方法是什么？

What is the fastest way to download web pages'es sources?

c#

multithreading

task

httpwebrequest

async-await