C# 在控制台应用程序上并行异步下载多个文件时遇到问题

Question

在你们大发雷霆之前，我已经花了两天时间研究这个问题，观看关于异步编程的 youtube 教程，浏览类似的 Whosebug posts 等，我我一辈子都想不出如何将文件的异步并行下载应用到我的项目中。

首先，一些背景：

我正在创建一个程序，当用户输入查询时，将调用 twitch API 并下载剪辑。

我的程序分为两部分

1- 生成 .json 文件的网络抓取工具，其中包含下载文件所需的所有详细信息和

2 - 下载器。

第 1 部分工作得很好并且生成 .json 文件没有问题。

My Downloader 包含对 Data class 的引用，它是常用属性和方法的处理程序，例如我的 ClientID、Authentication、OutputPath， JsonFile、QueryURL。它还包含为这些属性赋值的方法。

以下是我的 FileDownloader.cs 的两个方法有问题：

public async static void DownloadAllFiles(Data clientData)
{
    data = clientData;

    data.OutputFolderExists();


    // Deserialize .json file and get ClipInfo list
    List<ClipInfo> clips = JsonConvert.DeserializeObject<List<ClipInfo>>(File.ReadAllText(data.JsonFile));
            
    tasks = new List<Task>();

    foreach(ClipInfo clip in clips)
    {
        tasks.Add(DownloadFilesAsync(clip));
    }

    await Task.WhenAll(tasks);
}

private async static Task DownloadFilesAsync(ClipInfo clip)
{
    WebClient client = new WebClient();
    string url = GetClipURL(clip);
    string filepath = data.OutputPath + clip.id + ".mp4";

    await client.DownloadFileTaskAsync(new Uri(url), filepath);
}

这只是我多次下载文件的尝试之一，我的灵感来自于此 post:

Whosebug_link

我还尝试过 IAmTimCorey 的 YouTube 视频中的以下方法：

video_link

我已经花了很多小时来解决这个问题，老实说，我无法弄清楚为什么我的任何尝试都不起作用。非常感谢您的帮助。

谢谢，

本

下面是我的全部代码，如果有人出于任何原因需要它。

代码结构：

我下载的唯一外部库是Newtonsoft.Json

ClipInfo.cs

using System;
using System.Collections.Generic;
using System.Text;

namespace Downloader
{
    public class ClipInfo
    {
        public string id { get; set; }
        public string url { get; set; }
        public string embed_url { get; set; }
        public string broadcaster_id { get; set; }
        public string broadcaster_name { get; set; }
        public string creator_id { get; set; }
        public string creator_name { get; set; }
        public string video_id { get; set; }
        public string game_id { get; set; }
        public string language { get; set; }
        public string title { get; set; }
        public int view_count { get; set; }
        public DateTime created_at { get; set; }
        public string thumbnail_url { get; set; }
    }
}

Pagination.cs

namespace Downloader
{
    public class Pagination
    {
        public string cursor { get; set; }
    }

}

Root.cs

using System.Collections.Generic;

namespace Downloader
{
    public class Root
    {
        public List<ClipInfo> data { get; set; }
        public Pagination pagination { get; set; }
    }
}

Data.cs

using System;
using System.IO;

namespace Downloader
{
    public class Data
    {
        private static string directory = Directory.GetCurrentDirectory();
        private readonly static string defaultJsonFile = directory + @"\clips.json";
        private readonly static string defaultOutputPath = directory + @"\Clips\";
        private readonly static string clipsLink = "https://api.twitch.tv/helix/clips?";

        public string OutputPath { get; set; }
        public string JsonFile { get; set; }
        public string ClientID { get; private set; }
        public string Authentication { get; private set; }
        public string QueryURL { get; private set; }
    

        public Data()
        {
            OutputPath = defaultOutputPath;
            JsonFile = defaultJsonFile;
        }
        public Data(string clientID, string authentication)
        {
            ClientID = clientID;
            Authentication = authentication;
            OutputPath = defaultOutputPath;
            JsonFile = defaultJsonFile;
        }
        public Data(string clientID, string authentication, string outputPath)
        {
            ClientID = clientID;
            Authentication = authentication;
            OutputPath = directory + @"\" + outputPath + @"\";
            JsonFile = OutputPath + outputPath + ".json";
        }

        public void GetQuery()
        {
            Console.Write("Please enter your query: ");
            QueryURL = clipsLink + Console.ReadLine();
        }

        public void GetClientID()
        {
            Console.WriteLine("Enter your client ID");
            ClientID = Console.ReadLine();
        }

        public void GetAuthentication()
        {
            Console.WriteLine("Enter your Authentication");
            Authentication = Console.ReadLine();
        }

        public void OutputFolderExists()
        {
            if (!Directory.Exists(OutputPath))
            {
                Directory.CreateDirectory(OutputPath);
            }
        }

    }
}

JsonGenerator.cs

using System;
using System.IO;
using System.Net.Http.Headers;
using System.Net.Http;
using System.Threading.Tasks;
using Newtonsoft.Json;
using System.Linq;


namespace Downloader
{
    public static class JsonGenerator
    {
        // This class has no constructor.
        // You call the Generate methods, passing in all required data.
        // The file will then be generated.
        private static Data data;

        public static async Task Generate(Data clientData)
        {
            data = clientData;
            string responseContent = null;

            // Loop that runs until the api request goes through
            bool authError = true;
            while (authError)
            {
                authError = false;
                try
                {
                    responseContent = await GetHttpResponse();
                }
                catch (HttpRequestException)
                {
                    Console.WriteLine("Invalid authentication, please enter client-ID and authentication again!");
                    data.GetClientID();
                    data.GetAuthentication();

                    authError = true;
                }
                catch (Exception e)
                {
                    Console.WriteLine(e.Message);
                    authError = true;
                }
            }

            data.OutputFolderExists();
            GenerateJson(responseContent);
        }

        // Returns the contents of the resopnse to the api call as a string
        private static async Task<string> GetHttpResponse()
        {
            // Creating client
            HttpClient client = new HttpClient();

            if (data.QueryURL == null)
            {
                data.GetQuery();
            }


            // Setting up request
            HttpRequestMessage requestMessage = new HttpRequestMessage(HttpMethod.Get, data.QueryURL);

            // Adding Headers to request
            requestMessage.Headers.Add("client-id", data.ClientID);
            requestMessage.Headers.Authorization = new AuthenticationHeaderValue("Bearer", data.Authentication);

            // Receiving response to the request
            HttpResponseMessage responseMessage = await client.SendAsync(requestMessage);

            // Gets the content of the response as a string
            string responseContent = await responseMessage.Content.ReadAsStringAsync();

            return responseContent;
        }

        // Generates or adds to the .json file that contains data on each clip
        private static void GenerateJson(string responseContent)
        {
            // Parses the data from the response to the api request
            Root responseResult = JsonConvert.DeserializeObject<Root>(responseContent);

            // If the file doesn't exist, we need to create it and add a '[' at the start
            if (!File.Exists(data.JsonFile))
            {
                FileStream file = File.Create(data.JsonFile);
                file.Close();
                // The array of json objects needs to be wrapped inside []
                File.AppendAllText(data.JsonFile, "[\n");
            }
            else
            {
                // For a pre-existing .json file, The last object won't have a comma at the
                // end of it so we need to add it now, before we add more objects
                string[] jsonLines = File.ReadAllLines(data.JsonFile);
                File.WriteAllLines(data.JsonFile, jsonLines.Take(jsonLines.Length - 1).ToArray());
                File.AppendAllText(data.JsonFile, ",");
            }

            // If the file already exists, but there was no [ at the start for whatever reason,
            // we need to add it
            if (File.ReadAllText(data.JsonFile).Length == 0 || File.ReadAllText(data.JsonFile)[0] != '[')
            {
                File.WriteAllText(data.JsonFile, "[\n" + File.ReadAllText(data.JsonFile));
            }

            string json;

            // Loops through each ClipInfo object that the api returned
            for (int i = 0; i < responseResult.data.Count; i++)
            {
                // Serializes the ClipInfo object into a json style string
                json = JsonConvert.SerializeObject(responseResult.data[i]);

                // Adds the serialized contents of ClipInfo to the .json file
                File.AppendAllText(data.JsonFile, json);

                if (i != responseResult.data.Count - 1)
                {
                    // All objects except the last require a comma at the end of the
                    // object in order to correctly format the array of json objects
                    File.AppendAllText(data.JsonFile, ",");
                }

                // Adds new line after object entry
                File.AppendAllText(data.JsonFile, "\n");
            }
            // Adds the ] at the end of the file to close off the json objects array
            File.AppendAllText(data.JsonFile, "]");
        }
    }
}

FileDownloader.cs

using Newtonsoft.Json;
using System;
using System.Collections.Generic;
using System.IO;
using System.Net;
using System.Threading.Tasks;

namespace Downloader
{
    public class FileDownloader
    {
        private static Data data;
        private static List<Task> tasks;
        public async static void DownloadAllFiles(Data clientData)
        {
            data = clientData;

            data.OutputFolderExists();


            // Deserialize .json file and get ClipInfo list
            List<ClipInfo> clips = JsonConvert.DeserializeObject<List<ClipInfo>>(File.ReadAllText(data.JsonFile));

            tasks = new List<Task>();

            foreach (ClipInfo clip in clips)
            {
                tasks.Add(DownloadFilesAsync(clip));
            }

            await Task.WhenAll(tasks);
        }

        private static void GetData()
        {
            if (data.ClientID == null)
            {
                data.GetClientID();
            }
            if (data.Authentication == null)
            {
                data.GetAuthentication();
            }
            if (data.QueryURL == null)
            {
                data.GetQuery();
            }
        }

        private static string GetClipURL(ClipInfo clip)
        {
            // Example thumbnail URL:
            // https://clips-media-assets2.twitch.tv/AT-cm%7C902106752-preview-480x272.jpg
            // You can get the URL of the location of clip.mp4
            // by removing the -preview.... from the thumbnail url */

            string url = clip.thumbnail_url;
            url = url.Substring(0, url.IndexOf("-preview")) + ".mp4";
            return url;
        }
            
        private async static Task DownloadFilesAsync(ClipInfo clip)
        {
            WebClient client = new WebClient();
            string url = GetClipURL(clip);
            string filepath = data.OutputPath + clip.id + ".mp4";

            await client.DownloadFileTaskAsync(new Uri(url), filepath);
        }

        private static void FileDownloadComplete(object sender, System.ComponentModel.AsyncCompletedEventArgs e)
        {
            tasks.Remove((Task)sender);
        }
    }
}

Program.cs

using System;
using System.Threading.Tasks;
using Downloader;

namespace ClipDownloader
{
    class Program
    {
        private static string clientID = "{your_client_id}";
        private static string authentication = "{your_authentication}";
        async static Task Main(string[] args)
        {
            Console.WriteLine("Enter your output path");
            string outputPath = Console.ReadLine();


            Data data = new Data(clientID, authentication, outputPath);
            Console.WriteLine(data.OutputPath);

            //await JsonGenerator.Generate(data);
            FileDownloader.DownloadAllFiles(data);
        }
    }
}

我通常输入的示例查询是“game_id=510218”

Answer 1

async void 是你的问题

改变

public static async void DownloadAllFiles(Data clientData)

到

public static async Task DownloadAllFiles(Data clientData)

那你就等着吧

await FileDownloader.DownloadAllFiles(data);

长话短说：

async void 运行未被观察到（即发即弃）。你不能等他们完成。本质上，一旦您的程序启动任务，它就会完成，并拆除 App Domain 和您所有的子任务，让您相信没有任何工作。

Answer 2

我尽量在这里停留在主题上，但是当使用 JsonConvert.DeserializeObject{T} 时，T 不应该是一个封装根对象类型吗？我从来没有像你那样使用过它，所以我很好奇这是否是你的错误。我可能完全错了，如果我错了，请原谅我，但是 JSON 是基于 key:value 的。直接反序列化为 List 并没有多大意义。除非反序列化器中有特殊情况？ List 将是一个纯 ClipInfo 值数组的文件，被反序列化为 List{T} 的成员（private T[] _items、private int _size 等）它需要一个父根对象。

// current JSON file format implication(which i dont think is valid JSON?(correct me please) 
clips:
[
  // clip 1
  {  "id": "", "url": "" },

  // clip N
  {  "id": "", "url": "" },
]

// correct(?) JSON file format
{ // { } is the outer encasing object
    clips:
    [
        // clip 1
        {  "id": "", "url": "" },

        // clip N
        {  "id": "", "url": "" },
    ]
}

class ClipInfoJSONFile
{
    public List<ClipInfo> Info { get; set; }
}

var clipInfoList = JsonConverter.DeserializeObject<ClipInfoJSONFile>(...);

C# 在控制台应用程序上并行异步下载多个文件时遇到问题

C# Having Trouble Asynchronously Downloading Multiple Files in Parallel on Console Application

c#

asynchronous

webclient

twitch-api