Stream.CopyToAsync 有进度报告 - 即使在复制完成后也会报告进度

Question

我构建了一个简单的控制台应用程序，可以从 Internet 下载文件。
因为 I had problems with WebClient 我决定使用 HttpClient 编写我的应用程序。

基本上我正在请求读取 headers，然后使用 ReadAsStreamAsync 我正在获取流，我正在使用 CopyToAsync.[=22= 将其复制到本地文件]

我找到了支持 IProgress 的流扩展方法：

public static class StreamExtensions
{
    public static async Task CopyToAsync(this Stream source, Stream destination, IProgress<long> progress, CancellationToken cancellationToken = default(CancellationToken), int bufferSize = 0x1000)
    {
        var buffer = new byte[bufferSize];
        int bytesRead;
        long totalRead = 0;
        while ((bytesRead = await source.ReadAsync(buffer, 0, buffer.Length, cancellationToken)) > 0)
        {
            await destination.WriteAsync(buffer, 0, bytesRead, cancellationToken);
            cancellationToken.ThrowIfCancellationRequested();
            totalRead += bytesRead;
            //Thread.Sleep(10);
            progress.Report(totalRead);
        }
    }
}

我的应用程序可以运行，但我得到的进度信息不正确。
例如，当下载 2 个文件时，我在输出 window:

中看到了这个

file1.tmp 60.95%
file2.tmp 98.09%
file1.tmp 60.98%
file2.tmp 98.21%
file2.tmp 98.17%
file2.tmp 98.25%
file1.tmp 61.02%
file2.tmp 98.41%
file2.tmp downloaded
file2.tmp 98.29%
file2.tmp 98.37%
file1.tmp 61.06%
file2.tmp 89.27%
file2.tmp 89.31%
file2.tmp 98.33%
file2.tmp 98.45%
file2.tmp 98.48%
file1.tmp 61.10%
file1.tmp 61.14%
file2.tmp 98.52%
file1.tmp 61.22%
file2.tmp 98.60%
file2.tmp 98.56%
file1.tmp 61.30%
file2.tmp 98.88%
file2.tmp 90.44%
file1.tmp 61.53%
file2.tmp 98.72%
file1.tmp 61.41%
file1.tmp 61.73%
file2.tmp 98.80%
file1.tmp 61.26%
file1.tmp 61.49%
file1.tmp 61.57%
file1.tmp 61.69%
...
file1.tmp 99.31%
file1.tmp 98.84%
file1.tmp 98.80%
file1.tmp 99.04%
file1.tmp 99.43%
file1.tmp 99.12%
file1.tmp 99.00%
file1.tmp downloaded
file1.tmp 100.00%
file1.tmp 98.73%
file1.tmp 98.88%
file1.tmp 99.47%
file1.tmp 99.98%
file1.tmp 99.90%
file1.tmp 98.96%
file1.tmp 99.78%
file1.tmp 99.99%
file1.tmp 99.74%
file1.tmp 99.59%
file1.tmp 99.94%
file1.tmp 98.49%
file1.tmp 98.53%
ALL FILES DOWNLOADED
file1.tmp 99.55%
file1.tmp 98.41%
file1.tmp 99.62%
file1.tmp 98.34%
file1.tmp 99.66%
file1.tmp 98.69%
file1.tmp 98.37%

如您所见，我获得了文件 2 已下载的信息，但我仍在从 CopyToAsync 获得进度报告，与文件 1 相同。

因此我有时会得到这个奇怪的控制台输出：

理想情况下，我想比打电话时更确定：

await streamToReadFrom.CopyToAsync(streamToWriteTo, progress, source.Token,0x2000);
Debug.WriteLine(filename+" downloaded");

在我获得调试信息后，没有任何进展报告（文件已下载）。我以为 await 会解决我的问题，但事实并非如此。

我该如何解决这个问题？作为临时解决方案，我在报告进度之前将 Thread.Sleep 添加到 CopyToAsync。

下面是我当前的代码：

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Linq;
using System.Net.Http;
using System.Threading;
using System.Threading.Tasks;

namespace AsyncDownloadTest
{
    class Program
    {
        private const string LocalPath = @"D:\TEMP";

        static void Main()
        {
            try
            {
                var filesToDownlad = new List<Tuple<string, string>>
                {
                    new Tuple<string, string>("file1.tmp", "http://ipv4.download.thinkbroadband.com/10MB.zip"),
                    new Tuple<string, string>("file2.tmp", "http://ipv4.download.thinkbroadband.com/10MB.zip")
                };
                _consolePosition = -1;
                Console.CursorVisible = false;

                Parallel.ForEach(filesToDownlad, new ParallelOptions { MaxDegreeOfParallelism = 4 }, doc =>
                {
                    DownloadFile(doc.Item2,doc.Item1).Wait();
                });
                Debug.WriteLine("ALL FILES DOWNLOADED");
                Console.CursorVisible = true;    
            }
            catch (Exception e)
            {
                Console.WriteLine(e);
                Console.ReadLine();
            }
        }

        private static readonly object ConsoleLock = new object();
        private static int _consolePosition;

        static readonly CancellationTokenSource source = new CancellationTokenSource();

        private static async Task DownloadFile(string url, string filename)
        {
            int currenctLineNumber = 0;
            int currectProgress = 0;

            try
            {
                lock (ConsoleLock)
                {
                    _consolePosition++;
                    currenctLineNumber = _consolePosition;
                }

                long fileSize = -1;

                IProgress<long> progress = new Progress<long>(value =>
                {
                    decimal tmp = (decimal)(value * 100) / fileSize;

                    if (tmp != currectProgress && tmp > currectProgress)
                    {
                        lock (ConsoleLock)
                        {
                            currectProgress = (int)tmp;
                            Console.CursorTop = currenctLineNumber;
                            Console.CursorLeft = 0;
                            Console.Write("{0,10} - {2,11} - {1,6:N2}%", filename, tmp, "DOWNLOADING");
                        }
                        Debug.WriteLine("{1} {0:N2}%", tmp, filename);
                    }
                });

                using (HttpClient client = new HttpClient())
                {
                    using (HttpResponseMessage response = await client.GetAsync(url, HttpCompletionOption.ResponseHeadersRead, source.Token))
                    {
                        response.EnsureSuccessStatusCode();
                        if (response.Content.Headers.ContentLength.HasValue) fileSize = response.Content.Headers.ContentLength.Value;

                        if (response.Content.Headers.ContentDisposition != null)
                        {
                            var tmp = response.Content.Headers.ContentDisposition.FileName.Replace("\"", "");
                            Debug.WriteLine("Real name: {0}",tmp);
                        }

                        using (Stream streamToReadFrom = await response.Content.ReadAsStreamAsync())
                        {
                            using (Stream streamToWriteTo = File.Open(Path.Combine(LocalPath, filename), FileMode.Create, FileAccess.Write))
                            {
                                await streamToReadFrom.CopyToAsync(streamToWriteTo, progress, source.Token,0x2000);

                                Debug.WriteLine(filename+" downloaded");

                                lock (ConsoleLock)
                                {
                                    Console.CursorTop = currenctLineNumber;
                                    Console.CursorLeft = 0;
                                    var oldColor = Console.ForegroundColor;
                                    Console.ForegroundColor = ConsoleColor.Green;
                                    Console.Write("{0,10} - {2,11} - {1,6:N2}%", filename, 100, "SUCCESS");
                                    Console.ForegroundColor = oldColor;
                                }
                            }
                        }
                    }
                }
            }
            catch (Exception e)
            {
                Debug.WriteLine(e.Message);
                lock (ConsoleLock)
                {
                    Console.CursorTop = currenctLineNumber;
                    Console.CursorLeft = 0;
                    var oldColor = Console.ForegroundColor;
                    Console.ForegroundColor = ConsoleColor.Red;
                    Console.Write("{0,10} - {2,11} - {1,6:N2}%", filename, currectProgress, "ERROR");
                    Console.ForegroundColor = oldColor;
                }
            }
        }
    }

    public static class StreamExtensions
    {
        public static async Task CopyToAsync(this Stream source, Stream destination, IProgress<long> progress, CancellationToken cancellationToken = default(CancellationToken), int bufferSize = 0x1000)
        {
            var buffer = new byte[bufferSize];
            int bytesRead;
            long totalRead = 0;
            while ((bytesRead = await source.ReadAsync(buffer, 0, buffer.Length, cancellationToken)) > 0)
            {
                await destination.WriteAsync(buffer, 0, bytesRead, cancellationToken);
                cancellationToken.ThrowIfCancellationRequested();
                totalRead += bytesRead;
                Thread.Sleep(10);
                progress.Report(totalRead);
            }
        }
    }
}

Answer 1

你的问题其实在这里：

new Progress<long>

Progress<T> class always invokes its callbacks in a SynchronizationContext - 在本例中是线程池 SynchronizationContext。这意味着当进度报告代码调用Report时，它只是将回调排队到线程池。因此，可能会看到它们乱序（或者在下载实际完成后仍然出现）。

要解决此问题，您可以创建自己的自定义实现 IProgress<T>:

//C#6.0
public sealed class SynchronousProgress<T> : IProgress<T>
{
  private readonly Action<T> _callback;
  public SynchronousProgress(Action<T> callback) { _callback = callback; }
  void IProgress<T>.Report(T data) => _callback(data);
}
//older version
public sealed class SynchronousProgress<T> : IProgress<T>
{
    private readonly Action<T> _callback;

    public SynchronousProgress(Action<T> callback)
    {
        _callback = callback;
    }

    void IProgress<T>.Report(T data)
    {
        _callback(data);
    }
}

然后替换行

IProgress<long> progress = new Progress<long>(value =>

和

IProgress<long> progress = new SynchronousProgress<long>(value =>

Answer 2

OP 要求我在评论中展示如何使用 TPL 数据流执行他的程序。这实际上是一个相当简单的转换。首先添加对 NuGet 包的引用 System.Threading.Tasks.Dataflow。然后只需将您的主要功能更改为

static void Main()
{
    try
    {
        var filesToDownlad = new List<Tuple<string, string>>
        {
            new Tuple<string, string>("file1.tmp", "http://ipv4.download.thinkbroadband.com/10MB.zip"),
            new Tuple<string, string>("file2.tmp", "http://ipv4.download.thinkbroadband.com/10MB.zip")
        };
        _consolePosition = -1;
        Console.CursorVisible = false;

        var downloadBlock = new ActionBlock<Tuple<string, string>>(doc => DownloadFile(doc.Item2, doc.Item1),
                                                                   new ExecutionDataflowBlockOptions {MaxDegreeOfParallelism = 4});

        foreach (var file in filesToDownlad)
        {
            downloadBlock.Post(file);
        }
        downloadBlock.Complete();
        downloadBlock.Completion.Wait();


        Debug.WriteLine("ALL FILES DOWNLOADED");
        Console.CursorVisible = true;
    }
    catch (Exception e)
    {
        Console.WriteLine(e);
        Console.ReadLine();
    }
}

如果您使用具有同步上下文的程序执行此操作并且您希望等待完成和发布而不是执行同步操作，您可以这样做

static async Task Example()
{
    try
    {
        var filesToDownlad = new List<Tuple<string, string>>
        {
            new Tuple<string, string>("file1.tmp", "http://ipv4.download.thinkbroadband.com/10MB.zip"),
            new Tuple<string, string>("file2.tmp", "http://ipv4.download.thinkbroadband.com/10MB.zip")
        };
        _consolePosition = -1;
        Console.CursorVisible = false;

        var downloadBlock = new ActionBlock<Tuple<string, string>>(doc => DownloadFile(doc.Item2, doc.Item1),
                                                                   new ExecutionDataflowBlockOptions {MaxDegreeOfParallelism = 4});

        foreach (var file in filesToDownlad)
        {
            await downloadBlock.SendAsync(file);
        }
        downloadBlock.Complete();
        await downloadBlock.Completion;


        Debug.WriteLine("ALL FILES DOWNLOADED");
        Console.CursorVisible = true;
    }
    catch (Exception e)
    {
        Console.WriteLine(e);
        Console.ReadLine();
    }
}

请注意，这并不能解决您 "ALL FILES DOWNLOADED" 运行的问题。您需要使用。如果在调用线程上可能存在 SynchronizationContext 的情况下此代码是运行，那么所有这一切都是为了修复潜在的死锁。

Stream.CopyToAsync 有进度报告 - 即使在复制完成后也会报告进度

Stream.CopyToAsync with progress reporting - progress is reported even after copying finish

c#

asynchronous

dotnet-httpclient

.net-4.6