测试分段文件上传 Azure 函数

Testing a Multipart file upload Azure Function

所以我编写了一个简单的 Azure 函数 (AF),它接受(通过 Http Post 方法)IFormCollection,遍历文件 collection,将每个文件推送到 Azure Blob 存储容器中returns url 每个文件。

当我使用 'multipart/form-data' header 通过 Postman 处理单个文件或多个文件 post 时,该功能本身完美运行。但是,当我尝试通过 xUnit 测试 post 文件时,出现以下错误:

System.IO.InvalidDataException:多部分 body 超过长度限制 16384。

我四处寻找解决方案,尝试了不同的方法,即;

到目前为止,

None 个已提供帮助。

具体项目如下:

Azure Function v3: 目标 .netcoreapp3.1

Startup.cs

public class Startup : FunctionsStartup
{
    public IConfiguration Configuration { get; private set; }

    public override void Configure(IFunctionsHostBuilder builder)
    {
        var x = builder;

        InitializeConfiguration(builder);

        builder.Services.AddSingleton(Configuration.Get<UploadImagesAppSettings>());

        builder.Services.AddLogging();

        builder.Services.AddSingleton<IBlobService,BlobService>();
    }

    private void InitializeConfiguration(IFunctionsHostBuilder builder)
    {
        var executionContextOptions = builder
            .Services
            .BuildServiceProvider()
            .GetService<IOptions<ExecutionContextOptions>>()
            .Value;

        Configuration = new ConfigurationBuilder()
            .SetBasePath(executionContextOptions.AppDirectory)
            .AddJsonFile("appsettings.json")
            .AddJsonFile("appsettings.Development.json", optional: true)
            .AddEnvironmentVariables()
            .Build();
    }
}

UploadImages.cs

    public class UploadImages
    {
        private readonly IBlobService BlobService;


        public UploadImages(IBlobService blobService)
        {
            BlobService = blobService;
        }

        [FunctionName("UploadImages")]
        [RequestFormLimits(ValueLengthLimit = int.MaxValue, 
            MultipartBodyLengthLimit = 60000000, ValueCountLimit = 10)]
        public async Task<IActionResult> Run(
            [HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = "images")] HttpRequest req)
        {
            List<Uri> returnUris = new List<Uri>();

            if (req.ContentLength == 0)
            {
                string badResponseMessage = $"Request has no content";
                return new BadRequestObjectResult(badResponseMessage);
            }

            if (req.ContentType.Contains("multipart/form-data") && req.Form.Files.Count > 0) 
            {
                foreach (var file in req.Form.Files)
                {
                    if (!file.IsValidImage()) 
                    {
                        string badResponseMessage = $"{file.FileName} is not a valid/accepted Image file";
                        return new BadRequestObjectResult(badResponseMessage);
                    }

                    var uri = await BlobService.CreateBlobAsync(file);
                    if (uri == null)
                    {
                        return new ObjectResult($"Could not blob the file {file.FileName}.");
                    }

                    returnUris.Add(uri);
                }
            }

            if (!returnUris.Any()) 
            {
                return new NoContentResult();
            }

            return new OkObjectResult(returnUris);
        }
    }

异常抛出:

当上面的第二个 if 语句试图处理 req.Form.Files.Count > 0 时抛出以下异常,即

if (req.ContentType.Contains("multipart/form-data") && req.Form.Files.Count > 0) {}

Message: 
    System.IO.InvalidDataException : Multipart body length limit 16384 exceeded.
  Stack Trace: 
    MultipartReaderStream.UpdatePosition(Int32 read)
    MultipartReaderStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
    StreamHelperExtensions.DrainAsync(Stream stream, ArrayPool`1 bytePool, Nullable`1 limit, CancellationToken cancellationToken)
    MultipartReader.ReadNextSectionAsync(CancellationToken cancellationToken)
    FormFeature.InnerReadFormAsync(CancellationToken cancellationToken)
    FormFeature.ReadForm()
    DefaultHttpRequest.get_Form()
    UploadImages.Run(HttpRequest req) line 42
    UploadImagesTests.HttpTrigger_ShouldReturnListOfUploadedUris(String fileNames)

xUnit 测试项目: 目标 .netcoreapp3.1

转到 xUnit 测试项目,基本上我正在尝试编写集成测试。该项目引用了 AF 项目并具有以下 classes:

TestHost.cs

public class TestHost
{
    public TestHost()
    {
        var startup = new TestStartup();
        var host = new HostBuilder()
            .ConfigureWebJobs(startup.Configure)
            .ConfigureServices(ReplaceTestOverrides)
            .Build();

        ServiceProvider = host.Services;
    }

    public IServiceProvider ServiceProvider { get; }

    private void ReplaceTestOverrides(IServiceCollection services)
    {
        // services.Replace(new ServiceDescriptor(typeof(ServiceToReplace), testImplementation));
    }

    private class TestStartup : Startup
    {
        public override void Configure(IFunctionsHostBuilder builder)
        {
            SetExecutionContextOptions(builder);
            base.Configure(builder);
        }

        private static void SetExecutionContextOptions(IFunctionsHostBuilder builder)
        {
            builder.Services.Configure<ExecutionContextOptions>(o => o.AppDirectory = Directory.GetCurrentDirectory());
        }
    }
}

TestCollection.cs

[CollectionDefinition(Name)]
public class TestCollection : ICollectionFixture<TestHost>
{ 
    public const string Name = nameof(TestCollection);

}

HttpRequestFactory.cs: 创建 Http Post 请求

public static class HttpRequestFactory
{
    public static DefaultHttpRequest Create(string method, string contentType, Stream body) 
    {
        var request = new DefaultHttpRequest(new DefaultHttpContext());
        var contentTypeWithBoundary = new MediaTypeHeaderValue(contentType) 
        {
            Boundary = $"----------------------------{DateTime.Now.Ticks.ToString("x")}"
        };

        var boundary = MultipartRequestHelper.GetBoundary(
            contentTypeWithBoundary, (int)body.Length);

        request.Method = method;
        request.Headers.Add("Cache-Control", "no-cache");
        request.Headers.Add("Content-Type", contentType);
        request.ContentType = $"{contentType}; boundary={boundary}";
        request.ContentLength = body.Length;
        request.Body = body;

        return request;
    }

    private static string GetBoundary(MediaTypeHeaderValue contentType, int lengthLimit)
    {
        var boundary = HeaderUtilities.RemoveQuotes(contentType.Boundary);
        if (string.IsNullOrWhiteSpace(boundary.Value))
        {
            throw new InvalidDataException("Missing content-type boundary.");
        }

        if (boundary.Length > lengthLimit)
        {
            throw new InvalidDataException(
                $"Multipart boundary length limit {lengthLimit} exceeded.");
        }

        return boundary.Value;
    }

}

The MultipartRequestHelper.cs class is available here

最后测试 class:

[Collection(TestCollection.Name)]
public class UploadImagesTests
{
    readonly UploadImages UploadImagesFunction;

    public UploadImagesTests(TestHost testHost)
    {
        UploadImagesFunction = new UploadImages(testHost.ServiceProvider.GetRequiredService<IBlobService>());
    }

    [Theory]
    [InlineData("testfile2.jpg")]
    public async void HttpTrigger_ShouldReturnListOfUploadedUris(string fileNames)
    {
        var formFile = GetFormFile(fileNames);
        var fileStream = formFile.OpenReadStream();
        var request = HttpRequestFactory.Create("POST", "multipart/form-data", fileStream);
        var response = (OkObjectResult)await UploadImagesFunction.Run(request);
        //fileStream.Close();
        Assert.True(response.StatusCode == StatusCodes.Status200OK);
    }

    private static IFormFile GetFormFile(string fileName)
    {
        string fileExtension = fileName.Substring(fileName.IndexOf('.') + 1);
        string fileNameandPath = GetFilePathWithName(fileName);
        IFormFile formFile;
        var stream = File.OpenRead(fileNameandPath);

        switch (fileExtension)
        {
            case "jpg":
                formFile = new FormFile(stream, 0, stream.Length,
                    fileName.Substring(0, fileName.IndexOf('.')),
                    fileName)
                {
                    Headers = new HeaderDictionary(),
                    ContentType = "image/jpeg"
                };
                break;

            case "png":
                formFile = new FormFile(stream, 0, stream.Length,
                    fileName.Substring(0, fileName.IndexOf('.')),
                    fileName)
                {
                    Headers = new HeaderDictionary(),
                    ContentType = "image/png"
                };
                break;

            case "pdf":
                formFile = new FormFile(stream, 0, stream.Length,
                    fileName.Substring(0, fileName.IndexOf('.')),
                    fileName)
                {
                    Headers = new HeaderDictionary(),
                    ContentType = "application/pdf"
                };
                break;

            default:
                formFile = null;
                break;
        }

        return formFile;
    }

    private static string GetFilePathWithName(string filename)
    {
        var outputFolder = Path.GetDirectoryName(System.Reflection.Assembly.GetExecutingAssembly().Location);
        return $"{outputFolder.Substring(0, outputFolder.IndexOf("bin"))}testfiles\{filename}";
    }
}

测试似乎命中了函数,req.ContentLength确实有一个值。考虑到这一点,它是否与文件流的管理方式有关?也许方法不对?

如有任何意见,我们将不胜感激。

谢谢

更新 1

根据 post,我还尝试在 Azure 函数启动时设置 ValueLengthLimit 和 MultipartBodyLengthLimit and/or 测试项目而不是 Azure 函数上的属性.然后异常更改为:

“内部流位置意外改变”

在此之后,我再将测试项目中的fileStream位置设置为SeekOrigin.Begin。我开始遇到同样的错误:

“多部分 body 超出长度限制 16384。”

我骑了 50 公里的自行车,睡了个好觉,但我终于想通了:-)。

Azure 函数 (AF) 接受一个名为 'req' 的 HttpRequest 对象作为参数,即

public 异步任务 运行( [HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = "images")] HttpRequest req)

HttpRequest 对象中文件对象的层次结构(连同参数名称)如下:

  • HttpRequest -> 请求
    • FormCollection -> 表单
      • FormFileCollection -> 文件

这是 AF 接受的内容,可以使用 req.Form.Files

访问文件集合

在我的测试用例中,我没有 posting 一个 FormCollection 对象,而是试图 post 一个文件流到 Azure 函数。

var formFile = GetFormFile(fileNames);
var fileStream = formFile.OpenReadStream();
var request = HttpRequestFactory.Create("POST", "multipart/form-data", fileStream);

因此,req.Form 有一个无法解释的 Stream 值,req.Form.Files 引发异常。

为了纠正这个问题,我必须执行以下操作:

  • 还原作为更新 1 的一部分所做的所有更改。这意味着我从启动文件中删除了 'RequestFormLimits' 设置,并将它们作为函数 运行 方法的属性保留。
  • 实例化 FormFileCollection 对象并向其添加 IFormFile
  • 使用此 FormFileCollection 作为参数实例化 FormCollection 对象。
  • 将 FormCollection 添加到请求对象。

为了实现上述目标,我不得不对代码进行以下更改。

在 HttpRequestFactory

中更改 'Create' 方法
    public static DefaultHttpRequest Create(string method, string contentType, FormCollection formCollection) 
    {
        var request = new DefaultHttpRequest(new DefaultHttpContext());
        var boundary = $"----------------------------{DateTime.Now.Ticks.ToString("x")}";

        request.Method = method;
        request.Headers.Add("Cache-Control", "no-cache");
        request.Headers.Add("Content-Type", contentType);
        request.ContentType = $"{contentType}; boundary={boundary}";
        request.Form = formCollection;
        

        return request;
    }

添加私有静态 GetFormFiles() 方法

我编写了一个额外的 GetFormFiles() 方法,它调用现有的 GetFormFile() 方法,实例化一个 FormFileCollection 对象并将 IFormFile 添加到它。这个方法又returns一个FormFileCollection.

    private static FormFileCollection GetFormFiles(string fileNames) 
    {
        var formFileCollection = new FormFileCollection();
        foreach (var file in fileNames.Split(','))
        {
            formFileCollection.Add(GetFormFile(file));
        }
        return formFileCollection;
    }

更改测试方法

测试方法调用GetFormFiles()获取一个FormFileCollection然后 使用此 FormFileCollection 作为参数实例化 FormCollection 对象,然后将 FormCollection 对象作为参数传递给 HttpRequest 对象,而不是传递 Stream。

    [Theory]
    [InlineData("testfile2.jpg")]
    public async void HttpTrigger_ShouldReturnListOfUploadedUris(string fileNames)
    {
        var formFiles = GetFormFiles(fileNames);

        var formCollection = new FormCollection(null, formFiles);
        var request = HttpRequestFactory.Create("POST", "multipart/form-data", formCollection);
        var response = (OkObjectResult) await UploadImagesFunction.Run(request);

        Assert.True(response.StatusCode == StatusCodes.Status200OK);
    }

所以最终问题不在于 'RequestFormLimits',而在于我在 POST 消息中提交的数据类型。

我希望这个答案能为遇到相同错误消息的人提供不同的视角。

干杯。