使用 FluentFTP 将文件从 FTP 复制到 blob 存储的 Azure 函数

Question

我有一个添加每日文件的 FTP 源，我需要每天使用 azure 函数中的 FluentFTP 库将文件从 FTP 复制到 blob 存储

我在 Azure 函数中使用 C#，我完成了所有编码部分，但没有从 FTP 下载文件以将其直接复制到 blob 目标。

//#r "FluentFTP"
#r "Newtonsoft.Json"
#r "System.Data"
#r "Microsoft.WindowsAzure.Storage"

using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using System.Collections.Generic;
using System.Globalization;
using System;
using System.Configuration;
using System.Data;
using System.Data.SqlClient;
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Auth;
using Microsoft.WindowsAzure.Storage.Blob;

using System.Net;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Primitives;
using Newtonsoft.Json;
using FluentFTP;

public static async Task<IActionResult> Run(HttpRequest req, ILogger log)
{
    string blobConnectionString = "ConnectionString";

    // Gestione BLOB Storage
    CloudStorageAccount storageAccount = CloudStorageAccount.Parse(blobConnectionString);
    CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
    CloudBlobContainer container = blobClient.GetContainerReference("out");

    using (FtpClient conn = new FtpClient()) 
    {
        conn.Host = "ftp server";
        conn.Credentials = new NetworkCredential("username", "pass");

        // get a list of files and directories in the "/OUT" folder
        foreach (FtpListItem item in conn.GetListing("/OUT")) 
        {
            // if this is a file and ends with CSV
            if (
            item.Type == FtpFileSystemObjectType.File
            &&
            item.FullName.ToLower().EndsWith(".csv")
            )
            {

                string yyyy = item.FullName.Substring(10,4);
                string mm   = item.FullName.Substring(14,2);
                string dd   = item.FullName.Substring(16,2);
                var fileName = "out/brt/" + yyyy + "/"+ mm + "/"+ dd + "/" + item.Name;

                CloudBlockBlob blockBlob = container.GetBlockBlobReference(fileName);
                // download the file
                conn.DownloadFile( blockBlob , item.FullName);
            }
        }

        return new OkObjectResult($"Hello");
    }
}

如果我可以使用 blob 容器作为 FluentFTP 函数的目标，那将是最好的，但是这样我得到的错误是我正在使用的这个 blob 块不是一个目的地

这是我遇到的错误

cannot convert from 'Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob' to 'string'

我不知道是否有另一种方法可以在本地下载文件，这样我就可以将它上传到 blob 而不是使用这个 client.DownloadFile 函数。

Answer 1

您不能使用 SDK 直接将文件从 FTP 移动到 Blob。您必须先临时下载文件 - 要么下载到流中（取决于文件的大小），要么下载到临时文件中。

但是，如果您只想按计划将文件从 FTP 移动到 Blob，我实际上会考虑使用专为此类任务构建的 Azure 数据工厂：https://docs.microsoft.com/en-us/azure/data-factory/connector-ftp

Answer 2

Fluent FTP.

可能做不到

有 FtpClient.Download 方法需要 Stream。

public bool Download(Stream outStream, string remotePath, IProgress<double> progress = null)

但是好像没有API得到"blob upload Stream".

相反，您无法从 FluentFTP 获得 "FTP download Stream"（您可以将其与 blob API 一起使用）。

但您可以使用本机 .NET FtpWebRequest FTP 客户端，它具有 API 以获得 "FTP download Stream":

public static async System.Threading.Tasks.Task RunAsync([TimerTrigger("0 */1 * * * *")]TimerInfo myTimer, ILogger log)
{
    var ftpUserName = "xxxxxx";
    var ftpPassword = "xxxxxxxxx";
    var filename = "test.png";
    string blobConnectionString = "xxxxxxx";
    CloudStorageAccount storageAccount = CloudStorageAccount.Parse(blobConnectionString);
    CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
    CloudBlobContainer container = blobClient.GetContainerReference("myblobcontainer");
    FtpWebRequest fileRequest = (FtpWebRequest)WebRequest.Create("ftp://xxxxxx/" + filename);
    fileRequest.Method = WebRequestMethods.Ftp.DownloadFile;
    fileRequest.Credentials = new NetworkCredential(ftpUserName, ftpPassword);
    FtpWebResponse fileResponse = (FtpWebResponse)fileRequest.GetResponse();
    Stream fileStream = fileResponse.GetResponseStream();
    CloudBlockBlob blockBlob = container.GetBlockBlobReference(filename);

    await blockBlob.UploadFromStreamAsync(fileStream);
    log.LogInformation($"C# Timer trigger function executed at: {DateTime.Now}");
}

Answer 3

如果您想使用 FluentFTP，您可以使用以下两种方法之一获取 blob 上传流：

然后你可以使用 FTPClient.Download 方法，它需要一个 Stream

public bool Download(Stream outStream, string remotePath, IProgress<double> progress = null)

像这样：

[FunctionName("HttpTriggerCSharp")]
    public static async Task<IActionResult> Run([HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = null)] HttpRequest req, ILogger log)
    {
        log.LogInformation("C# HTTP trigger function processed a request.");

        var ftpHost = "xxxxx";
        var ftpUserName = "xxxx";
        var ftpPassword = "xxxx";
        var filename = "xxxxx";            
        string blobConnectionString = "xxxxxxxxxxx";

        CloudStorageAccount storageAccount = CloudStorageAccount.Parse(blobConnectionString);
        CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
        CloudBlobContainer container = blobClient.GetContainerReference("xxxxx");


        FtpClient client = new FtpClient(ftpHost, ftpUserName, ftpPassword); // or set Host & Credentials
        client.EncryptionMode = FtpEncryptionMode.Implicit;
        client.SslProtocols = SslProtocols.None;
        client.ValidateCertificate += new FtpSslValidation(OnValidateCertificate);
        client.Connect();

        void OnValidateCertificate(FtpClient control, FtpSslValidationEventArgs e) {
            // add logic to test if certificate is valid here
            e.Accept = true;
        }

        CloudBlockBlob blockBlob = container.GetBlockBlobReference(filename);
        var outStream = blockBlob.OpenWrite();
        client.Download(outStream,filename);
        outStream.Commit(); // I'm not sure if this is needed?

        log.LogInformation($"C# Timer trigger function executed at: {DateTime.Now}");


        return  (ActionResult)new OkObjectResult($"Action succeeded.");
    }

使用 FluentFTP 将文件从 FTP 复制到 blob 存储的 Azure 函数

Azure function to copy files from FTP to blob storage using FluentFTP

c#

ftp

azure-blob-storage

azure-functions

fluentftp