Azure Batch Pool 启动任务以从 Blob FileShare 下载资源文件

Question

我是 Azure 的新手，我想在这里构建一个简单的 Azure 批处理案例。

我正在批处理池启动任务...

我在美国东部创建了一个批处理账户和一个存储账户，然后我在存储账户中创建了一个通用的--Fileshare 和一个容器。我手动更新了一个名为 Test.txt 的文件。

我想做的是让Batch Pool在启动任务时下载这个文件。

所以代码是：

string storageConnectionString = String.Format("DefaultEndpointsProtocol=https;AccountName={0};AccountKey={1}", StorageAccountName, StorageAccountKey);

// Retrieve the storage account
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(storageConnectionString);
CloudFileClient fileClient = storageAccount.CreateCloudFileClient();
SharedAccessFilePolicy fileShareConstraint = new SharedAccessFilePolicy
{
    SharedAccessExpiryTime = DateTime.UtcNow.AddHours(48),
    Permissions = SharedAccessFilePermissions.Read,
};
var fileShare = fileClient.GetShareReference(inputContainerName);
var rootDir = fileShare.GetRootDirectoryReference();
var testFile = rootDir.GetFileReference("test.txt");
var sasUrl = fileShare.GetSharedAccessSignature(fileShareConstraint);
var fileUrl = string.Format("{0}{1}", testFile.StorageUri.PrimaryUri, sasUrl);

var list = new List<ResourceFile>();
var testResourceFile = new ResourceFile(fileUrl, "test.txt");

list.Add(testResourceFile );
await CreatePoolAsync(batchClient, PoolId, list);

然后是 CreatePoolAsync 方法：

private static async Task CreatePoolAsync(BatchClient batchClient, string poolId, IList<ResourceFile> resourceFiles)
{

    Console.WriteLine("Creating pool [{0}]...", poolId);

    // Create the unbound pool. Until we call CloudPool.Commit() or CommitAsync(), no pool is actually created in the
    // Batch service. This CloudPool instance is therefore considered "unbound," and we can modify its properties.

    //if(await batchClient.PoolOperations.GetPoolAsync(poolId) == null)
    //{

    CloudPool pool = batchClient.PoolOperations.CreatePool(
        poolId: poolId,
        targetDedicated: 1,                                                         // 3 compute nodes
        virtualMachineSize: "small",                                                // single-core, 1.75 GB memory, 225 GB disk
        cloudServiceConfiguration: new CloudServiceConfiguration(osFamily: "4"));   // Windows Server 2012 R2
    pool.MaxTasksPerComputeNode = 2;

    // Create and assign the StartTask that will be executed when compute nodes join the pool.
    // In this case, we copy the StartTask's resource files (that will be automatically downloaded
    // to the node by the StartTask) into the shared directory that all tasks will have access to.
    pool.StartTask = new StartTask
    {
        // Specify a command line for the StartTask that copies the task application files to the
        // node's shared directory. Every compute node in a Batch pool is configured with a number
        // of pre-defined environment variables that can be referenced by commands or applications
        // run by tasks.

        // Since a successful execution of robocopy can return a non-zero exit code (e.g. 1 when one or
        // more files were successfully copied) we need to manually exit with a 0 for Batch to recognize
        // StartTask execution success.

        CommandLine = "cmd /c (robocopy %AZ_BATCH_TASK_WORKING_DIR% %AZ_BATCH_NODE_SHARED_DIR%) ^& IF %ERRORLEVEL% LEQ 1 exit 0",
        //CommandLine = "cmd /c net use E: \krisblob2.file.core.windows.net\krisfilecontiner1 /u:krisblob2 aqTFKyPqcpeI3BrEnlx8RTBAmDaN5FK+mxpBtdgn3v6IT+IbPgDhVU4ojRA1wAmMpYPEHQ9Gzh/A1mAHtxNs+A==",
        //CommandLine = $@"cmd /c net use Z: \{StorageAccountName}.file.core.windows.net\krisfilecontiner1 /u:{StorageAccountName} {StorageAccountKey}",
        //CommandLine = "cmd /c %AZ_BATCH_TASK_WORKING_DIR%\WinPcap_4_1_3.exe /passive",
        ResourceFiles = resourceFiles,
        WaitForSuccess = true
    };
}

inputcontainer 是我在文件共享中为容器命名的。

当我运行代码时，启动任务总是失败并出现错误：

BlobDownloadMisc 错误消息

Miscellaneous error encountered while downloading one of the specified Azure Blob(s) Details The value for one of the HTTP headers is not in the correct format. RequestId:944807de-001a-00bb-73ae-4ac746000000 Time:2016-11-30T02:04:59.8679984Z

谁能帮我解决这个问题？

谢谢！

Answer 1

Azure Batch resource files 只能来自 Azure Blob 存储，而不是 Azure 文件存储。

您将需要 net use 安装共享并手动复制文件或将文件移至 Azure Blob 存储。

Azure Batch Pool 启动任务以从 Blob FileShare 下载资源文件

Azure Batch Pool Start up task to download resource file from Blob FileShare

azure

azure-blob-storage

azure-batch