Long 运行 Azure WebJob 失败 - 客户端无法在指定的超时时间内完成操作

Question

我有一个很长的运行 Azure WebJob (2-4h)，它在大约 90 分钟后一直失败并出现存储异常。我正在使用 WebJobs 2.3.0 SDK 和 WindowsAzure.Storage 9.3.3。

[09/17/2019 05:14:23 > b0c2e2: ERR ] 
[09/17/2019 05:14:23 > b0c2e2: ERR ] Unhandled Exception: Microsoft.WindowsAzure.Storage.StorageException: The client could not finish the operation within specified timeout. ---> System.TimeoutException: The client could not finish the operation within specified timeout.
[09/17/2019 05:14:23 > b0c2e2: ERR ]    --- End of inner exception stack trace ---
[09/17/2019 05:14:23 > b0c2e2: ERR ]    at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndExecuteAsync[T](IAsyncResult result) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Executor\Executor.cs:line 51
[09/17/2019 05:14:23 > b0c2e2: ERR ]    at Microsoft.WindowsAzure.Storage.Queue.CloudQueue.EndExists(IAsyncResult asyncResult) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Queue\CloudQueue.cs:line 994
[09/17/2019 05:14:23 > b0c2e2: ERR ]    at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.<>c__DisplayClass2`1.<CreateCallback>b__0(IAsyncResult ar) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Util\AsyncExtensions.cs:line 69
[09/17/2019 05:14:23 > b0c2e2: ERR ] --- End of stack trace from previous location where exception was thrown ---
[09/17/2019 05:14:23 > b0c2e2: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
[09/17/2019 05:14:23 > b0c2e2: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
[09/17/2019 05:14:23 > b0c2e2: ERR ]    at Microsoft.Azure.WebJobs.Host.Queues.Listeners.QueueListener.<ExecuteAsync>d__25.MoveNext()
[09/17/2019 05:14:23 > b0c2e2: ERR ] --- End of stack trace from previous location where exception was thrown ---
[09/17/2019 05:14:23 > b0c2e2: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
[09/17/2019 05:14:23 > b0c2e2: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
[09/17/2019 05:14:23 > b0c2e2: ERR ]    at Microsoft.Azure.WebJobs.Host.Timers.TaskSeriesTimer.<RunAsync>d__14.MoveNext()
[09/17/2019 05:14:23 > b0c2e2: ERR ] --- End of stack trace from previous location where exception was thrown ---
[09/17/2019 05:14:23 > b0c2e2: ERR ]    at Microsoft.Azure.WebJobs.Host.Timers.WebJobsExceptionHandler.<>c__DisplayClass3_0.<OnUnhandledExceptionAsync>b__0()
[09/17/2019 05:14:23 > b0c2e2: ERR ]    at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
[09/17/2019 05:14:23 > b0c2e2: ERR ]    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[09/17/2019 05:14:23 > b0c2e2: ERR ]    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[09/17/2019 05:14:23 > b0c2e2: ERR ]    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
[09/17/2019 05:14:23 > b0c2e2: ERR ]    at System.Threading.ThreadHelper.ThreadStart()
[09/17/2019 05:14:23 > b0c2e2: SYS ERR ] Job failed due to exit code -532462766
[09/17/2019 05:14:23 > b0c2e2: SYS INFO] Process went down, waiting for 0 seconds
[09/17/2019 05:14:23 > b0c2e2: SYS INFO] Status changed to PendingRestart

这项工作没有对 Azure 存储做任何事情，从其他问题我收集到这可能与 WebJob 将日志文件写入 Azure 存储有关，为此我配置了一个自定义的 StorageClientFactory，服务器超时时间很长但是这似乎没有什么区别。

工作配置：

var config = new JobHostConfiguration()
config.Queues.MaxDequeueCount = 1; 
config.Queues.BatchSize = 1; 
ServicePointManager.DefaultConnectionLimit = int.MaxValue;
config.StorageClientFactory = new CustomStorageClientFactory();
var host = new JobHost(config);
host.RunAndBlock();



public class CustomStorageClientFactory : StorageClientFactory
{
    public override CloudBlobClient CreateCloudBlobClient(StorageClientFactoryContext context)
    {
        CloudBlobClient client = context.Account.CreateCloudBlobClient();
        client.DefaultRequestOptions.ServerTimeout = TimeSpan.FromHours(6);
        return client;
    }
}

Answer 1

这应该是 Azure WebJob SDK 2.X 的设计问题。在后端，它使用 HttpWebRequest 访问存储 API.The 问题是默认情况下每个服务器只允许 2 个并发连接。所以如果2个http连接被其他请求占用，其他异步请求会超时。

解决方法是将 DefaultConnectionLimit 设置为更大的值，如下所示：

static void Main(string[] args)
{
    // Set this immediately so that it's used by all requests.
    ServicePointManager.DefaultConnectionLimit = Int32.MaxValue;

    var host = new JobHost();
    host.RunAndBlock();
}

您也可以简单地将 Azure Web SDK 升级到解决此问题的 3.x 版本。

详见Managing concurrent connections and https://github.com/Azure/azure-webjobs-sdk/issues/755#issuecomment-319094679。

Long 运行 Azure WebJob 失败 - 客户端无法在指定的超时时间内完成操作

Long running Azure WebJob fails - The client could not finish the operation within specified timeout

asp.net

azure

azure-storage

azure-webjobs

azure-storage-queues