Azure Redis 缓存 - 多个错误 TimeoutException：执行 GET {key} 超时

Question

我们将应用程序部署到 Azure。它正在使用 Azure Redis 缓存，我们遇到了很多超时。即：

[TimeoutException: Timeout performing GET textobjectDetails__23290_TextObject, inst: 1, mgr: Inactive, queue: 5, qu=0, qs=5, qc=0, wr=0/0, in=56864/0]

[TimeoutException: Timeout performing GET featured_series_CachedSeries, inst: 1, mgr: Inactive, queue: 4, qu=0, qs=4, qc=0, wr=0/0, in=44470/0]

[TimeoutException: Timeout performing GET SeriesByFranchiseId_1_CachedSeries, inst: 1, mgr: Inactive, queue: 3, qu=0, qs=3, qc=0, wr=0/0, in=11252/0]

[TimeoutException: Timeout performing GET media_silo-1-1-0_Media, inst: 1, mgr: Inactive, queue: 3, qu=0, qs=3, qc=0, wr=0/0, in=15188/0]

[TimeoutException: Timeout performing GET textobjectDetails__3092_TextObject, inst: 3, mgr: Inactive, queue: 7, qu=0, qs=7, qc=0, wr=0/0, in=65536/0]

[TimeoutException: Timeout performing GET textobjectbytype_104__TextObject, inst: 11, mgr: Inactive, queue: 9, qu=0, qs=9, qc=0, wr=0/0, in=65536/0]

[TimeoutException: Timeout performing GET groupnews_2_14_TextObject, inst: 1, mgr: Inactive, queue: 7, qu=0, qs=7, qc=0, wr=0/0, in=65536/0]

[TimeoutException: Timeout performing GET archived_news_by_group_13586_1_TextObject, inst: 2, mgr: Inactive, queue: 7, qu=0, qs=7, qc=0, wr=0/0, in=65536/0]

[TimeoutException: Timeout performing GET textobjectDetails__24404_TextObject, inst: 11, mgr: Inactive, queue: 12, qu=0, qs=12, qc=0, wr=0/0, in=65536/0]

[TimeoutException: Timeout performing GET standings_232_lcds_fixtures, inst: 2, mgr: Inactive, queue: 11, qu=0, qs=11, qc=0, wr=0/0, in=65536/0]

[TimeoutException: Timeout performing GET player_name1099_Player, inst: 4, mgr: Inactive, queue: 11, qu=0, qs=11, qc=0, wr=0/0, in=65536/0]

[TimeoutException: Timeout performing GET groupnews_1_6_TextObject, inst: 4, mgr: Inactive, queue: 9, qu=0, qs=9, qc=0, wr=0/0, in=65536/0]

[TimeoutException: Timeout performing GET archivednewscount__20789_TextObject, inst: 2, mgr: Inactive, queue: 11, qu=0, qs=11, qc=0, wr=0/0, in=65536/0]

[TimeoutException: Timeout performing GET media_id3648_Media, inst: 1, mgr: Inactive, queue: 10, qu=0, qs=10, qc=0, wr=0/0, in=65536/0]

所有异常的正文都是相同的：

StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor`1 processor, ServerEndPoint server):509
    StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor`1 processor, ServerEndPoint server):25
    StackExchange.Redis.RedisDatabase.StringGet(RedisKey key, CommandFlags flags):16
    AB.SiteCaching.Providers.RedisDataSource+<>c__DisplayClasse`1.<RetrieveCacheObject>b__b():0
    Microsoft.Practices.TransientFaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func):115
    AB.SiteCaching.Providers.RedisDataSource.RetrieveCacheObject[T](String fullCacheKey):56
    AB.SiteCaching.Providers.RedisDataSource.RetrieveCached[T](String key, Func`1 onNotCached, TimeSpan timeOut):61
    DataAccess.Data.Caching.CachedSeries.GetSeriesByFranchiseId(Int32 franchiseId):64
    Shared.Services.SeriesService.LatestYearByFranchiseId(Int32 franchiseId):0
    AllBlacksdotcom.Controllers.FixturesController._MostRecentYearOfFixtures(Int32 franchiseId):0
    (unknown).lambda_method(Closure , ControllerBase , Object[] ):-1
    System.Web.Mvc.ActionMethodDispatcher.Execute(ControllerBase controller, Object[] parameters):0
    System.Web.Mvc.ReflectedActionDescriptor.Execute(ControllerContext controllerContext, IDictionary`2 parameters):87
    System.Web.Mvc.ControllerActionInvoker.InvokeActionMethod(ControllerContext controllerContext, ActionDescriptor actionDescriptor, IDictionary`2 parameters):0
    System.Web.Mvc.Async.AsyncControllerActionInvoker.<BeginInvokeSynchronousActionMethod>b__39(IAsyncResult asyncResult, ActionInvocation innerInvokeState):0
    System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResult`2.CallEndDelegate(IAsyncResult asyncResult):0
    System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResultBase`1.End():41
    System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethod(IAsyncResult asyncResult):0
    System.Web.Mvc.Async.AsyncControllerActionInvoker+AsyncInvocationWithFilters.<InvokeActionMethodFilterAsynchronouslyRecursive>b__3d():20
    System.Web.Mvc.Async.AsyncControllerActionInvoker+AsyncInvocationWithFilters+<>c__DisplayClass46.<InvokeActionMethodFilterAsynchronouslyRecursive>b__3f():134
    System.Web.Mvc.Async.AsyncControllerActionInvoker+<>c__DisplayClass33.<BeginInvokeActionMethodWithFilters>b__32(IAsyncResult asyncResult):0
    System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResult`1.CallEndDelegate(IAsyncResult asyncResult):0
    System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResultBase`1.End():41
    System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethodWithFilters(IAsyncResult asyncResult):0
    System.Web.Mvc.Async.AsyncControllerActionInvoker+<>c__DisplayClass21+<>c__DisplayClass2b.<BeginInvokeAction>b__1c():0
    System.Web.Mvc.Async.AsyncControllerActionInvoker+<>c__DisplayClass21.<BeginInvokeAction>b__1e(IAsyncResult asyncResult):65
    System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResult`1.CallEndDelegate(IAsyncResult asyncResult):0
    System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResultBase`1.End():41
    System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeAction(IAsyncResult asyncResult):0
    System.Web.Mvc.Controller.<BeginExecuteCore>b__1d(IAsyncResult asyncResult, ExecuteCoreState innerState):0
    System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult):0
    System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResultBase`1.End():41
    System.Web.Mvc.Controller.EndExecuteCore(IAsyncResult asyncResult):0
    System.Web.Mvc.Controller.<BeginExecute>b__15(IAsyncResult asyncResult, Controller controller):0
    System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult):0
    System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResultBase`1.End():41
    System.Web.Mvc.Controller.EndExecute(IAsyncResult asyncResult):0
    System.Web.Mvc.Controller.System.Web.Mvc.Async.IAsyncController.EndExecute(IAsyncResult asyncResult):0
    System.Web.Mvc.MvcHandler.<BeginProcessRequest>b__5(IAsyncResult asyncResult, ProcessRequestState innerState):0
    System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult):0
    System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResultBase`1.End():41
    System.Web.Mvc.MvcHandler.EndProcessRequest(IAsyncResult asyncResult):0
    System.Web.Mvc.MvcHandler.System.Web.IHttpAsyncHandler.EndProcessRequest(IAsyncResult result):0
    System.Web.Mvc.HttpHandlerUtil+ServerExecuteHttpHandlerAsyncWrapper+<>c__DisplayClassa.<EndProcessRequest>b__9():0
    System.Web.Mvc.HttpHandlerUtil+ServerExecuteHttpHandlerWrapper+<>c__DisplayClass4.<Wrap>b__3():0
    System.Web.Mvc.HttpHandlerUtil+ServerExecuteHttpHandlerWrapper.Wrap[TResult](Func`1 func):0
    System.Web.Mvc.HttpHandlerUtil+ServerExecuteHttpHandlerWrapper.Wrap(Action action):25
    System.Web.Mvc.HttpHandlerUtil+ServerExecuteHttpHandlerAsyncWrapper.EndProcessRequest(IAsyncResult result):32
    System.Web.HttpServerUtility.ExecuteInternal(IHttpHandler handler, TextWriter writer, Boolean preserveForm, Boolean setPreviousPage, VirtualPath path, VirtualPath filePath, String physPath, Exception error, String queryStringOverride):772

请查看我们的超时设置：

retryTimeoutInMilliseconds = "5000" 
connectionTimeoutInMilliseconds = "5000" 
operationTimeoutInMilliseconds = "1000"

如何处理这些超时？增加 operationTimeoutInMilliseconds 会成功吗？我还了解到 g-zip 压缩有助于减少从 Redis 读取数据所需的时间。

所有与 Redis 相关的 nuget 包都是最新版本。我们正在使用 C1 版本的 Azure Redis 缓存（增加到 C2 会有帮助吗？）。

Answer 1

在您所有的异常消息中，您都有类似 "in=65536/0" 的内容。这表明 65536 字节已在本地接收（在本地机器上的内核套接字缓冲区中）但尚未被 StackExchange.Redis 处理。这将指出客户端应用程序方面的问题，导致系统无法处理您的响应。我能想到的两种可能会导致这种情况：

您的系统没有足够的空闲线程池线程来足够快地处理数据。如果您更新到 StackExchange.Redis 的 1.0.450 或更高版本，超时消息将包含有关线程池繁忙程度的统计信息，这将有助于诊断此问题。或者，您可以使用 https://github.com/JonCole/SampleCode/blob/master/ThreadPoolMonitor/ThreadPoolLogger.cs 之类的代码在遇到异常时自己记录相同的信息。
您正在从缓存中接收到一个非常大的对象，并且在 StackExchange.Redis 配置的超时时间内未接收到该对象的大小。发生这种情况时，在大项目超时时，所有在大项目后面排队到同一连接上的调用都会超时。

无论如何，希望这对您有所帮助。

Answer 2

转到 Azure 并获得更大的 Redis 实例。那t

Answer 3

改善我们情况的几点：

Protobuf-net 而不是 BinaryFormatter

我建议使用 protobuf-net，因为它会减少要存储在缓存中的值的大小。

public interface ICacheDataSerializer
    {
        byte[] Serialize(object o);
        T Deserialize<T>(byte[] stream);
    }

public class ProtobufNetSerializer : ICacheDataSerializer
    {
        public byte[] Serialize(object o)
        {
            using (var memoryStream = new MemoryStream())
            {
                Serializer.Serialize(memoryStream, o);

                return memoryStream.ToArray();
            }
        }

        public T Deserialize<T>(byte[] stream)
        {
            var memoryStream = new MemoryStream(stream);

            return Serializer.Deserialize<T>(memoryStream);
        }
    }

实施重试策略

实施这个 RedisCacheTransientErrorDetectionStrategy 来处理超时问题。

using Microsoft.Practices.TransientFaultHandling;

public class RedisCacheTransientErrorDetectionStrategy : ITransientErrorDetectionStrategy
    {
        /// <summary>
        /// Custom Redis Transient Error Detenction Strategy must have been implemented to satisfy Redis exceptions.
        /// </summary>
        /// <param name="ex"></param>
        /// <returns></returns>
        public bool IsTransient(Exception ex)
        {
            if (ex == null) return false;

            if (ex is TimeoutException) return true;

            if (ex is RedisServerException) return true;

            if (ex is RedisException) return true;

            if (ex.InnerException != null)
            {
                return IsTransient(ex.InnerException);
            }

            return false;
        }
    }

像这样实例化：

private readonly RetryPolicy _retryPolicy;

// CODE
var retryStrategy = new FixedInterval(3, TimeSpan.FromSeconds(2));
            _retryPolicy = new RetryPolicy<RedisCacheTransientErrorDetectionStrategy>(retryStrategy);

这样使用：

var cachedString = _retryPolicy.ExecuteAction(() => dataCache.StringGet(fullCacheKey));

检查您的代码 以尽量减少缓存调用和存储在缓存中的值。我通过更有效地存储值减少了很多错误。

如果 none 有帮助。移动到更高的缓存（我们最终使用 C3 而不是 C1）。

Azure Redis 缓存 - 多个错误 TimeoutException：执行 GET {key} 超时

Azure Redis Cache - Multiple errors TimeoutException: Timeout performing GET {key}

caching

azure

redis

stackexchange.redis

azure-redis-cache