清漆(?)中断文件下载

Varnish (?) interrupting file download

在 debian buster 服务器上,我有一个 apache2 网络服务器,前面有一个缓存作为缓存,实际上 varnish 侦听端口 80 并从侦听端口 8080 的 apache 获取。

可能自从 stretch 升级后,文件下载不再完成。

正在使用 curl 下载:

$ wget -v -t 1 https://myserver.org/myfile.zip
--2020-07-10 19:52:34--  https://myserver.org/myfile.zip
Resolving myserver.org (myserver.org)... w.x.y.z
Connecting to myserver.org (myserver.org)|w.x.y.z|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 333774113 (318M) [application/zip]
Saving to: 'myfile.zip.13'

myfile.zip.13                                                         1%[=>    ]   4.43M  8.29MB/s    in 0.5s    

2020-07-10 19:52:35 (8.29 MB/s) - Connection closed at byte 4649063. Giving up.

每个请求的传输量都不同。

这是此请求的清漆日志:

*   << Request  >> 2104598   
-   Begin          req 2104597 rxreq
-   Timestamp      Start: 1594404000.669087 0.000000 0.000000
-   Timestamp      Req: 1594404000.669087 0.000000 0.000000
-   ReqStart       127.0.0.1 40966 a0
-   ReqMethod      GET
-   ReqURL         /myfile.zip
-   ReqProtocol    HTTP/1.1
-   ReqHeader      User-Agent: Wget/1.19.4 (linux-gnu)
-   ReqHeader      Accept: */*
-   ReqHeader      Accept-Encoding: identity
-   ReqHeader      Host: myserver.org
-   ReqHeader      X-SSL: 1
-   ReqHeader      X-Forwarded-Proto: https
-   ReqHeader      X-Forwarded-For: a.b.c.d
-   ReqHeader      Connection: close
-   ReqUnset       X-Forwarded-For: a.b.c.d
-   ReqHeader      X-Forwarded-For: a.b.c.d, 127.0.0.1
-   VCL_call       RECV
-   ReqUnset       Accept-Encoding: identity
-   VCL_return     hash
-   VCL_call       HASH
-   VCL_return     lookup
-   VCL_call       MISS
-   VCL_return     fetch
-   Link           bereq 2104599 fetch
-   Timestamp      Fetch: 1594404000.669596 0.000509 0.000509
-   RespProtocol   HTTP/1.1
-   RespStatus     200
-   RespReason     OK
-   RespHeader     Date: Fri, 10 Jul 2020 18:00:00 GMT
-   RespHeader     Last-Modified: Sun, 28 Jun 2020 14:46:46 GMT
-   RespHeader     ETag: "13e4fd21-5a92602e5bcc1"
-   RespHeader     Content-Length: 333774113
-   RespHeader     Content-Type: application/zip
-   RespHeader     X-Varnish: 2104598
-   RespHeader     Age: 0
-   RespHeader     Via: 1.1 varnish (Varnish/6.1)
-   VCL_call       DELIVER
-   RespHeader     X-Cache: uncached
-   RespUnset      Via: 1.1 varnish (Varnish/6.1)
-   RespUnset      X-Varnish: 2104598
-   VCL_return     deliver
-   Timestamp      Process: 1594404000.669609 0.000521 0.000012
-   RespHeader     Accept-Ranges: bytes
-   RespHeader     Connection: close
-   Timestamp      Resp: 1594404000.884468 0.215381 0.214859
-   ReqAcct        237 0 237 260 4649063 4649323
-   End

这里是后端交互(其他wget 运行,在字节6291456关闭):

*   << BeReq    >> 2612416   
-   Begin          bereq 2612415 fetch
-   Timestamp      Start: 1594674394.093379 0.000000 0.000000
-   BereqMethod    GET
-   BereqURL       /myfile.zip
-   BereqProtocol  HTTP/1.1
-   BereqHeader    User-Agent: Wget/1.19.4 (linux-gnu)
-   BereqHeader    Accept: */*
-   BereqHeader    Host: myserver.org
-   BereqHeader    X-SSL: 1
-   BereqHeader    X-Forwarded-Proto: https
-   BereqHeader    X-Forwarded-For: a.b.c.d, 127.0.0.1
-   BereqHeader    Accept-Encoding: gzip
-   BereqHeader    X-Varnish: 2612416
-   VCL_call       BACKEND_FETCH
-   VCL_return     fetch
-   BackendOpen    28 default 127.0.0.1 8080 127.0.0.1 50218
-   BackendStart   127.0.0.1 8080
-   Timestamp      Bereq: 1594674394.093413 0.000034 0.000034
-   Timestamp      Beresp: 1594674394.093710 0.000330 0.000297
-   BerespProtocol HTTP/1.1
-   BerespStatus   200
-   BerespReason   OK
-   BerespHeader   Date: Mon, 13 Jul 2020 21:06:34 GMT
-   BerespHeader   Server: Apache
-   BerespHeader   Last-Modified: Sun, 28 Jun 2020 14:46:46 GMT
-   BerespHeader   ETag: "13e4fd21-5a92602e5bcc1"
-   BerespHeader   Accept-Ranges: bytes
-   BerespHeader   Content-Length: 333774113
-   BerespHeader   Content-Type: application/zip
-   TTL            RFC 120 10 0 1594674394 1594674394 1594674394 0 0 cacheable
-   VCL_call       BACKEND_RESPONSE
-   BerespUnset    Server: Apache
-   TTL            VCL 1800 10 0 1594674394 cacheable
-   TTL            VCL 1800 30 0 1594674394 cacheable
-   VCL_return     deliver
-   Filters        
-   Storage        malloc s0
-   Fetch_Body     3 length stream
-   ExpKill        LRU_Cand p=0x7f97b4a358c0 f=0x0 r=1
-   ExpKill        LRU x=2631322
-   ExpKill        LRU_Cand p=0x7f97efa67980 f=0x0 r=1
-   ExpKill        LRU x=2284258
-   ExpKill        LRU_Cand p=0x7f97b4a6b1c0 f=0x0 r=1
-   ExpKill        LRU x=2631334
-   ExpKill        LRU_Cand p=0x7f97efa67a40 f=0x0 r=1
-   ExpKill        LRU x=2284266
-   ExpKill        LRU_Cand p=0x7f97efa67b00 f=0x0 r=1
-   ExpKill        LRU x=2284269
-   ExpKill        LRU_Cand p=0x7f97d484c780 f=0x0 r=1
-   ExpKill        LRU x=1667855
-   ExpKill        LRU_Cand p=0x7f97d9af1bc0 f=0x0 r=1
-   ExpKill        LRU x=2950859
-   ExpKill        LRU_Cand p=0x7f97e7886040 f=0x0 r=1
-   ExpKill        LRU x=2472240
-   ExpKill        LRU_Cand p=0x7f97d9af1c80 f=0x0 r=1
-   ExpKill        LRU x=2950862
-   ExpKill        LRU_Cand p=0x7f97d484e640 f=0x0 r=1
-   ExpKill        LRU x=1667862
-   ExpKill        LRU_Cand p=0x7f97d484eac0 f=0x0 r=1
-   ExpKill        LRU x=1667865
-   ExpKill        LRU_Cand p=0x7f97d9af1e00 f=0x0 r=1
-   ExpKill        LRU x=2950865
-   ExpKill        LRU_Cand p=0x7f97d48d2180 f=0x0 r=1
-   ExpKill        LRU x=1667871
-   ExpKill        LRU_Cand p=0x7f97d9af2040 f=0x0 r=1
-   ExpKill        LRU x=2950871
-   ExpKill        LRU_Cand p=0x7f97d9af2100 f=0x0 r=1
-   ExpKill        LRU x=2950876
-   ExpKill        LRU_Cand p=0x7f97b4b11000 f=0x0 r=1
-   ExpKill        LRU x=2631343
-   ExpKill        LRU_Cand p=0x7f97efa67bc0 f=0x0 r=1
-   ExpKill        LRU x=2284275
-   ExpKill        LRU_Cand p=0x7f97efa67c80 f=0x0 r=1
-   ExpKill        LRU x=2284278
-   ExpKill        LRU_Cand p=0x7f97d48d3b00 f=0x0 r=1
-   ExpKill        LRU x=1667880
-   ExpKill        LRU_Cand p=0x7f97e7886100 f=0x0 r=1
-   ExpKill        LRU x=2472249
-   ExpKill        LRU_Cand p=0x7f97d9a3d3c0 f=0x0 r=1
-   ExpKill        LRU x=2950879
-   ExpKill        LRU_Cand p=0x7f97d9a3d900 f=0x0 r=1
-   ExpKill        LRU x=2950886
-   ExpKill        LRU_Cand p=0x7f97d48d4040 f=0x0 r=1
-   ExpKill        LRU x=1667895
-   ExpKill        LRU_Cand p=0x7f97e79d1e80 f=0x0 r=1
-   ExpKill        LRU x=2631367
-   ExpKill        LRU_Cand p=0x7f97d6bf6780 f=0x0 r=1
-   ExpKill        LRU x=1205206
-   ExpKill        LRU_Cand p=0x7f97cbb3c2c0 f=0x0 r=1
-   ExpKill        LRU x=2325555
-   ExpKill        LRU_Cand p=0x7f97d6bf6840 f=0x0 r=1
-   ExpKill        LRU x=1205217
-   ExpKill        LRU_Cand p=0x7f97e7886340 f=0x0 r=1
-   ExpKill        LRU x=2472273
-   ExpKill        LRU_Cand p=0x7f97cbb3c380 f=0x0 r=1
-   ExpKill        LRU x=2325569
-   ExpKill        LRU_Cand p=0x7f97cbb3c440 f=0x0 r=1
-   ExpKill        LRU x=2325572
-   ExpKill        LRU_Cand p=0x7f97d6bf69c0 f=0x0 r=1
-   ExpKill        LRU x=1205230
-   ExpKill        LRU_Cand p=0x7f97e7886400 f=0x0 r=1
-   ExpKill        LRU x=2472276
-   ExpKill        LRU_Cand p=0x7f97d6bf6b40 f=0x0 r=1
-   ExpKill        LRU x=1205239
-   ExpKill        LRU_Cand p=0x7f97b4b11480 f=0x0 r=1
-   ExpKill        LRU x=2631405
-   ExpKill        LRU_Cand p=0x7f97b4b11540 f=0x0 r=1
-   ExpKill        LRU x=2631410
-   ExpKill        LRU_Cand p=0x7f97e7886640 f=0x0 r=1
-   ExpKill        LRU x=2472294
-   ExpKill        LRU_Cand p=0x7f97b4b116c0 f=0x0 r=1
-   ExpKill        LRU x=2631416
-   ExpKill        LRU_Cand p=0x7f97e7886700 f=0x0 r=1
-   ExpKill        LRU x=2472299
-   ExpKill        LRU_Cand p=0x7f97b4b11780 f=0x0 r=1
-   ExpKill        LRU x=2631419
-   ExpKill        LRU_Cand p=0x7f97e78867c0 f=0x0 r=1
-   ExpKill        LRU x=2472302
-   ExpKill        LRU_Cand p=0x7f97e7886880 f=0x0 r=1
-   ExpKill        LRU x=2472305
-   ExpKill        LRU_Cand p=0x7f97b4b11b40 f=0x0 r=1
-   ExpKill        LRU x=2631428
-   ExpKill        LRU_Cand p=0x7f97e7886940 f=0x0 r=1
-   ExpKill        LRU x=2472308
-   ExpKill        LRU_Cand p=0x7f97e7886ac0 f=0x0 r=1
-   ExpKill        LRU x=2472314
-   ExpKill        LRU_Cand p=0x7f97b4b12080 f=0x0 r=1
-   ExpKill        LRU x=2631434
-   ExpKill        LRU_Cand p=0x7f97d6bf6c00 f=0x0 r=1
-   ExpKill        LRU x=1205267
-   ExpKill        LRU_Cand p=0x7f97e7886b80 f=0x0 r=1
-   ExpKill        LRU x=2472348
-   ExpKill        LRU_Cand p=0x7f97d6bf6cc0 f=0x0 r=1
-   ExpKill        LRU x=1205274
-   ExpKill        LRU_Cand p=0x7f97b4b12200 f=0x0 r=1
-   ExpKill        LRU x=2631441
-   ExpKill        LRU_Cand p=0x7f97e7886c40 f=0x0 r=1
-   ExpKill        LRU x=2472360
-   ExpKill        LRU reached nuke_limit
-   FetchError     Could not get storage
-   BackendClose   28 default
-   BereqAcct      244 0 244 230 10764288 10764518
-   End            

我无法理解发生了什么...

这是我的清漆 default.vcl:

# Marker to tell the VCL compiler that this VCL has been adapted to the
# new 4.0 format.
vcl 4.0;

# Default backend definition. Set this to point to your content server.
backend default {
    .host = "127.0.0.1";
    .port = "8080";
}

# access control list for "purge": open to only localhost and other local nodes
acl purge {
    "127.0.0.1";
    "localhost";
}

# vcl_recv is called whenever a request is received 
sub vcl_recv {
        # Serve objects up to 2 minutes past their expiry if the backend
        # is slow to respond.
        #set req.grace = 120s;
        if (req.http.X-Forwarded-Proto !~ "https")
        {
          unset req.http.X-Forwarded-For;
          set req.http.X-Forwarded-For = client.ip;
        }
        set req.backend_hint= default;

        # This uses the ACL action called "purge". Basically if a request to
        # PURGE the cache comes from anywhere other than localhost, ignore it.
        if (req.method == "PURGE") 
            {
                if (!client.ip ~ purge)
                    {return(synth(405,"Not allowed."));}
                #return(hash);}
                return(purge);
            }

        # Pass any requests that Varnish does not understand straight to the backend.
        if (req.method != "GET" && req.method != "HEAD" &&
            req.method != "PUT" && req.method != "POST" &&
            req.method != "TRACE" && req.method != "OPTIONS" &&
            req.method != "DELETE") 
            {return(pipe);}     /* Non-RFC2616 or CONNECT which is weird. */

        # Pass anything other than GET and HEAD directly.
        if (req.method != "GET" && req.method != "HEAD")
           {return(pass);}      /* We only deal with GET and HEAD by default */

        # Pass requests from logged-in users directly.
        #if (req.http.Authorization || req.http.Cookie)
        #   {return(pass);}      /* Not cacheable by default */

        # Pass any requests with the "If-None-Match" header directly.
        if (req.http.If-None-Match)
           {return(pass);}

        # Force lookup if the request is a no-cache request from the client.
        # DISATTIVO PERCHÉ ban RICHIEDE UNA COMPARAZIONE
        #if (req.http.Cache-Control ~ "no-cache")
        #   {ban(req.url);}

        # normalize Accept-Encoding to reduce vary
        if (req.http.Accept-Encoding) {
          if (req.http.User-Agent ~ "MSIE 6") {
            unset req.http.Accept-Encoding;
          } elsif (req.http.Accept-Encoding ~ "gzip") {
            set req.http.Accept-Encoding = "gzip";
          } elsif (req.http.Accept-Encoding ~ "deflate") {
            set req.http.Accept-Encoding = "deflate";
          } else {
            unset req.http.Accept-Encoding;
          }
        }


        # remove the cookies form static files in order to serve them from cache
        if (req.url ~ "(?i)\.(jpeg|jpg|png|gif|ico|webp|js|css|txt|pdf|gz|zip|lzma|bz2|tgz|tbz|html|htm)$") {
            unset req.http.cookie;
        }

        # Do not cache HTTP authentication and HTTP Cookie
        if (req.http.Authorization || req.http.Cookie) {
            # Not cacheable by default
            return (pass);
        }

        return(hash);
        #return(pass);
}

sub vcl_pipe {
        # Note that only the first request to the backend will have
        # X-Forwarded-For set.  If you use X-Forwarded-For and want to
        # have it set for all requests, make sure to have:
        # set req.http.connection = "close";
 
        # This is otherwise not necessary if you do not do any request rewriting.
 
        set req.http.connection = "close";
}

# Called if the cache has a copy of the page.
sub vcl_hit {
        if (req.method == "PURGE") 
            {ban(req.url);
            return(synth(200,"Purged"));}
 
        if (!obj.ttl > 0s)
           {return(pass);}
}
 
# Called if the cache does not have a copy of the page.
sub vcl_miss {
        if (req.method == "PURGE") 
           {return(synth(200,"Not in cache"));}
}

# Called after a document has been successfully retrieved from the backend.
sub vcl_backend_response {
 
    # Remove some headers we never want to see
    unset beresp.http.Server;
    unset beresp.http.X-Powered-By;

    # For static content strip all backend cookies
    if (bereq.url ~ "\.(css|js|png|gif|jp(e?)g)|swf|ico") {
        unset beresp.http.cookie;
    }
    # Don't store backend
    if (bereq.url ~ "wp-(login|admin)" || bereq.url ~ "preview=true") {
        set beresp.uncacheable = true;
        set beresp.ttl = 30s;
        return (deliver);
    }


    # don't cache response to posted requests or those with basic auth
    if ( bereq.method == "POST" || bereq.http.Authorization ) {
            set beresp.uncacheable = true;
        set beresp.ttl = 120s;
        return (deliver);
        }

        # don't cache search results
    if ( bereq.url ~ "\?s=" ){
        set beresp.uncacheable = true;
                set beresp.ttl = 120s;
                return (deliver);
    }

    # only cache status ok
    if ( beresp.status != 200 ) {
        set beresp.uncacheable = true;
                set beresp.ttl = 120s;
                return (deliver);
    }

    # A TTL of 2h
    set beresp.ttl = 30m;
    # Define the default grace period to serve cached content
    set beresp.grace = 30s;
        return (deliver);
}

sub vcl_deliver {
    if (obj.hits > 0) {
        set resp.http.X-Cache = "cached";
    } else {
        set resp.http.x-Cache = "uncached";
    }

    # Remove some headers: PHP version
    unset resp.http.X-Powered-By;

    # Remove some headers: Apache version & OS
    unset resp.http.Server;

    # Remove some heanders: Varnish
    unset resp.http.Via;
    unset resp.http.X-Varnish;
    
    return(deliver);
}

#sub vcl_fetch {
#   if (req.url ~ "^/piwik") {
#       return(hit_for_pass);
#   }
#}

根据后端 varnishlog 信息,我可以断定您 运行 存储空间不足。

这些是确认这一点的日志项:

-   ExpKill        LRU reached nuke_limit
-   FetchError     Could not get storage

当缓存已满且 Varnish 想要在缓存中存储另一个对象时,LRU(最近最少使用) 机制将启动。

这意味着它开始在缓存中寻找最近最少使用的项目,并将它们从缓存中逐出。 Varnish 多次执行此操作,直到它到达默认设置为 50nuke_limit

在这种情况下,Varnish 将尝试删除 50 个最近最少使用的对象,以便将新对象放入缓存中。

我的假设是 50 个最近最少使用的对象在大小方面相当小,释放的 space 不符合新对象的大小要求。

这些是您的一些选择:

  • 使用 -p 运行时选项增加 nuke_limit 参数
  • 对某些对象应用不同的 TTL 以确保它们更快过期
  • 主动使部分缓存无效以保证 space 可用
  • 增加内存大小以避免这些问题
  • 使用 Varnish Enterprise and benefit from the Massive Storage Engine 始终保持 space 在缓存中可用,并动态调整缓存大小。