varnish 错误 503 后端提取仅在通过 nginx 交付的 Web 应用程序上失败(附有 varnishlog)

varnish Error 503 Backend fetch failed only on web app delivered via nginx (varnishlog attached)

我想用清漆缓存两个应用程序。一个是静态 Vue 应用程序,第二个应用程序是 Go REST API 服务器。全部在 docker 群上运行并且 API 响应良好,但是当我通过清漆查询 Vue 容器时,我得到 503 Backend fetch failed 错误。

这两个应用程序都可以通过 http://ip:8080http://ip:3000/api/... 在服务器上直接访问,但如前所述,静态 Web 应用程序不响应(端口=8080)。

我知道 varnish 回复默认值是不健康的,但我不知道为什么,因为它是可访问的并且我在 default.vcl 中添加了超时等(见下文)。

我运行日志得到错误:

$ docker exec -it ... varnishlog -g raw -i backend_health
         0 Backend_health - default Still sick -------- 0 3 5 0.000000 0.000000 "Open error 110 (Connection timed out)"

怀疑

我唯一的怀疑是 Vue 应用程序的 nginx 容器中的 URL 重定向以 .probe 不起作用的方式重定向请求。这是 Vue 应用程序的 nginx.conf

server {
    listen          80;
    server_name     _ default_server;
    index           index.html;

    location / {
        root        /usr/share/nginx/html;
        index       index.html;
        try_files   $uri $uri/ /index.html;
    }
}

文件

这是docker-compose.yml文件(我只是添加了host.docker.internal用于测试,当我使用不同的地址时没有任何变化,例如:client,服务器IP或host.docker.internal 作为 default.vcl 中的 host

version: '3.9'
services:
  server:
    image: ...
    ports:
      - target: 3000
        published: 3000
        mode: host
    networks:
      - web
      - db-net
    depends_on:
      - db
    command: [ "./wait-for-it.sh", "db:5432", "--", "./streamsink" ]
    deploy:
      mode: replicated
      replicas: 1
      restart_policy:
        condition: on-failure
        delay: 10s
        max_attempts: 10
        window: 120s
      placement:
        constraints:
          - node.role == manager
    client:
        image: ...
        extra_hosts:
          - "host.docker.internal:host-gateway"
        ports:
          - target: 80
            published: 8080
            mode: host
        depends_on:
          - server
        networks:
          - web
        deploy:
          restart_policy:
            condition: on-failure

  varnish:
    image: varnish:latest
    extra_hosts:
      - "host.docker.internal:host-gateway"
    ports:
      - target: 80
        published: 80
        mode: host
    environment:
      - VARNISH_SIZE=2G
    tmpfs:
      - /var/lib/varnish/varnishd:exec
    volumes:
      - ./default.vcl:/etc/varnish/default.vcl:ro
    networks:
      - web
    depends_on:
      - client
      - server
    deploy:
      mode: replicated
      replicas: 1
      restart_policy:
        condition: on-failure
      placement:
        constraints:
          - node.role == manager

    ...

    networks:
      web:
        external: true
      db-net:
        external: true

这是 default.vcl 文件

vcl 4.1;

# import vmod_dynamic for better backend name resolution
import dynamic;

# we won't use any static backend, but Varnish still need a default one
backend default {
    .host = "host.docker.internal";
    .port = "8080";
    .connect_timeout = 600s;
    .first_byte_timeout = 600s;
    .between_bytes_timeout = 600s;
    .probe = {
        .url = "/";
        .interval = 5s;
        .timeout = 50s;
        .window = 5;
        .threshold = 3;
    }
}

backend api {
    .host = "host.docker.internal";
    .port = "3000";
}

sub vcl_backend_response {
  set beresp.ttl = 12h;
}

sub vcl_recv {
    if (req.url ~ "^/api/") {
        set req.backend_hint = api;
    }
    else {
        set req.backend_hint = default;
    }
}

这是 Varnish 服务器上请求的日志:

*   << Request  >> 46        
-   Begin          req 45 rxreq
-   Timestamp      Start: 1651165979.997021 0.000000 0.000000
-   Timestamp      Req: 1651165979.997021 0.000000 0.000000
-   VCL_use        boot
-   ReqStart       172.20.0.1 59730 http
-   ReqMethod      GET
-   ReqURL         /
-   ReqProtocol    HTTP/1.1
-   ReqHeader      Host: 192.168.0.123
-   ReqHeader      Connection: keep-alive
-   ReqHeader      Cache-Control: max-age=0
-   ReqHeader      DNT: 1
-   ReqHeader      Upgrade-Insecure-Requests: 1
-   ReqHeader      User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36
-   ReqHeader      Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
-   ReqHeader      Accept-Encoding: gzip, deflate
-   ReqHeader      Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7
-   ReqHeader      sec-gpc: 1
-   ReqHeader      X-Forwarded-For: 172.20.0.1
-   VCL_call       RECV
-   VCL_return     pass
-   VCL_call       HASH
-   VCL_return     lookup
-   VCL_call       PASS
-   VCL_return     fetch
-   Link           bereq 47 pass
-   Timestamp      Fetch: 1651165979.997170 0.000149 0.000149
-   RespProtocol   HTTP/1.1
-   RespStatus     503
-   RespReason     Backend fetch failed
-   RespHeader     Date: Thu, 28 Apr 2022 17:12:59 GMT
-   RespHeader     Server: Varnish
-   RespHeader     Content-Type: text/html; charset=utf-8
-   RespHeader     Retry-After: 5
-   RespHeader     X-Varnish: 46
-   RespHeader     Age: 0
-   RespHeader     Via: 1.1 varnish (Varnish/7.1)
-   VCL_call       DELIVER
-   VCL_return     deliver
-   Timestamp      Process: 1651165979.997182 0.000161 0.000011
-   Filters        
-   RespHeader     Content-Length: 279
-   RespHeader     Connection: keep-alive
-   Timestamp      Resp: 1651165979.997244 0.000222 0.000061
-   ReqAcct        528 0 528 247 279 526
-   End            
**  << BeReq    >> 47        
--  Begin          bereq 46 pass
--  VCL_use        boot
--  Timestamp      Start: 1651165979.997101 0.000000 0.000000
--  BereqMethod    GET
--  BereqURL       /
--  BereqProtocol  HTTP/1.1
--  BereqHeader    Host: 192.168.0.123
--  BereqHeader    Cache-Control: max-age=0
--  BereqHeader    DNT: 1
--  BereqHeader    Upgrade-Insecure-Requests: 1
--  BereqHeader    User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36
--  BereqHeader    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
--  BereqHeader    Accept-Encoding: gzip, deflate
--  BereqHeader    Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7
--  BereqHeader    sec-gpc: 1
--  BereqHeader    X-Forwarded-For: 172.20.0.1
--  BereqHeader    X-Varnish: 47
--  VCL_call       BACKEND_FETCH
--  VCL_return     fetch
--  Timestamp      Fetch: 1651165979.997116 0.000015 0.000015
--  FetchError     backend default: unhealthy
--  Timestamp      Beresp: 1651165979.997121 0.000020 0.000005
--  Timestamp      Error: 1651165979.997123 0.000022 0.000002
--  BerespProtocol HTTP/1.1
--  BerespStatus   503
--  BerespReason   Backend fetch failed
--  BerespHeader   Date: Thu, 28 Apr 2022 17:12:59 GMT
--  BerespHeader   Server: Varnish
--  VCL_call       BACKEND_ERROR
--  BerespHeader   Content-Type: text/html; charset=utf-8
--  BerespHeader   Retry-After: 5
--  VCL_return     deliver
--  Storage        malloc Transient
--  Length         279
--  BereqAcct      0 0 0 0 0 0
--  End            

*   << Request  >> 48        
-   Begin          req 45 rxreq
-   Timestamp      Start: 1651165980.087213 0.000000 0.000000
-   Timestamp      Req: 1651165980.087213 0.000000 0.000000
-   VCL_use        boot
-   ReqStart       172.20.0.1 59730 http
-   ReqMethod      GET
-   ReqURL         /favicon.ico
-   ReqProtocol    HTTP/1.1
-   ReqHeader      Host: 192.168.0.123
-   ReqHeader      Connection: keep-alive
-   ReqHeader      User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36
-   ReqHeader      DNT: 1
-   ReqHeader      Accept: image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8
-   ReqHeader      Referer: http://192.168.0.123/
-   ReqHeader      Accept-Encoding: gzip, deflate
-   ReqHeader      Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7
-   ReqHeader      sec-gpc: 1
-   ReqHeader      X-Forwarded-For: 172.20.0.1
-   VCL_call       RECV
-   VCL_return     pass
-   VCL_call       HASH
-   VCL_return     lookup
-   VCL_call       PASS
-   VCL_return     fetch
-   Link           bereq 49 pass
-   Timestamp      Fetch: 1651165980.087367 0.000153 0.000153
-   RespProtocol   HTTP/1.1
-   RespStatus     503
-   RespReason     Backend fetch failed
-   RespHeader     Date: Thu, 28 Apr 2022 17:13:00 GMT
-   RespHeader     Server: Varnish
-   RespHeader     Content-Type: text/html; charset=utf-8
-   RespHeader     Retry-After: 5
-   RespHeader     X-Varnish: 48
-   RespHeader     Age: 0
-   RespHeader     Via: 1.1 varnish (Varnish/7.1)
-   VCL_call       DELIVER
-   VCL_return     deliver
-   Timestamp      Process: 1651165980.087382 0.000168 0.000014
-   Filters        
-   RespHeader     Content-Length: 279
-   RespHeader     Connection: keep-alive
-   Timestamp      Resp: 1651165980.087445 0.000231 0.000063
-   ReqAcct        443 0 443 247 279 526
-   End            
**  << BeReq    >> 49        
--  Begin          bereq 48 pass
--  VCL_use        boot
--  Timestamp      Start: 1651165980.087307 0.000000 0.000000
--  BereqMethod    GET
--  BereqURL       /favicon.ico
--  BereqProtocol  HTTP/1.1
--  BereqHeader    Host: 192.168.0.123
--  BereqHeader    User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36
--  BereqHeader    DNT: 1
--  BereqHeader    Accept: image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8
--  BereqHeader    Referer: http://192.168.0.123/
--  BereqHeader    Accept-Encoding: gzip, deflate
--  BereqHeader    Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7
--  BereqHeader    sec-gpc: 1
--  BereqHeader    X-Forwarded-For: 172.20.0.1
--  BereqHeader    X-Varnish: 49
--  VCL_call       BACKEND_FETCH
--  VCL_return     fetch
--  Timestamp      Fetch: 1651165980.087327 0.000019 0.000019
--  FetchError     backend default: unhealthy
--  Timestamp      Beresp: 1651165980.087332 0.000025 0.000005
--  Timestamp      Error: 1651165980.087334 0.000027 0.000002
--  BerespProtocol HTTP/1.1
--  BerespStatus   503
--  BerespReason   Backend fetch failed
--  BerespHeader   Date: Thu, 28 Apr 2022 17:13:00 GMT
--  BerespHeader   Server: Varnish
--  VCL_call       BACKEND_ERROR
--  BerespHeader   Content-Type: text/html; charset=utf-8
--  BerespHeader   Retry-After: 5
--  VCL_return     deliver
--  Storage        malloc Transient
--  Length         279
--  BereqAcct      0 0 0 0 0 0
--  End            
Open error 110 (Connection timed out)

尝试通过 bash shell 访问 Varnish 容器,如下图所示:

docker exec -ti varnish bash

Assuming that name of your Varnish container is varnish

运行以下命令安装一些调试依赖:

apt-get update
apt-get install -y telnet curl dnsutils

然后运行以下调试命令:

dig host.docker.internal
telnet host.docker.internal 8080
curl http://host.docker.internal:8080
  1. 第一个命令将检查 host.docker.internal 是否解析为 IP 地址
  2. 第二个命令将检查主机 host.docker.internal 是否侦听端口 8080
  3. 上的传入连接
  4. 第 3 个命令将在 http://host.docker.internal:8080
  5. 上尝试 HTTP 请求

请记住,Varnish 只能通过纯 HTTP 连接后端。不支持 HTTPS 后端。

根据这些调试命令的结果,你应该知道发生了什么,服务是否可达,后端是否及时响应。

解决方案比较奇怪。我只需要删除 docker 中的 ports: 部分,并使用 default.vcl 文件中的服务名称。

撰写:

client:
    image: ...
    ports: <-- remove
      - target: 80
        published: 8080
        mode: host

直接访问default.vcl

中的内部端口
backend default {
    .host = "client";
    .port = "80";
}

这可能与 docker 群有关。不知道为什么。