添加清漆禁令但返回旧 object
Varnish ban is added but old object is returned
我在切片服务器前使用清漆来缓存 mapbox 切片。为了移除旧的图块，我打算使用 bans 来有效地移除大量缓存的图块。我的问题是清漆仍然使用缓存的 objects（至少响应中的 age 表明了这一点）并且不联系后端。
我首先请求 http://varnish/5/3/4.pbf，然后使用 curl -X BAN -H 'X-Purge-Regex: 5/3/4.pbf' varnish 或 varnishadm 添加禁令，然后 ban obj.http.url ~ 5/3/4.pbf 然后请求 http: //varnish/5/3/4.pbf 再次.
一开始我的黑名单是空的：
Present bans:
1610117471.434488 1 C
添加封禁成功 curl -X BAN -H 'X-Purge-Regex: 5/3/4.pbf' varnish
<!DOCTYPE html>
<html>
<head>
<title>200 Ban added</title>
</head>
<body>
<h1>Error 200 Ban added</h1>
<p>Ban added</p>
<h3>Guru Meditation:</h3>
<p>XID: 8</p>
<hr>
<p>Varnish cache server</p>
</body>
</html>
并出现在黑名单中
Present bans:
1610117369.028870 0 - obj.http.url ~ 5/3/4.pbf
1610117307.220739 1 C
再次请求http:///varnish/5/3/4.pbf后，ban列表提示使用了ban
Present bans:
1610117471.434488 1 - obj.http.url ~ 5/3/4.pbf
但是响应的年龄不是 0，因为它仍然是第一个请求的 object。
不久之后，禁令被解除：
Present bans:
1610117471.434488 1 C
我的 vcl_recv 看起来像这样，但错误可能出在其他地方，因为它也不适用于 varnishadm:
sub vcl_recv {
unset req.http.cookie;
# Allowing PURGE from localhost
if (req.method == "BAN"||req.method == "PURGE") {
if (!client.ip ~ purge) {
return(synth(405,"Not allowed."));
}
if (req.method == "BAN") {
ban("obj.http.url ~ " + req.http.X-Purge-Regex);
# Throw a synthetic page so the
# request won't go to the backend.
return(synth(200, "Ban added"));
}
if (req.method == "PURGE") {
return (purge);
}
}
}
我也尝试使用 中的 vcl_purge 但这似乎对禁令没有帮助 (?)。
我正在使用 X-Purge-Regex header 不用担心像 那样必须转义特殊字符，但像 obj.http.url ~ 0 这样的禁令不起作用.
我使用 varnish 6.5 和 vcl 4.0。
禁止请求
* << Request >> 54
- Begin req 53 rxreq
- Timestamp Start: 1610121483.345437 0.000000 0.000000
- Timestamp Req: 1610121483.345437 0.000000 0.000000
- VCL_use boot
- ReqStart 192.168.48.2 50882 http
- ReqMethod BAN
- ReqURL /
- ReqProtocol HTTP/1.1
- ReqHeader Host: varnish-volatile
- ReqHeader User-Agent: curl/7.64.0
- ReqHeader Accept: */*
- ReqHeader X-Purge-Regex: 0
- ReqHeader X-Forwarded-For: 192.168.48.2
- VCL_call RECV
- VCL_acl MATCH purge "importer"
- VCL_return synth
- VCL_call HASH
- VCL_return lookup
- RespProtocol HTTP/1.1
- RespStatus 200
- RespReason Ban added
- RespHeader Date: Fri, 08 Jan 2021 15:58:03 GMT
- RespHeader Server: Varnish
- RespHeader X-Varnish: 54
- VCL_call SYNTH
- RespHeader Content-Type: text/html; charset=utf-8
- RespHeader Retry-After: 5
- VCL_return deliver
- Timestamp Process: 1610121483.347281 0.001844 0.001844
- RespHeader Content-Length: 246
- Storage malloc Transient
- Filters
- RespHeader Accept-Ranges: bytes
- RespHeader Connection: keep-alive
- Timestamp Resp: 1610121483.347557 0.002120 0.000276
- ReqAcct 98 0 98 218 246 464
- End
加禁后GET
* << Request >> 32806
- Begin req 32805 rxreq
- Timestamp Start: 1610121552.733872 0.000000 0.000000
- Timestamp Req: 1610121552.733872 0.000000 0.000000
- VCL_use boot
- ReqStart 192.168.48.1 55176 http
- ReqMethod GET
- ReqURL /public.snow_db/0/0/0.pbf
- ReqProtocol HTTP/1.1
- ReqHeader Host: localhost:8090
- ReqHeader User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:84.0) Gecko/20100101 Firefox/84.0
- ReqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
- ReqHeader Accept-Language: en-US,en;q=0.5
- ReqHeader Accept-Encoding: gzip, deflate
- ReqHeader DNT: 1
- ReqHeader Connection: keep-alive
- ReqHeader Upgrade-Insecure-Requests: 1
- ReqHeader Pragma: no-cache
- ReqHeader Cache-Control: no-cache
- ReqHeader X-Forwarded-For: 192.168.48.1
- VCL_call RECV
- ReqUnset Host: localhost:8090
- ReqHeader host: localhost:8090
- VCL_return hash
- ReqUnset Accept-Encoding: gzip, deflate
- ReqHeader Accept-Encoding: gzip
- VCL_call HASH
- VCL_return lookup
- Hit 28 601789.331504 10.000000 0.000000
- VCL_call HIT
- VCL_return deliver
- RespProtocol HTTP/1.1
- RespStatus 200
- RespReason OK
- RespHeader content-encoding: gzip
- RespHeader content-type: application/x-protobuf
- RespHeader date: Fri, 08 Jan 2021 15:09:02 GMT
- RespHeader Vary: Accept-Encoding
- RespHeader X-Varnish: 32806 28
- RespHeader Age: 3010
- RespHeader Via: 1.1 varnish (Varnish/6.5)
- VCL_call DELIVER
- VCL_return deliver
- Timestamp Process: 1610121552.734070 0.000197 0.000197
- Filters
- RespHeader Accept-Ranges: bytes
- RespHeader Content-Length: 295
- RespHeader Connection: keep-alive
- Timestamp Resp: 1610121552.734217 0.000345 0.000147
- ReqAcct 414 0 414 272 295 567
- End
重现问题
要重现错误：
git clone https://github.com/Baschdl/varnish-ban-setup.git && cd varnish-ban-setupdocker-compose up- 打开http://localhost:8092/5/3/1.pbf
docker-compose exec varnish varnishadm ban obj.http.url ~ pbf- 再次打开http://localhost:8092/5/3/1.pbf，你会得到旧的object
您发出的 obj.http.url ~ 5/3/4.pbf 禁令与 url 响应相匹配 header。
Remember: the URL is a request header, not a response header. No reason to panic, what you're doing makes perfect sense, and is related to the scope of the so-called ban lurker.
禁止潜伏者
ban lurker 是一个异步处理 ban 列表上的 ban 并将 objects 匹配到 bans 以删除 [=44= 的模式的线程]s 来自缓存。
ban lurker 不在请求范围内运行，但只知道 object 范围。
为了成功匹配请求信息，可以添加请求上下文作为响应header。这就是您通过 obj.http.url
所做的
那么为什么禁令不起作用？
你的禁令不起作用的原因是你没有在你的 VCL 文件中设置 obj.http.url。结果，ban lurker 无法匹配任何 objects 到它。
如何解决问题
解决方法很简单：在后端响应上下文中设置缺失的headers，如下图：
sub vcl_backend_response {
set beresp.http.url = bereq.url;
set beresp.http.host = bereq.http.host;
}
当后端响应时，就在 object 存储到缓存之前，我们可以设置缺少的 headers。
之后，ban lurker 将能够匹配右边 objects 的 ban 表达式，并将它们从缓存中删除。
Don't forget the objects aren't immediately matched: they are only removed when they reach the ban_lurker_age, which is set to 1 minute by default.
我在切片服务器前使用清漆来缓存 mapbox 切片。为了移除旧的图块，我打算使用 bans 来有效地移除大量缓存的图块。我的问题是清漆仍然使用缓存的 objects（至少响应中的 age 表明了这一点）并且不联系后端。
我首先请求 http://varnish/5/3/4.pbf，然后使用 curl -X BAN -H 'X-Purge-Regex: 5/3/4.pbf' varnish 或 varnishadm 添加禁令，然后 ban obj.http.url ~ 5/3/4.pbf 然后请求 http: //varnish/5/3/4.pbf 再次.
一开始我的黑名单是空的：
Present bans:
1610117471.434488 1 C
添加封禁成功 curl -X BAN -H 'X-Purge-Regex: 5/3/4.pbf' varnish
<!DOCTYPE html>
<html>
<head>
<title>200 Ban added</title>
</head>
<body>
<h1>Error 200 Ban added</h1>
<p>Ban added</p>
<h3>Guru Meditation:</h3>
<p>XID: 8</p>
<hr>
<p>Varnish cache server</p>
</body>
</html>
并出现在黑名单中
Present bans:
1610117369.028870 0 - obj.http.url ~ 5/3/4.pbf
1610117307.220739 1 C
再次请求http:///varnish/5/3/4.pbf后，ban列表提示使用了ban
Present bans:
1610117471.434488 1 - obj.http.url ~ 5/3/4.pbf
但是响应的年龄不是 0，因为它仍然是第一个请求的 object。
不久之后，禁令被解除：
Present bans:
1610117471.434488 1 C
我的 vcl_recv 看起来像这样，但错误可能出在其他地方，因为它也不适用于 varnishadm:
sub vcl_recv {
unset req.http.cookie;
# Allowing PURGE from localhost
if (req.method == "BAN"||req.method == "PURGE") {
if (!client.ip ~ purge) {
return(synth(405,"Not allowed."));
}
if (req.method == "BAN") {
ban("obj.http.url ~ " + req.http.X-Purge-Regex);
# Throw a synthetic page so the
# request won't go to the backend.
return(synth(200, "Ban added"));
}
if (req.method == "PURGE") {
return (purge);
}
}
}
我也尝试使用 中的 vcl_purge 但这似乎对禁令没有帮助 (?)。
我正在使用 X-Purge-Regex header 不用担心像 那样必须转义特殊字符，但像 obj.http.url ~ 0 这样的禁令不起作用.
我使用 varnish 6.5 和 vcl 4.0。
禁止请求
* << Request >> 54
- Begin req 53 rxreq
- Timestamp Start: 1610121483.345437 0.000000 0.000000
- Timestamp Req: 1610121483.345437 0.000000 0.000000
- VCL_use boot
- ReqStart 192.168.48.2 50882 http
- ReqMethod BAN
- ReqURL /
- ReqProtocol HTTP/1.1
- ReqHeader Host: varnish-volatile
- ReqHeader User-Agent: curl/7.64.0
- ReqHeader Accept: */*
- ReqHeader X-Purge-Regex: 0
- ReqHeader X-Forwarded-For: 192.168.48.2
- VCL_call RECV
- VCL_acl MATCH purge "importer"
- VCL_return synth
- VCL_call HASH
- VCL_return lookup
- RespProtocol HTTP/1.1
- RespStatus 200
- RespReason Ban added
- RespHeader Date: Fri, 08 Jan 2021 15:58:03 GMT
- RespHeader Server: Varnish
- RespHeader X-Varnish: 54
- VCL_call SYNTH
- RespHeader Content-Type: text/html; charset=utf-8
- RespHeader Retry-After: 5
- VCL_return deliver
- Timestamp Process: 1610121483.347281 0.001844 0.001844
- RespHeader Content-Length: 246
- Storage malloc Transient
- Filters
- RespHeader Accept-Ranges: bytes
- RespHeader Connection: keep-alive
- Timestamp Resp: 1610121483.347557 0.002120 0.000276
- ReqAcct 98 0 98 218 246 464
- End
加禁后GET
* << Request >> 32806
- Begin req 32805 rxreq
- Timestamp Start: 1610121552.733872 0.000000 0.000000
- Timestamp Req: 1610121552.733872 0.000000 0.000000
- VCL_use boot
- ReqStart 192.168.48.1 55176 http
- ReqMethod GET
- ReqURL /public.snow_db/0/0/0.pbf
- ReqProtocol HTTP/1.1
- ReqHeader Host: localhost:8090
- ReqHeader User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:84.0) Gecko/20100101 Firefox/84.0
- ReqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
- ReqHeader Accept-Language: en-US,en;q=0.5
- ReqHeader Accept-Encoding: gzip, deflate
- ReqHeader DNT: 1
- ReqHeader Connection: keep-alive
- ReqHeader Upgrade-Insecure-Requests: 1
- ReqHeader Pragma: no-cache
- ReqHeader Cache-Control: no-cache
- ReqHeader X-Forwarded-For: 192.168.48.1
- VCL_call RECV
- ReqUnset Host: localhost:8090
- ReqHeader host: localhost:8090
- VCL_return hash
- ReqUnset Accept-Encoding: gzip, deflate
- ReqHeader Accept-Encoding: gzip
- VCL_call HASH
- VCL_return lookup
- Hit 28 601789.331504 10.000000 0.000000
- VCL_call HIT
- VCL_return deliver
- RespProtocol HTTP/1.1
- RespStatus 200
- RespReason OK
- RespHeader content-encoding: gzip
- RespHeader content-type: application/x-protobuf
- RespHeader date: Fri, 08 Jan 2021 15:09:02 GMT
- RespHeader Vary: Accept-Encoding
- RespHeader X-Varnish: 32806 28
- RespHeader Age: 3010
- RespHeader Via: 1.1 varnish (Varnish/6.5)
- VCL_call DELIVER
- VCL_return deliver
- Timestamp Process: 1610121552.734070 0.000197 0.000197
- Filters
- RespHeader Accept-Ranges: bytes
- RespHeader Content-Length: 295
- RespHeader Connection: keep-alive
- Timestamp Resp: 1610121552.734217 0.000345 0.000147
- ReqAcct 414 0 414 272 295 567
- End
重现问题
要重现错误：
git clone https://github.com/Baschdl/varnish-ban-setup.git && cd varnish-ban-setupdocker-compose up- 打开http://localhost:8092/5/3/1.pbf
docker-compose exec varnish varnishadm ban obj.http.url ~ pbf- 再次打开http://localhost:8092/5/3/1.pbf，你会得到旧的object
您发出的 obj.http.url ~ 5/3/4.pbf 禁令与 url 响应相匹配 header。
Remember: the URL is a request header, not a response header. No reason to panic, what you're doing makes perfect sense, and is related to the scope of the so-called ban lurker.
禁止潜伏者
ban lurker 是一个异步处理 ban 列表上的 ban 并将 objects 匹配到 bans 以删除 [=44= 的模式的线程]s 来自缓存。
ban lurker 不在请求范围内运行，但只知道 object 范围。
为了成功匹配请求信息，可以添加请求上下文作为响应header。这就是您通过 obj.http.url
那么为什么禁令不起作用？
你的禁令不起作用的原因是你没有在你的 VCL 文件中设置 obj.http.url。结果，ban lurker 无法匹配任何 objects 到它。
如何解决问题
解决方法很简单：在后端响应上下文中设置缺失的headers，如下图：
sub vcl_backend_response {
set beresp.http.url = bereq.url;
set beresp.http.host = bereq.http.host;
}
当后端响应时，就在 object 存储到缓存之前，我们可以设置缺少的 headers。
之后，ban lurker 将能够匹配右边 objects 的 ban 表达式，并将它们从缓存中删除。
Don't forget the objects aren't immediately matched: they are only removed when they reach the
ban_lurker_age, which is set to 1 minute by default.