Cloudflare 不会通过 CURL 请求缓存页面
Cloudflare won't cache page via CURL request
One.php 向 two.php 发出两个单独的 CURL 请求。我希望 Cloudflare 缓存第一个响应，并在第二个响应中提供该缓存。但出于某种原因，Cloudflare 没有缓存任何一个响应。如果我随后使用浏览器访问 two.php，它仍然没有被缓存。如果我随后在浏览器中刷新 two.php，我最终会收到缓存命中。如何通过 CURL 请求在 Cloudflare 上触发缓存？
基本测试：
./one.php
header("Cache-Control: no-store");
$url = "http://www.example.com/two.php";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url );
curl_exec($ch); // prints current timestamp, see two.php
curl_close($ch);
sleep(1);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url );
curl_exec($ch); // prints current timestamp, see two.php
curl_close($ch);
./two.php
header( 'Cache-Control: public, max-age=10' );
print( time()."<br>\r\n" );
扩展测试：
我尝试设置 User-Agent header 并使用 Cloudflare 的 cfduid cookie。我尝试在两个不同服务器上的两个不同网站上和两个网站之间发出 CURL 请求。我尝试了 HTTP 和 HTTPS。我尝试了 运行 one.php 来自 direct.example.com （绕过 cloudflare）。
./one.php
header("Cache-Control: no-store");
$url = "http://www.example.com/target.php";
$request_headers = array(
'Connection: keep-alive',
'Upgrade-Insecure-Requests: 1',
'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36',
'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
'Accept-Encoding: compressed',
'Accept-Language: en-US,en;q=0.9',
);
// Fetch CFDUID Cookie
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url );
curl_setopt($ch, CURLOPT_HTTPHEADER, $request_headers);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_NOBODY, 1);
$response_headers = curl_exec($ch);
curl_close($ch);
// Extract Cookies
preg_match_all( '/^Set-Cookie:\s*(.+?);?\s*$/mi', $response_headers, $cookies);
$cookies = implode( ';', $cookies[1] );
// Setup First request
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url );
curl_setopt($ch, CURLOPT_HTTPHEADER, $request_headers);
curl_setopt($ch, CURLOPT_COOKIE, $cookies );
// Print First Response
echo( "First Response:<br>\r\n<pre>" );
curl_exec($ch);
curl_close($ch);
echo( "</pre>" );
// Pause
sleep(1);
// Setup Second request
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url );
curl_setopt($ch, CURLOPT_HTTPHEADER, $request_headers);
curl_setopt($ch, CURLOPT_COOKIE, $cookies );
// Record CURL Request Info
curl_setopt($ch, CURLOPT_VERBOSE, true);
$verbose = fopen('php://temp', 'w+');
curl_setopt($ch, CURLOPT_STDERR, $verbose);
// Print Second Response
echo( "Second Response:<br>\r\n<pre>" );
curl_exec($ch);
curl_close($ch);
echo( "</pre>" );
// Show CURL Request Info
rewind($verbose);
$verboseLog = stream_get_contents($verbose);
echo "CURL Verbose Info:<br>\r\n<pre>", htmlspecialchars($verboseLog), "</pre>\r\n";
./two.php
header("Cache-Control: public, max-age=10");
header( "set-cookie: test1=true; test2=true" );
echo( time() . "\r\n" );
var_dump( getallheaders() );
扩展测试输出（截尾）：
First Response:
1536190748
array(25) {
["Content-Type"]=>
string(0) ""
["Content-Length"]=>
string(1) "0"
["X-Original-Url"]=>
string(20) "/target.php"
["Was-Default-Hostname"]=>
string(33) "*****.azurewebsites.net"
["X-Site-Deployment-Id"]=>
string(15) "*****"
["Disguised-Host"]=>
string(21) "www.*****.com"
["X-Arr-Log-Id"]=>
string(36) "c157*****e003"
["Is-Service-Tunneled"]=>
string(1) "0"
["Client-Ip"]=>
string(21) "*.*.*.175:18968"
["X-Waws-Unencoded-Url"]=>
string(20) "/target.php"
["Cf-Connecting-Ip"]=>
string(12) "*.*.*.33"
["Upgrade-Insecure-Requests"]=>
string(1) "1"
["Cf-Visitor"]=>
string(17) "{"scheme":"http"}"
["X-Forwarded-Proto"]=>
string(4) "http"
["Cf-Ray"]=>
string(20) "455c*****-ATL"
["X-Forwarded-For"]=>
string(35) "*.*.*.33, *.*.*.175:18968"
["Cf-Ipcountry"]=>
string(2) "US"
["User-Agent"]=>
string(115) "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/*.*.*.106 Safari/537.36"
["Max-Forwards"]=>
string(2) "10"
["Host"]=>
string(21) "www.*****.com"
["Cookie"]=>
string(279) "__cfduid=d2be*****0748; expires=Thu, 05-Sep-19 23:39:08 GMT; path=/; domain=.*****.com; HttpOnly;test1=true; test2=true;ARRAffinity=91cd*****4f3f;Path=/;HttpOnly;Domain=www.*****.com"
["Accept-Language"]=>
string(14) "en-US,en;q=0.9"
["Accept-Encoding"]=>
string(4) "gzip"
["Accept"]=>
string(85) "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8"
["Connection"]=>
string(10) "Keep-Alive"
}
Second Response:
1536190749
array(25) {
["Content-Type"]=>
string(0) ""
["Content-Length"]=>
string(1) "0"
["X-Original-Url"]=>
string(20) "/target.php"
["Was-Default-Hostname"]=>
string(33) "*****.azurewebsites.net"
["X-Site-Deployment-Id"]=>
string(15) "*****"
["Disguised-Host"]=>
string(21) "www.*****.com"
["X-Arr-Log-Id"]=>
string(36) "588f*****3a02"
["Is-Service-Tunneled"]=>
string(1) "0"
["Client-Ip"]=>
string(21) "*.*.*.175:19820"
["X-Waws-Unencoded-Url"]=>
string(20) "/target.php"
["Cf-Connecting-Ip"]=>
string(12) "*.*.*.33"
["Upgrade-Insecure-Requests"]=>
string(1) "1"
["Cf-Visitor"]=>
string(17) "{"scheme":"http"}"
["X-Forwarded-Proto"]=>
string(4) "http"
["Cf-Ray"]=>
string(20) "455c8*****-ATL"
["X-Forwarded-For"]=>
string(35) "*.*.*.33, *.*.*.175:19820"
["Cf-Ipcountry"]=>
string(2) "US"
["User-Agent"]=>
string(115) "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/*.*.*.106 Safari/537.36"
["Max-Forwards"]=>
string(2) "10"
["Host"]=>
string(21) "www.*****.com"
["Cookie"]=>
string(279) "__cfduid=d2be*****0748; expires=Thu, 05-Sep-19 23:39:08 GMT; path=/; domain=.*****.com; HttpOnly;test1=true; test2=true;ARRAffinity=91cd*****4f3f;Path=/;HttpOnly;Domain=www.*****.com"
["Accept-Language"]=>
string(14) "en-US,en;q=0.9"
["Accept-Encoding"]=>
string(4) "gzip"
["Accept"]=>
string(85) "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8"
["Connection"]=>
string(10) "Keep-Alive"
}
CURL Verbose Info:
* Hostname www.*****.com was found in DNS cache
* Trying *.*.*.46...
* TCP_NODELAY set
* Connected to www.*****.com (*.*.*.46) port 80 (#0)
> GET /target.php HTTP/1.1
Host: www.*****.com
Cookie: __cfduid=d2be*****0748; expires=Thu, 05-Sep-19 23:39:08 GMT; path=/; domain=.*****.com; HttpOnly;test1=true; test2=true;ARRAffinity=91cd*****4f3f;Path=/;HttpOnly;Domain=www.*****.com
Connection: keep-alive
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/*.*.*.106 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Accept-Encoding: compressed
Accept-Language: en-US,en;q=0.9
< HTTP/1.1 200 OK
< Date: Wed, 05 Sep 2018 23:39:10 GMT
< Content-Type: text/html; charset=UTF-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Cache-Control: public, max-age=10
< Vary: Accept-Encoding
< Set-Cookie: test1=true; test2=true
< X-Powered-By: PHP/5.6.37
< X-Powered-By: ASP.NET
< CF-Cache-Status: MISS
< Server: cloudflare
< CF-RAY: 455c*****-ATL
<
* Connection #0 to host www.*****.com left intact
对 CloudFlare 的请求是地理分布的。很可能您的服务器访问的 CloudFlare 边缘位置与您访问的不同（目前有 152 个），因此您无法从其调用中受益。
看看这个 github 存储库 cloudflare-bypass。他的代码有一个 cache on/off 参数。我认为这可能对你有帮助。请参阅 blow 他的代码片段。
<?php
require __DIR__ . '/../vendor/autoload.php';
use CloudflareBypass\RequestMethod\CFCurl;
$curl_cf_wrapper = new CFCurl(array(
'cache' => true, // Caching now enabled by default; stores clearance tokens in Cache folder
'max_retries' => 5 // Max attempts to try and get CF clearance
));
// Get Example: 1
$ch = curl_init("http://thebot.net/");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36');
echo $curl_cf_wrapper->exec($ch); // Done! NOTE: HEAD requests not supported!
curl_close($ch);
好吧，现在是 2020 年了，看来我问题中的 "bare bones" 示例现在可以正常工作了。我猜 Cloudflare 最终改变了一些东西。请注意：我的 "extended" 示例应该不起作用，因为 Cloudflare 不会 缓存设置自定义 cookie 的页面！ （在这种情况下 test1=true。）
One.php 向 two.php 发出两个单独的 CURL 请求。我希望 Cloudflare 缓存第一个响应，并在第二个响应中提供该缓存。但出于某种原因，Cloudflare 没有缓存任何一个响应。如果我随后使用浏览器访问 two.php，它仍然没有被缓存。如果我随后在浏览器中刷新 two.php，我最终会收到缓存命中。如何通过 CURL 请求在 Cloudflare 上触发缓存？
基本测试：
./one.php
header("Cache-Control: no-store");
$url = "http://www.example.com/two.php";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url );
curl_exec($ch); // prints current timestamp, see two.php
curl_close($ch);
sleep(1);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url );
curl_exec($ch); // prints current timestamp, see two.php
curl_close($ch);
./two.php
header( 'Cache-Control: public, max-age=10' );
print( time()."<br>\r\n" );
扩展测试：
我尝试设置 User-Agent header 并使用 Cloudflare 的 cfduid cookie。我尝试在两个不同服务器上的两个不同网站上和两个网站之间发出 CURL 请求。我尝试了 HTTP 和 HTTPS。我尝试了 运行 one.php 来自 direct.example.com （绕过 cloudflare）。
./one.php
header("Cache-Control: no-store");
$url = "http://www.example.com/target.php";
$request_headers = array(
'Connection: keep-alive',
'Upgrade-Insecure-Requests: 1',
'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36',
'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
'Accept-Encoding: compressed',
'Accept-Language: en-US,en;q=0.9',
);
// Fetch CFDUID Cookie
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url );
curl_setopt($ch, CURLOPT_HTTPHEADER, $request_headers);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_NOBODY, 1);
$response_headers = curl_exec($ch);
curl_close($ch);
// Extract Cookies
preg_match_all( '/^Set-Cookie:\s*(.+?);?\s*$/mi', $response_headers, $cookies);
$cookies = implode( ';', $cookies[1] );
// Setup First request
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url );
curl_setopt($ch, CURLOPT_HTTPHEADER, $request_headers);
curl_setopt($ch, CURLOPT_COOKIE, $cookies );
// Print First Response
echo( "First Response:<br>\r\n<pre>" );
curl_exec($ch);
curl_close($ch);
echo( "</pre>" );
// Pause
sleep(1);
// Setup Second request
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url );
curl_setopt($ch, CURLOPT_HTTPHEADER, $request_headers);
curl_setopt($ch, CURLOPT_COOKIE, $cookies );
// Record CURL Request Info
curl_setopt($ch, CURLOPT_VERBOSE, true);
$verbose = fopen('php://temp', 'w+');
curl_setopt($ch, CURLOPT_STDERR, $verbose);
// Print Second Response
echo( "Second Response:<br>\r\n<pre>" );
curl_exec($ch);
curl_close($ch);
echo( "</pre>" );
// Show CURL Request Info
rewind($verbose);
$verboseLog = stream_get_contents($verbose);
echo "CURL Verbose Info:<br>\r\n<pre>", htmlspecialchars($verboseLog), "</pre>\r\n";
./two.php
header("Cache-Control: public, max-age=10");
header( "set-cookie: test1=true; test2=true" );
echo( time() . "\r\n" );
var_dump( getallheaders() );
扩展测试输出（截尾）：
First Response:
1536190748
array(25) {
["Content-Type"]=>
string(0) ""
["Content-Length"]=>
string(1) "0"
["X-Original-Url"]=>
string(20) "/target.php"
["Was-Default-Hostname"]=>
string(33) "*****.azurewebsites.net"
["X-Site-Deployment-Id"]=>
string(15) "*****"
["Disguised-Host"]=>
string(21) "www.*****.com"
["X-Arr-Log-Id"]=>
string(36) "c157*****e003"
["Is-Service-Tunneled"]=>
string(1) "0"
["Client-Ip"]=>
string(21) "*.*.*.175:18968"
["X-Waws-Unencoded-Url"]=>
string(20) "/target.php"
["Cf-Connecting-Ip"]=>
string(12) "*.*.*.33"
["Upgrade-Insecure-Requests"]=>
string(1) "1"
["Cf-Visitor"]=>
string(17) "{"scheme":"http"}"
["X-Forwarded-Proto"]=>
string(4) "http"
["Cf-Ray"]=>
string(20) "455c*****-ATL"
["X-Forwarded-For"]=>
string(35) "*.*.*.33, *.*.*.175:18968"
["Cf-Ipcountry"]=>
string(2) "US"
["User-Agent"]=>
string(115) "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/*.*.*.106 Safari/537.36"
["Max-Forwards"]=>
string(2) "10"
["Host"]=>
string(21) "www.*****.com"
["Cookie"]=>
string(279) "__cfduid=d2be*****0748; expires=Thu, 05-Sep-19 23:39:08 GMT; path=/; domain=.*****.com; HttpOnly;test1=true; test2=true;ARRAffinity=91cd*****4f3f;Path=/;HttpOnly;Domain=www.*****.com"
["Accept-Language"]=>
string(14) "en-US,en;q=0.9"
["Accept-Encoding"]=>
string(4) "gzip"
["Accept"]=>
string(85) "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8"
["Connection"]=>
string(10) "Keep-Alive"
}
Second Response:
1536190749
array(25) {
["Content-Type"]=>
string(0) ""
["Content-Length"]=>
string(1) "0"
["X-Original-Url"]=>
string(20) "/target.php"
["Was-Default-Hostname"]=>
string(33) "*****.azurewebsites.net"
["X-Site-Deployment-Id"]=>
string(15) "*****"
["Disguised-Host"]=>
string(21) "www.*****.com"
["X-Arr-Log-Id"]=>
string(36) "588f*****3a02"
["Is-Service-Tunneled"]=>
string(1) "0"
["Client-Ip"]=>
string(21) "*.*.*.175:19820"
["X-Waws-Unencoded-Url"]=>
string(20) "/target.php"
["Cf-Connecting-Ip"]=>
string(12) "*.*.*.33"
["Upgrade-Insecure-Requests"]=>
string(1) "1"
["Cf-Visitor"]=>
string(17) "{"scheme":"http"}"
["X-Forwarded-Proto"]=>
string(4) "http"
["Cf-Ray"]=>
string(20) "455c8*****-ATL"
["X-Forwarded-For"]=>
string(35) "*.*.*.33, *.*.*.175:19820"
["Cf-Ipcountry"]=>
string(2) "US"
["User-Agent"]=>
string(115) "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/*.*.*.106 Safari/537.36"
["Max-Forwards"]=>
string(2) "10"
["Host"]=>
string(21) "www.*****.com"
["Cookie"]=>
string(279) "__cfduid=d2be*****0748; expires=Thu, 05-Sep-19 23:39:08 GMT; path=/; domain=.*****.com; HttpOnly;test1=true; test2=true;ARRAffinity=91cd*****4f3f;Path=/;HttpOnly;Domain=www.*****.com"
["Accept-Language"]=>
string(14) "en-US,en;q=0.9"
["Accept-Encoding"]=>
string(4) "gzip"
["Accept"]=>
string(85) "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8"
["Connection"]=>
string(10) "Keep-Alive"
}
CURL Verbose Info:
* Hostname www.*****.com was found in DNS cache
* Trying *.*.*.46...
* TCP_NODELAY set
* Connected to www.*****.com (*.*.*.46) port 80 (#0)
> GET /target.php HTTP/1.1
Host: www.*****.com
Cookie: __cfduid=d2be*****0748; expires=Thu, 05-Sep-19 23:39:08 GMT; path=/; domain=.*****.com; HttpOnly;test1=true; test2=true;ARRAffinity=91cd*****4f3f;Path=/;HttpOnly;Domain=www.*****.com
Connection: keep-alive
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/*.*.*.106 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Accept-Encoding: compressed
Accept-Language: en-US,en;q=0.9
< HTTP/1.1 200 OK
< Date: Wed, 05 Sep 2018 23:39:10 GMT
< Content-Type: text/html; charset=UTF-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Cache-Control: public, max-age=10
< Vary: Accept-Encoding
< Set-Cookie: test1=true; test2=true
< X-Powered-By: PHP/5.6.37
< X-Powered-By: ASP.NET
< CF-Cache-Status: MISS
< Server: cloudflare
< CF-RAY: 455c*****-ATL
<
* Connection #0 to host www.*****.com left intact
对 CloudFlare 的请求是地理分布的。很可能您的服务器访问的 CloudFlare 边缘位置与您访问的不同（目前有 152 个），因此您无法从其调用中受益。
看看这个 github 存储库 cloudflare-bypass。他的代码有一个 cache on/off 参数。我认为这可能对你有帮助。请参阅 blow 他的代码片段。
<?php
require __DIR__ . '/../vendor/autoload.php';
use CloudflareBypass\RequestMethod\CFCurl;
$curl_cf_wrapper = new CFCurl(array(
'cache' => true, // Caching now enabled by default; stores clearance tokens in Cache folder
'max_retries' => 5 // Max attempts to try and get CF clearance
));
// Get Example: 1
$ch = curl_init("http://thebot.net/");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36');
echo $curl_cf_wrapper->exec($ch); // Done! NOTE: HEAD requests not supported!
curl_close($ch);
好吧，现在是 2020 年了，看来我问题中的 "bare bones" 示例现在可以正常工作了。我猜 Cloudflare 最终改变了一些东西。请注意：我的 "extended" 示例应该不起作用，因为 Cloudflare 不会 缓存设置自定义 cookie 的页面！ （在这种情况下 test1=true。）