CloudFlare 在哪里检测网络和终端请求？！平等对待

Question

我的问题是在 CLI 中打开 CloudFlare 上的网站。
我不是说有挑战的时候我不想解决挑战。

以这个网站为例：https://pegaxy.io
在新安装的任何网络浏览器上首次打开时。它打开没有任何问题。收到代码 200。

但是当我点击 Copy as cURL 并在终端中出现 403 错误。

CURL 代码：

curl 'https://pegaxy.io/' \
  -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; rv:91.0) Gecko/20100101 Firefox/91.0' \
  -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' \
  -H 'Accept-Language: en-US,en;q=0.5' \
  -H 'Connection: keep-alive' \
  -H 'Upgrade-Insecure-Requests: 1' \
  -H 'Sec-Fetch-Dest: document' \
  -H 'Sec-Fetch-Mode: navigate' \
  -H 'Sec-Fetch-Site: none' \
  -H 'Sec-Fetch-User: ?1' \
  -H 'Pragma: no-cache' \
  -H 'Cache-Control: no-cache' \
  --compressed --verbose

日志：

$ curl 'https://pegaxy.io/' \
>   -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; rv:91.0) Gecko/20100101 Firefox/91.0' \
>   -H 'Upgrade-Insecure-Requests: 1' \
>   -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' \
>   -H 'Accept-Language: en-US,en;q=0.5' \
>   -H 'Connection: keep-alive' \
>   -H 'Upgrade-Insecure-Requests: 1' \
>   -H 'Sec-Fetch-Dest: document' \
>   -H 'Sec-Fetch-Mode: navigate' \
>   -H 'Sec-Fetch-Site: none' \
>   -H 'Sec-Fetch-User: ?1' \
>   -H 'Pragma: no-cache' \
>   -H 'Cache-Control: no-cache' \
>   --compressed --verbose
*   Trying 172.67.10.157:443...
* Connected to pegaxy.io (172.67.10.157) port 443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: C:/Program Files/Git/mingw64/ssl/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.pegaxy.io
*  start date: Mar  3 05:22:24 2022 GMT
*  expire date: Jun  1 05:22:23 2022 GMT
*  subjectAltName: host "pegaxy.io" matched cert's "pegaxy.io"
*  issuer: C=US; O=Let's Encrypt; CN=E1
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x671500)
> GET / HTTP/2
> Host: pegaxy.io
> accept-encoding: deflate, gzip
> user-agent: Mozilla/5.0 (Windows NT 10.0; rv:91.0) Gecko/20100101 Firefox/91.0
> accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
> accept-language: en-US,en;q=0.5
> connection: keep-alive
> upgrade-insecure-requests: 1
> sec-fetch-dest: document
> sec-fetch-mode: navigate
> sec-fetch-site: none
> sec-fetch-user: ?1
> pragma: no-cache
> cache-control: no-cache
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 403
< date: Fri, 18 Mar 2022 13:03:04 GMT
< content-type: text/html; charset=UTF-8
< cache-control: max-age=15
< expires: Fri, 18 Mar 2022 13:03:19 GMT
< x-frame-options: SAMEORIGIN
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< set-cookie: __cf_bm=rlU7vb3eTQzw02vcpzOo6gweMMadJXkNxsft3MqPLSY-1647608584-0-AXk3yx3EOmlDZ+tIGWB3S+1ud6hWmykBwT7IwKtO+e+eCdY36JjTgyM3SkdIyBeWvtZphzvnBZLCVE4R6YogbxI=; path=/; expires=Fri, 18-Mar-22 13:33:04 GMT; domain=.pegaxy.io; HttpOnly; Secure; SameSite=None
< vary: Accept-Encoding
< server: cloudflare
< cf-ray: 6ede2a920d7392c5-FRA
< content-encoding: gzip
<
<!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
<!--[if IE 7]>    <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->
<!--[if IE 8]>    <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]-->
<head>
<title>Attention Required! | Cloudflare</title>
<meta charset="UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=Edge,chrome=1" />
<meta name="robots" content="noindex, nofollow" />
<meta name="viewport" content="width=device-width,initial-scale=1" />
<link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/cf.errors.css" type="text/css" media="screen,projection" />
<!--[if lt IE 9]><link rel="stylesheet" id='cf_styles-ie-css' href="/cdn-cgi/styles/cf.errors.ie.css" type="text/css" media="screen,projection" /><![endif]-->
<style type="text/css">body{margin:0;padding:0}</style>


<!--[if gte IE 10]><!-->
<script>
  if (!navigator.cookieEnabled) {
    window.addEventListener('DOMContentLoaded', function () {
      var cookieEl = document.getElementById('cookie-alert');
      cookieEl.style.display = 'block';
    })
  }
</script>
<!--<![endif]-->


</head>
<body>
  <div id="cf-wrapper">
    <div class="cf-alert cf-alert-error cf-cookie-error" id="cookie-alert" data-translate="enable_cookies">Please enable cookies.</div>
    <div id="cf-error-details" class="cf-error-details-wrapper">
      <div class="cf-wrapper cf-header cf-error-overview">
        <h1 data-translate="block_headline">Sorry, you have been blocked</h1>
        <h2 class="cf-subheadline"><span data-translate="unable_to_access">You are unable to access</span> pegaxy.io</h2>
      </div><!-- /.header -->

      <div class="cf-section cf-highlight">
        <div class="cf-wrapper">
          <div class="cf-screenshot-container cf-screenshot-full">

              <span class="cf-no-screenshot error"></span>

          </div>
        </div>
      </div><!-- /.captcha-container -->

      <div class="cf-section cf-wrapper">
        <div class="cf-columns two">
          <div class="cf-column">
            <h2 data-translate="blocked_why_headline">Why have I been blocked?</h2>

            <p data-translate="blocked_why_detail">This website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting
a certain word or phrase, a SQL command or malformed data.</p>
          </div>

          <div class="cf-column">
            <h2 data-translate="blocked_resolve_headline">What can I do to resolve this?</h2>

            <p data-translate="blocked_resolve_detail">You can email the site owner to let them know you were blocked. Please include what you were doing when this page came up and the Cloudflare Ray ID found at the bottom of this page.</p>
          </div>
        </div>
      </div><!-- /.section -->

      <div class="cf-error-footer cf-wrapper w-240 lg:w-full py-10 sm:py-4 sm:px-8 mx-auto text-center sm:text-left border-solid border-0 border-t border-gray-300">
  <p class="text-13">
    <span class="cf-footer-item sm:block sm:mb-1">Cloudflare Ray ID: <strong class="font-semibold">6ede2a920d7392c5</strong></span>
    <span class="cf-footer-separator sm:hidden">&bull;</span>
    <span class="cf-footer-item sm:block sm:mb-1"><span>Your IP</span>: 46.62.217.20</span>
    <span class="cf-footer-separator sm:hidden">&bull;</span>
    <span class="cf-footer-item sm:block sm:mb-1"><span>Performance &amp; security by</span> <a rel="noopener noreferrer" href="https://www.cloudflare.com/5xx-error-landing" id="brand_link" target="_blank">Cloudflare</a></span>

  </p>
</div><!-- /.error-footer -->


    </div><!-- /#cf-error-details -->
  </div><!-- /#cf-wrapper -->

  <script type="text/javascript">
  window._cf_translation = {};


</script>

</body>
</html>
* Connection #0 to host pegaxy.io left intact

在 windows 和 linux 上测试。

$ curl -V
curl 7.70.0 (x86_64-w64-mingw32) libcurl/7.70.0 OpenSSL/1.1.1g (Schannel) zlib/1.2.11 libidn2/2.3.0 libssh2/1.9.0 nghttp2/1.40.0
Release-Date: 2020-04-29
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz Metalink MultiSSL NTLM SPNEGO SSL SSPI TLS-SRP

请注意，谈论的正是第一个请求。所以这不能与 cookie 或一段检查浏览器状态的 JavaScript 代码有关。

此外，我什至知道 CloudFlare 对 IP 和请求速率的敏感性，但这些是针对连续且混乱的请求，并最终显示挑战的情况。但是在我的测试中，网络上的同一个IP是没有问题的，只有第一个请求出现。

在 Chrome、Firefox、Edge、Brave、Tor 网络浏览器上没有问题。
但是命令行的CURL、wget、nghttp、lynx有问题。我还测试了几个有问题的 nodejs 包。

问题：当所有条件都相同时，CloudFlare如何发现请求不是来自浏览器，如何在不使用命令行的情况下模拟或绕过网络浏览器？

请记住，我知道 Selenium、Flaresolverr、pupflare 等，我不打算使用浏览器，因为它们会呈现页面并减慢操作。

我做了和没有得到答案的事情：

我认为问题可能出在 CURL 命令行中缺少 push server http2 支持。于是我写在了PHP里面，并在里面实现了push server，但是问题并没有解决
我以为是证书的问题，所以我下载了浏览器证书并转换成pem文件在CURL中使用，但问题并没有解决

如果我在网络浏览器上的第一个请求中获得代码 200，我只想在第一种情况下在终端上获得相同的 200 响应代码！

Answer 1

Cloudflare 使用各种技术来确定用户代理是否是真正的浏览器。而且，站点所有者还可以通过 Cloudflare 平台确定他们可以允许的风险级别。

让我们讨论一下 Cloudflare 使用的一些技术（我知道）：

TLS 指纹识别这是 Cloudflare 臭名昭著的著名技术之一。这也是原生代理等工具大行其道的原因。 Link: https://github.com/klzgrad/naiveproxy
Cookies Cloudflare曾经有一些cf_相关的cookie，用于区分真实用户与否。

而且，这些只是一些技巧。 Cloudflare 还有更多。

而且，这个问题不仅限于Cloudflare，中国防火墙也因使用这种作案手法来区分各种事物而臭名昭著。

CloudFlare 在哪里检测网络和终端请求？！平等对待

Where does CloudFlare detect web and terminal requests?! On equal terms

curl

request

cloudflare

bypass