通过 Selenium Python 在 normal/headless 模式下使用 ChromeDriver/Chrome 访问 Cloudflare 网站有什么区别

Question

我对 Python Selenium 中 Chrome 的 --headless 模式有疑问。

代码

 from selenium import webdriver
 from selenium.webdriver.common.desired_capabilities import DesiredCapabilities

 CHROME_DRIVER_DIR = "selenium/chromedriver"

 chrome_options = webdriver.ChromeOptions()
 caps = DesiredCapabilities().CHROME
 chrome_options.add_argument("--disable-dev-shm-usage")
 chrome_options.add_argument("--remote-debugging-port=9222")
 chrome_options.add_argument("--headless")  # Runs Chrome in headless mode.
 chrome_options.add_argument('--no-sandbox')  # # Bypass OS security model
 chrome_options.add_argument("--disable-extensions")
 chrome_options.add_argument("--disable-gpu")

 browser = webdriver.Chrome(desired_capabilities=caps, executable_path=CHROME_DRIVER_DIR, options=chrome_options)

 browser.get("https://www.manta.com/c/mm2956g/mashuda-contractors")
 print(browser.page_source)
 browser.quit()

当我删除时 chrome_options.add_argument("--headless") 一切正常，但是 --headless* 有下一期

Please enable cookies.

Error 1020 Ray ID: 53fd62b4087d8116 • 2019-12-04 11:19:28 UTC

Access denied

What happened?
This website is using a security service to protect itself from online attacks.

Cloudflare Ray ID: 53fd62b4087d8116 • Your IP: 168.81.117.111 • Performance & security by Cloudflare

普通模式和--headless有什么区别？

Answer 1

我拿了你的代码，删除了可选的 arguments 并添加了一些 arguments 来执行测试如下：

代码块：

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

options = webdriver.ChromeOptions() 
options.add_argument("start-maximized")
options.add_argument("--headless")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
driver.get("https://www.manta.com/c/mm2956g/mashuda-contractors")
print(driver.page_source)
driver.quit()

控制台输出：

<html class="js" lang="en-US" style="opacity: 1; visibility: visible;"><!--<![endif]--><head>
<title>Access denied | www.manta.com used Cloudflare to restrict access</title>
<meta charset="UTF-8">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta http-equiv="X-UA-Compatible" content="IE=Edge,chrome=1">
<meta name="robots" content="noindex, nofollow">
<meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1">
<link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/cf.errors.css" type="text/css" media="screen,projection">
<!--[if lt IE 9]><link rel="stylesheet" id='cf_styles-ie-css' href="/cdn-cgi/styles/cf.errors.ie.css" type="text/css" media="screen,projection" /><![endif]-->
<style type="text/css">body{margin:0;padding:0}</style>


<!--[if gte IE 10]><!--><script type="text/javascript" src="/cdn-cgi/scripts/zepto.min.js"></script><!--<![endif]-->
<!--[if gte IE 10]><!--><script type="text/javascript" src="/cdn-cgi/scripts/cf.common.js"></script><!--<![endif]-->



</head>
<body>
  <div id="cf-wrapper">
    <div class="cf-alert cf-alert-error cf-cookie-error" id="cookie-alert" data-translate="enable_cookies">Please enable cookies.</div>
    <div id="cf-error-details" class="cf-error-details-wrapper">
      <div class="cf-wrapper cf-header cf-error-overview">
    <h1>
      <span class="cf-error-type" data-translate="error">Error</span>
      <span class="cf-error-code">1020</span>
      <small class="heading-ray-id">Ray ID: 53fd7c2fca12d5fc • 2019-12-04 11:36:52 UTC</small>
    </h1>
    <h2 class="cf-subheadline">Access denied</h2>
      </div><!-- /.header -->

      <section></section><!-- spacer -->

      <div class="cf-section cf-wrapper">
    <div class="cf-columns two">
      <div class="cf-column">
        <h2 data-translate="what_happened">What happened?</h2>
        <p>This website is using a security service to protect itself from online attacks.</p>
      </div>


    </div>
      </div><!-- /.section -->

      <div class="cf-error-footer cf-wrapper">
  <p>
    <span class="cf-footer-item">Cloudflare Ray ID: <strong>53fd7c2fca12d5fc</strong></span>
    <span class="cf-footer-separator">•</span>
    <span class="cf-footer-item"><span>Your IP</span>: 123.201.54.43</span>
    <span class="cf-footer-separator">•</span>
    <span class="cf-footer-item"><span>Performance &amp; security by</span> <a href="https://www.cloudflare.com/5xx-error-landing?utm_source=error_footer" id="brand_link" target="_blank">Cloudflare</a></span>

  </p>
</div><!-- /.error-footer -->


    </div><!-- /#cf-error-details -->
  </div><!-- /#cf-wrapper -->

  <script type="text/javascript">
  window._cf_translation = {};


</script>



</body></html>

分析

从提取的页面源代码中可以清楚地看到，使用 --headless 参数，您将访问具有以下内容的页面：

标题为：访问被拒绝| www.manta.com 使用 Cloudflare 限制访问。
一些信息：发生了什么事？：该网站正在使用安全服务来保护自己免受在线攻击。

结论

浏览上下文 即 Chrome 浏览器 session 被检测为 BOT 并且导航被封锁.

结尾

您可以在以下位置找到一些相关讨论：

Answer 2

Cloudflare 旨在阻止机器人程序。他们假设无头浏览器被数据抓取器使用，所以他们阻止了它。来自 Cloudflare What is Data Scraping?

*A headless browser is a type of web browser, much like Chrome or Firefox, but it doesn’t have a visual user interface by default, allowing it to move much faster than a typical web browser. By essentially running at the level of a command line, a headless browser is able to avoid rendering entire web applications. Data scrapers write bots that use headless browsers to request data more quickly, as there is no human viewing each page being scraped.

Answer 3

我使用这个 server-side 脚本进行了测试：

<?php
echo "<pre><code>";
var_dump($_SERVER);
echo "</code></pre>";
?>
<script>
    var el = document.getElementsByTagName('code')[0];
    for(var prop in window.navigator){
        var str = JSON.stringify(window.navigator[prop])
        el.innerHTML = el.innerHTML + "window.navigator." + prop + " = " + str + "\n";
    }
    var skip_props = ['parent', 'top', 'frames', 'self', 'window'];
    for(var prop in window){
        if (skip_props.indexOf(prop) > -1) { continue; }
        el.innerHTML = el.innerHTML + "window." + prop + " = ";
        var str = JSON.stringify(window[prop])
        el.innerHTML = el.innerHTML + str + "\n";
    }
</script>

我使用 ChromeDriver 加载此页面，使用和不使用 --headless，并使用 print(driver.find_element_by_tag_name('code').text) 打印输出。然后我 diff-ed 两个输出。
这是我发现的差异：

HTTP Accept-Language header：en-US,en;q=0.9 对比 en-US
HTTP User-Agent header：Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36 vs Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/83.0.4103.61 Safari/537.36（注意第二个字符串中提到的 HeadlessChrome。）
Javascript window.navigator.plugins: {"0":{"0":{}},"1":{"0":{}},"2":{"0":{},"1":{}}} 对比 {}
Javascript window.navigator.mimeTypes：{"0":{},"1":{},"2":{},"3":{}} 对比 {}
Javascript window.outerWidth: 1367 对比 0
Javascript window.outerHeight：641 对比 0

注意：在您发布的 Python 脚本中，您遗漏了几行，以删除 window.webdriver 属性（没有这个，服务器很容易检测到您正在使用 WebDriver）[ref]:

driver.execute_cdp_cmd("Page.addScriptToEvaluateOnNewDocument", {
    "source": """
    Object.defineProperty(navigator, 'webdriver', {
      get: () => undefined
    })
  """
})

Answer 4

这是 Cloudflare 不喜欢的 HTTP User-Agent header。

要解决此问题，只需更改 user-agent chrome 选项（以下代码适用于 Python 中的 Selenium）：

option.add_argument('--headless')
option.add_argument("user-agent=Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36")

通过 Selenium Python 在 normal/headless 模式下使用 ChromeDriver/Chrome 访问 Cloudflare 网站有什么区别

What is the difference in accessing Cloudflare website using ChromeDriver/Chrome in normal/headless mode through Selenium Python

python

selenium

cloudflare

selenium-chromedriver

google-chrome-headless

分析

结论

结尾