Chrome 具有公司代理身份验证的无头浏览器无法正常工作

Question

我正在尝试运行坐在公司代理后面的 Chrome 无头浏览器。我试过下面的代码。却无法通过

public class HeadlessChrome 
{
    WebDriver driver;

    @Test
    public void createChromeDriverHeadless() throws InterruptedException
    {
        System.setProperty("webdriver.chrome.driver", "D:\LocalData\workspace\Drivers and Libraries\driver\chromedriver.exe");

        ChromeOptions chromeOptions = new ChromeOptions();

        Proxy proxy = new Proxy();
        proxy.setHttpProxy("http://user:pwd@server:port");
        proxy.setSslProxy("http://user:pwd@server:port");

//      chromeOptions.setCapability("proxy", proxy);
        chromeOptions.addArguments("--proxy-server=user:pwd@server:port");
        chromeOptions.addArguments("--headless");
        chromeOptions.addArguments("--disable-gpu");
        chromeOptions.addArguments("start-maximized");  

        driver = new ChromeDriver(chromeOptions);
        driver.get("http://seleniumhq.org");       

        Thread.sleep(5000);
        System.out.println("Title : " + driver.getTitle());
        assertTrue(driver.findElement(By.id("q")).isDisplayed());
        driver.quit();
    }
}

请帮帮我。

Answer 1

如果您没有使用 headless，您可以使用下面的方法 link

但目前不允许使用无头扩展。所以现在你的选择是添加另一个代理

chrome -> (intermediate proxy w/o auth) -> corporate proxy w/ auth -> internet

一种选择是使用 polipo

https://www.irif.fr/~jch/software/polipo/

配置如下

parentAuthCredentials=username:password
parentProxy=corporateproxy:port

然后使用

chromeOptions.addArguments("--proxy-server=http://polipoproxy:port");

在 polipo 配置中不要覆盖，默认值为 127.0.0.1:8123。

您可以使用的其他选项

使用squid代理代替polipo
使用 python 或节点或您熟悉的任何其他语言编写您自己的代理转发器

Answer 2

与Luminati合作的名为Linken-sphere的无头浏览器。他们提供一些不错的报价。你应该检查一下。 https://miped.ru/f/threads/linken-sphere-antidetekt-brauzer-novogo-pokolenija.67098/

Answer 3

使用 selenium 4 很容易实现（目前处于测试阶段）。您可以通过多种方式进行：

您基本上需要注册检查是否将凭据应用于任何授权请求。适用于基本和代理身份验证弹出窗口。

ChromeDriver driver = new ChromeDriver(new ChromeOptions().setHeadless(true));
String USER_NAME = "guest";
String PASSWORD = "guest";
//register our check here
driver.register(UsernameAndPassword.of(USER_NAME, PASSWORD));
driver.get("https://jigsaw.w3.org/HTTP/");
//Click on the link to show an authentication popup
driver.findElement(By.linkText("Basic Authentication test")).click();
String msg = driver.findElement(By.tagName("html")).getText();
assert msg.equalsIgnoreCase("Your browser made it!");

在 puppeteer 中使用 CDP Network domain. Doesn't work for proxy authorization popup (for example here is the similar issue，然后进入 chrome 项目）

ChromeDriver driver = new ChromeDriver(new ChromeOptions().setHeadless(true));
String USER_NAME = "guest";
String PASSWORD = "guest";

DevTools devTools = driver.getDevTools();
//create a cdp session
devTools.createSession();
//enable network first
devTools.send(Network.enable(Optional.empty(), Optional.empty(), Optional.empty()));
//Open website
driver.get("https://jigsaw.w3.org/HTTP/");
//Create and send the authorization header
Map<String, Object> headers = new HashMap<>();
String basicAuth = "Basic " + new String(Base64.getEncoder().encode(String.format("%s:%s", USER_NAME, PASSWORD).getBytes()));
headers.put("Authorization", basicAuth);
devTools.send(Network.setExtraHTTPHeaders(new Headers(headers)));
//Click on the link to show an authentication popup
driver.findElement(By.linkText("Basic Authentication test")).click();
String msg = driver.findElement(By.tagName("html")).getText();
assert msg.equalsIgnoreCase("Your browser made it!");

正在使用CDP Fetch domain。适用于基本和代理身份验证弹出窗口。

ChromeDriver driver = new ChromeDriver(new ChromeOptions().setHeadless(true));
String USER_NAME = "guest";
String PASSWORD = "guest";

DevTools devTools = driver.getDevTools();
//create a cdp session
devTools.createSession();
//enable Fetch first
devTools.send(Fetch.enable(Optional.empty(), Optional.of(true)));
devTools.addListener(Fetch.requestPaused(), requestPaused -> devTools.send(Fetch.continueRequest(requestPaused.getRequestId(), Optional.empty(), Optional.empty(), Optional.empty(), Optional.empty())));
devTools.addListener(Fetch.authRequired(), authRequired -> devTools.send(Fetch.continueWithAuth(authRequired.getRequestId(), new AuthChallengeResponse(PROVIDECREDENTIALS, Optional.of(USER_NAME), Optional.of(PASSWORD)))));

//Open website
driver.get("https://jigsaw.w3.org/HTTP/");
//Click on the link to show an authentication popup
driver.findElement(By.linkText("Basic Authentication test")).click();
String msg = driver.findElement(By.tagName("html")).getText();
assert msg.equalsIgnoreCase("Your browser made it!");

Chrome 具有公司代理身份验证的无头浏览器无法正常工作

Chrome headless browser with corporate proxy authetication not working

proxy

headless

basic-authentication

selenium-chromedriver

chrome-options