如何让 HTTP::Proxy 使用 HTTPS URL？

Question

在下面的代码示例中，我使用 HTTP::Proxy 启动代理服务器并尝试使用它来请求 HTTPS URL，但代理服务器实际上并没有发出请求，或者 return 没有响应。但是，如果我让 URL 使用 HTTP（不安全），请求就会成功。我已经安装了 IO::Socket::SSL 和 LWP::UserAgent::https（是的秘密部门！），但我仍然无法让 HTTPS 请求通过代理。如何让 HTTP::Proxy 使用 HTTPS URLs？

这是我的代码：

#!/usr/bin/env perl

use strict;
use warnings;
use Data::Printer;
use HTTP::Proxy ':log';
use Mojo::UserAgent ();

my $URL = 'https://www.yahoo.com';
my $PROXY_PORT = 8667;

my $pid = fork();

if ($pid) { # I am the parent
    print "Press ^c to kill proxy server...\n";
    my $proxy = HTTP::Proxy->new( port => $PROXY_PORT );
    $proxy->logmask(ALL);
    $proxy->via(q{});
    $proxy->x_forwarded_for(0);
    $proxy->start;

    waitpid $pid, 0;
}
elsif ($pid == 0) { # I am the child
    sleep 3; # Allow the proxy server to start

    my $ua = Mojo::UserAgent->new;
    $ua->proxy
        ->http("http://127.0.0.1:$PROXY_PORT")
        ->https("http://127.0.0.1:$PROXY_PORT");

    my $tx = $ua->get($URL);

    if ($tx->error) {
        p $tx->error;
    }
    else {
        print "Success!\n";
    }
}
else {
    die 'Unknown result after forking';
}

将上面的脚本保存为testcase-so.pl and 运行 it:

$ MOJO_CLIENT_DEBUG=1 ./testcase-so.pl
Press ^c to kill proxy server...
-- Blocking request (https://www.yahoo.com)
-- Connect c66a92739c09c76fa24029e8079808c7 (https://www.yahoo.com:443)
-- Client >>> Server (https://www.yahoo.com)
CONNECT www.yahoo.com:443 HTTP/1.1\x0d
User-Agent: Mojolicious (Perl)\x0d
Content-Length: 0\x0d
Host: www.yahoo.com\x0d
Accept-Encoding: gzip\x0d
\x0d

-- Client >>> Server (https://www.yahoo.com)

[Tue Oct  9 12:02:54 2018] (12348) PROCESS: Forked child process 12352
[Tue Oct  9 12:02:54 2018] (12352) SOCKET: New connection from 127.0.0.1:45312
[Tue Oct  9 12:02:54 2018] (12352) REQUEST: CONNECT www.yahoo.com:443
[Tue Oct  9 12:02:54 2018] (12352) REQUEST: Accept-Encoding: gzip
[Tue Oct  9 12:02:54 2018] (12352) REQUEST: Host: www.yahoo.com
[Tue Oct  9 12:02:54 2018] (12352) REQUEST: User-Agent: Mojolicious (Perl)
[Tue Oct  9 12:02:54 2018] (12352) REQUEST: Content-Length: 0
[Tue Oct  9 12:02:54 2018] (12352) RESPONSE: 200 OK
[Tue Oct  9 12:02:54 2018] (12352) RESPONSE: Date: Tue, 09 Oct 2018 12:02:54 GMT
[Tue Oct  9 12:02:54 2018] (12352) RESPONSE: Transfer-Encoding: chunked
[Tue Oct  9 12:02:54 2018] (12352) RESPONSE: Server: HTTP::Proxy/0.304
-- Client <<< Server (https://www.yahoo.com)
HTTP/1.1 200 OK\x0d
Date: Tue, 09 Oct 2018 12:02:54 GMT\x0d
Transfer-Encoding: chunked\x0d
Server: HTTP::Proxy/0.304\x0d
\x0d

[Tue Oct  9 12:03:14 2018] (12352) CONNECT: Connection closed by the client
[Tue Oct  9 12:03:14 2018] (12352) PROCESS: Served 1 requests
[Tue Oct  9 12:03:14 2018] (12352) CONNECT: End of CONNECT proxyfication
\ {
    message   "Proxy connection failed"
}
[Tue Oct  9 12:03:15 2018] (12348) PROCESS: Reaped child process 12349
[Tue Oct  9 12:03:15 2018] (12348) PROCESS: 1 remaining kids: 12352
[Tue Oct  9 12:03:15 2018] (12348) PROCESS: Reaped child process 12352
[Tue Oct  9 12:03:15 2018] (12348) PROCESS: 0 remaining kids:
^C[Tue Oct  9 12:04:04 2018] (12348) STATUS: Processed 2 connection(s)
$

并且随着 $URL 切换为不使用 https:

$ MOJO_CLIENT_DEBUG=1 ./testcase-so.pl
Press ^c to kill proxy server...
-- Blocking request (http://www.yahoo.com)
-- Connect f792ee97a0362ab493575d8116e69e59 (http://127.0.0.1:8667)
-- Client >>> Server (http://www.yahoo.com)
GET http://www.yahoo.com HTTP/1.1\x0d
Accept-Encoding: gzip\x0d
Content-Length: 0\x0d
Host: www.yahoo.com\x0d
User-Agent: Mojolicious (Perl)\x0d
\x0d

[Tue Oct  9 12:09:38 2018] (12656) PROCESS: Forked child process 12659
-- Client >>> Server (http://www.yahoo.com)

[Tue Oct  9 12:09:38 2018] (12659) SOCKET: New connection from 127.0.0.1:58288
[Tue Oct  9 12:09:38 2018] (12659) REQUEST: GET http://www.yahoo.com
[Tue Oct  9 12:09:38 2018] (12659) REQUEST: Accept-Encoding: gzip
[Tue Oct  9 12:09:38 2018] (12659) REQUEST: Host: www.yahoo.com
[Tue Oct  9 12:09:38 2018] (12659) REQUEST: User-Agent: Mojolicious (Perl)
[Tue Oct  9 12:09:38 2018] (12659) REQUEST: Content-Length: 0
[Tue Oct  9 12:09:38 2018] (12659) RESPONSE: 301 Moved Permanently
[Tue Oct  9 12:09:38 2018] (12659) RESPONSE: Cache-Control: no-store, no-cache
[Tue Oct  9 12:09:38 2018] (12659) RESPONSE: Date: Tue, 09 Oct 2018 14:10:01 GMT
[Tue Oct  9 12:09:38 2018] (12659) RESPONSE: Transfer-Encoding: chunked
[Tue Oct  9 12:09:38 2018] (12659) RESPONSE: Via: http/1.1 media-router-fp1006.prod.media.bf1.yahoo.com (ApacheTrafficServer [c s f ])
[Tue Oct  9 12:09:38 2018] (12659) RESPONSE: Location: https://www.yahoo.com/
[Tue Oct  9 12:09:38 2018] (12659) RESPONSE: Server: ATS
[Tue Oct  9 12:09:38 2018] (12659) RESPONSE: Content-Language: en
[Tue Oct  9 12:09:38 2018] (12659) RESPONSE: Content-Length: 8
[Tue Oct  9 12:09:38 2018] (12659) RESPONSE: Content-Type: text/html
[Tue Oct  9 12:09:38 2018] (12659) RESPONSE: Content-Security-Policy: sandbox allow-forms allow-same-origin allow-scripts allow-popups allow-popups-to-escape-sandbox allow-presentation; report-uri https://csp.yahoo.com/beacon/csp?src=ats&site=frontpage&region=US&lang=en-US&device=desktop&yrid=&partner=;
[Tue Oct  9 12:09:38 2018] (12659) RESPONSE: X-Frame-Options: SAMEORIGIN
[Tue Oct  9 12:09:38 2018] (12659) RESPONSE: X-XSS-Protection: 1; report="https://csp.yahoo.com/beacon/csp?src=fp-hpkp-www"
-- Client <<< Server (http://www.yahoo.com)
HTTP/1.1 301 Moved Permanently\x0d
Cache-Control: no-store, no-cache\x0d
Date: Tue, 09 Oct 2018 14:10:01 GMT\x0d
Transfer-Encoding: chunked\x0d
Via: http/1.1 media-router-fp1006.prod.media.bf1.yahoo.com (ApacheTrafficServer [c s f ])\x0d
Location: https://www.yahoo.com/\x0d
Server: ATS\x0d
Content-Language: en\x0d
Content-Length: 8\x0d
Content-Type: text/html\x0d
Content-Security-Policy: sandbox allow-forms allow-same-origin allow-scripts allow-popups allow-popups-to-escape-sandbox allow-presentation; report-uri https://csp.yahoo.com/beacon/csp?src=ats&site=frontpage&region=US&lang=en-US&device=desktop&yrid=&partner=;\x0d
X-Frame-Options: SAMEORIGIN\x0d
X-XSS-Protection: 1; report="https://csp.yahoo.com/beacon/csp?src=fp-hpkp-www"\x0d
\x0d

-- Client <<< Server (http://www.yahoo.com)
8\x0d
redirect\x0d
0\x0d
\x0d

Success!
[Tue Oct  9 12:09:38 2018] (12659) SOCKET: Getting request failed: Client closed
[Tue Oct  9 12:09:39 2018] (12656) PROCESS: Reaped child process 12657
[Tue Oct  9 12:09:39 2018] (12656) PROCESS: 1 remaining kids: 12659
[Tue Oct  9 12:09:39 2018] (12656) PROCESS: Reaped child process 12659
[Tue Oct  9 12:09:39 2018] (12656) PROCESS: 0 remaining kids:
^C[Tue Oct  9 12:09:45 2018] (12656) STATUS: Processed 2 connection(s)
$

Answer 1

HTTP::Proxy 中存在错误，它 returns 对 CONNECT 请求的错误响应：

-- Client <<< Server (https://www.yahoo.com)
HTTP/1.1 200 OK\x0d
Date: Tue, 09 Oct 2018 12:02:54 GMT\x0d
Transfer-Encoding: chunked\x0d
Server: HTTP::Proxy/0.304\x0d
\x0d

对 CONNECT 请求的响应可以没有 body，这意味着它不应该像 Transfer-Encoding: chunked 那样有 HTTP header 宣布 body。此错误发生在所有使用 HTTP/1.1 执行 CONNECT 请求的客户端上。如果 CONNECT 是用 HTTP/1.0 完成的，问题就会消失，因为 Transfer-Encoding: chunked 还没有用 HTTP/1.0 定义，因此 HTTP::Proxy 不会发送它。

尝试将 curl 与 HTTP::Proxy 一起使用时会发生同样的问题，因此这不仅仅是 Mojo::UserAgent 的问题。我为 HTTP::Proxy 制作了一个补丁以正确响应。有关详细信息和您需要应用的（小）差异，请参阅 this pull request。

如何让 HTTP::Proxy 使用 HTTPS URL？

How can I make HTTP::Proxy work with HTTPS URLs?

perl

http-proxy