通过 Fiddler 和 NTLM 身份验证使用 Python 请求库不一致成功
Using Python requests library through Fiddler and NTLM authentication is inconsistently successful
我已经研究这个问题将近 20 个小时了。
我的所有网络流量都通过我机器上的 Fiddler 路由,然后连接到我们的公司代理。一切正常,除了 Python 应用程序尝试使用 https 访问远程服务器(http 始终正常)。
我导出了公司证书并粘贴到文件中:C:\anaconda2\envs\py36\Lib\site-packages\certifi\cacert.pem。我还使用 verify= 在我的 requests.get 调用中明确设置了它。行为没有区别。
我将本地fiddler代理信息设置为环境变量。 Fiddler 还配置为自动验证。使用 http 没有任何问题。
我似乎只能通过 https 连接到远程服务器,如果我先转到 http://www.google.com,然后快速尝试使用 https 连接。随后的尝试产生以下错误
requests.get('http://www.google.com') # always works for any website
<Response [200]>
requests.get('https://www.anaconda.com') # works after visiting http://www.google.com
<Response [200]>
requests.get('https://www.anaconda.com') # always fails, unless visiting http://www.google.com first
---------------------------------------------------------------------------
OSError Traceback (most recent call last)
C:\anaconda2\envs\py36\lib\site-packages\urllib3\connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
593 if is_new_proxy_conn:
--> 594 self._prepare_proxy(conn)
595
C:\anaconda2\envs\py36\lib\site-packages\urllib3\connectionpool.py in _prepare_proxy(self, conn)
804 conn.set_tunnel(self._proxy_host, self.port, self.proxy_headers)
--> 805 conn.connect()
806
C:\anaconda2\envs\py36\lib\site-packages\urllib3\connection.py in connect(self)
307 # self._tunnel_host below.
--> 308 self._tunnel()
309 # Mark this connection as not reusable
C:\anaconda2\envs\py36\lib\http\client.py in _tunnel(self)
918 raise OSError("Tunnel connection failed: %d %s" % (code,
--> 919 message.strip()))
920 while True:
OSError: Tunnel connection failed: 407 Proxy Authentication Required
During handling of the above exception, another exception occurred:
MaxRetryError Traceback (most recent call last)
C:\anaconda2\envs\py36\lib\site-packages\requests\adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
448 retries=self.max_retries,
--> 449 timeout=timeout
450 )
C:\anaconda2\envs\py36\lib\site-packages\urllib3\connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
637 retries = retries.increment(method, url, error=e, _pool=self,
--> 638 _stacktrace=sys.exc_info()[2])
639 retries.sleep()
C:\anaconda2\envs\py36\lib\site-packages\urllib3\util\retry.py in increment(self, method, url, response, error, _pool, _stacktrace)
397 if new_retry.is_exhausted():
--> 398 raise MaxRetryError(_pool, url, error or ResponseError(cause))
399
MaxRetryError: HTTPSConnectionPool(host='www.google.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 407 Proxy Authentication Required',)))
During handling of the above exception, another exception occurred:
ProxyError Traceback (most recent call last)
<ipython-input-49-df48f2544f7e> in <module>
----> 1 requests.get('https://www.google.com')
C:\anaconda2\envs\py36\lib\site-packages\requests\api.py in get(url, params, **kwargs)
73
74 kwargs.setdefault('allow_redirects', True)
---> 75 return request('get', url, params=params, **kwargs)
76
77
C:\anaconda2\envs\py36\lib\site-packages\requests\api.py in request(method, url, **kwargs)
58 # cases, and look like a memory leak in others.
59 with sessions.Session() as session:
---> 60 return session.request(method=method, url=url, **kwargs)
61
62
C:\anaconda2\envs\py36\lib\site-packages\requests\sessions.py in request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
531 }
532 send_kwargs.update(settings)
--> 533 resp = self.send(prep, **send_kwargs)
534
535 return resp
C:\anaconda2\envs\py36\lib\site-packages\requests\sessions.py in send(self, request, **kwargs)
644
645 # Send the request
--> 646 r = adapter.send(request, **kwargs)
647
648 # Total elapsed time of the request (approximately)
C:\anaconda2\envs\py36\lib\site-packages\requests\adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
508
509 if isinstance(e.reason, _ProxyError):
--> 510 raise ProxyError(e, request=request)
511
512 if isinstance(e.reason, _SSLError):
ProxyError: HTTPSConnectionPool(host='www.google.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 407 Proxy Authentication Required',)))
在我发出请求时,其中一位网络人员正在查看公司代理日志。当发出失败的 https 请求时,他没有在其日志中看到与公司代理的连接。
其他尝试过的东西:
- requests-ntlm.
- 重置 Fiddler 的所有证书。
- 明确将代理和公司证书信息传递给请求。
谢谢。
对于有同样问题的人。
进一步的研究让我下载了 Python 应用程序 Px (px.exe) Px on GitHub 和 turf Fiddler,它只对 Python 试图退出的应用程序间歇性工作到互联网。
在我的例子中,PX 本身需要 ZERO 配置。我只需要设置 http_proxy 和 https_proxy 环境变量,以便任何 Python 应用程序都知道将流量集中到哪里。然后我只是 运行 Px,一切正常。
希望这对人们有所帮助。
我已经研究这个问题将近 20 个小时了。
我的所有网络流量都通过我机器上的 Fiddler 路由,然后连接到我们的公司代理。一切正常,除了 Python 应用程序尝试使用 https 访问远程服务器(http 始终正常)。
我导出了公司证书并粘贴到文件中:C:\anaconda2\envs\py36\Lib\site-packages\certifi\cacert.pem。我还使用 verify= 在我的 requests.get 调用中明确设置了它。行为没有区别。
我将本地fiddler代理信息设置为环境变量。 Fiddler 还配置为自动验证。使用 http 没有任何问题。
我似乎只能通过 https 连接到远程服务器,如果我先转到 http://www.google.com,然后快速尝试使用 https 连接。随后的尝试产生以下错误
requests.get('http://www.google.com') # always works for any website
<Response [200]>
requests.get('https://www.anaconda.com') # works after visiting http://www.google.com
<Response [200]>
requests.get('https://www.anaconda.com') # always fails, unless visiting http://www.google.com first
---------------------------------------------------------------------------
OSError Traceback (most recent call last)
C:\anaconda2\envs\py36\lib\site-packages\urllib3\connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
593 if is_new_proxy_conn:
--> 594 self._prepare_proxy(conn)
595
C:\anaconda2\envs\py36\lib\site-packages\urllib3\connectionpool.py in _prepare_proxy(self, conn)
804 conn.set_tunnel(self._proxy_host, self.port, self.proxy_headers)
--> 805 conn.connect()
806
C:\anaconda2\envs\py36\lib\site-packages\urllib3\connection.py in connect(self)
307 # self._tunnel_host below.
--> 308 self._tunnel()
309 # Mark this connection as not reusable
C:\anaconda2\envs\py36\lib\http\client.py in _tunnel(self)
918 raise OSError("Tunnel connection failed: %d %s" % (code,
--> 919 message.strip()))
920 while True:
OSError: Tunnel connection failed: 407 Proxy Authentication Required
During handling of the above exception, another exception occurred:
MaxRetryError Traceback (most recent call last)
C:\anaconda2\envs\py36\lib\site-packages\requests\adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
448 retries=self.max_retries,
--> 449 timeout=timeout
450 )
C:\anaconda2\envs\py36\lib\site-packages\urllib3\connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
637 retries = retries.increment(method, url, error=e, _pool=self,
--> 638 _stacktrace=sys.exc_info()[2])
639 retries.sleep()
C:\anaconda2\envs\py36\lib\site-packages\urllib3\util\retry.py in increment(self, method, url, response, error, _pool, _stacktrace)
397 if new_retry.is_exhausted():
--> 398 raise MaxRetryError(_pool, url, error or ResponseError(cause))
399
MaxRetryError: HTTPSConnectionPool(host='www.google.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 407 Proxy Authentication Required',)))
During handling of the above exception, another exception occurred:
ProxyError Traceback (most recent call last)
<ipython-input-49-df48f2544f7e> in <module>
----> 1 requests.get('https://www.google.com')
C:\anaconda2\envs\py36\lib\site-packages\requests\api.py in get(url, params, **kwargs)
73
74 kwargs.setdefault('allow_redirects', True)
---> 75 return request('get', url, params=params, **kwargs)
76
77
C:\anaconda2\envs\py36\lib\site-packages\requests\api.py in request(method, url, **kwargs)
58 # cases, and look like a memory leak in others.
59 with sessions.Session() as session:
---> 60 return session.request(method=method, url=url, **kwargs)
61
62
C:\anaconda2\envs\py36\lib\site-packages\requests\sessions.py in request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
531 }
532 send_kwargs.update(settings)
--> 533 resp = self.send(prep, **send_kwargs)
534
535 return resp
C:\anaconda2\envs\py36\lib\site-packages\requests\sessions.py in send(self, request, **kwargs)
644
645 # Send the request
--> 646 r = adapter.send(request, **kwargs)
647
648 # Total elapsed time of the request (approximately)
C:\anaconda2\envs\py36\lib\site-packages\requests\adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
508
509 if isinstance(e.reason, _ProxyError):
--> 510 raise ProxyError(e, request=request)
511
512 if isinstance(e.reason, _SSLError):
ProxyError: HTTPSConnectionPool(host='www.google.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 407 Proxy Authentication Required',)))
在我发出请求时,其中一位网络人员正在查看公司代理日志。当发出失败的 https 请求时,他没有在其日志中看到与公司代理的连接。
其他尝试过的东西:
- requests-ntlm.
- 重置 Fiddler 的所有证书。
- 明确将代理和公司证书信息传递给请求。
谢谢。
对于有同样问题的人。
进一步的研究让我下载了 Python 应用程序 Px (px.exe) Px on GitHub 和 turf Fiddler,它只对 Python 试图退出的应用程序间歇性工作到互联网。
在我的例子中,PX 本身需要 ZERO 配置。我只需要设置 http_proxy 和 https_proxy 环境变量,以便任何 Python 应用程序都知道将流量集中到哪里。然后我只是 运行 Px,一切正常。
希望这对人们有所帮助。