使用袜子时 gevent + requests 块

Question

我使用 python(2.7.6),gevnet(1.1.2),requests(2.11.1) 使 http 请求并发，效果很好。但是当我向请求添加 socks 代理时，它会阻塞。

这是我的代码：

    import time
    import requests
    import logging
    import click

    import gevent
    from gevent import monkey
    monkey.patch_all()

    FORMAT = '%(asctime)-15s %(message)s'
    logging.basicConfig(format=FORMAT)
    logger = logging.getLogger('test')

    #socks proxy
    user = MY_SOCKS_PROXY_USERNAME
    password = MY_SOCKS_PROXY_PASSWORD
    host = MY_SOCKS_PROXY_HOST
    port = MY_SOCKS_PROXY_PORT
    proxies = {
            'http': 'socks5://{0}:{1}@{2}:{3}'.format(user, password, host, port),
            'https': 'socks5://{0}:{1}@{2}:{3}'.format(user, password, host, port),
    }
    url = 'https://www.youtube.com/user/NBA'


    def fetch_url(i,with_proxy):
        while True:
            logger.warning('thread %s fetch url'%i)
            try:
                if with_proxy:
                    res = requests.get(url,proxies=proxies, timeout=5)
                else:
                    res = requests.get(url, timeout=5)
            except Exception as e:
                logger.error(str(e))
                continue

            logger.warning(res.status_code)

    def do_other_thing():
        while True:
            logger.warning('do other thing...')
            time.sleep(1)

    @click.command()
    @click.option('--with_proxy/--without_proxy',help='if use proxy', default=True)
    def run(with_proxy):
        if with_proxy:
            logger.warning('with proxy......')
        else:
            logger.warning('without proxy......')
        ts = []
        ts.append(gevent.spawn(do_other_thing))
        for i in xrange(3):
            ts.append(gevent.spawn(fetch_url,i,with_proxy))
        gevent.joinall(ts)

    if __name__=='__main__':
        run()

这些图片显示了结果。

run with proxy

run without proxy

With proxy, do_other_thing will blocks before fetch_url done.

Without proxy， it works good. (timeout error occurs, because of the GFW)

谁能帮我解决这个问题？非常感谢！

Answer 1

我也在 github 上问过这个问题。合作者很好，帮我解决了这个问题。决心非常非常。

moving the gevent import and monkeypatch to the very top of the file before doing anything else

我也有一个包含多个文件的项目，将 import 和 monkeypatch 移动到 第一个文件 的最顶部解决了我的问题问题。

The issues at github.

使用袜子时 gevent + requests 块

gevent + requests blocks when using socks

python

socks

gevent

python-requests