gevent 中的 ThreadPool 异常不会返回

Question

我在 Python 2.7 上使用 gevent 来做一些多线程工作。但是，我无法捕获生成方法中引发的异常。我在 returned AsyncResult 对象上使用 .get() 方法（在线程池上调用 .spawn() 方法时 returned）。

根据 gevent 文档：http://www.gevent.org/gevent.event.html#gevent.event.AsyncResult.get:

get(block=True, timeout=None)

Return the stored value or raise the exception.

If this instance already holds a value or an exception, return or raise it immediately.

但是，没有 return 异常，.get() returns NoneType。此外，在控制台中，打印出异常详细信息。相反，.get() 应该 return 返回异常。

下面是一些示例代码和相应的输出，展示了这一点：

from gevent import threadpool

def this_will_fail():
    raise Exception('This will fail')

result_list = []
for i in range(1,5):
    tpool = threadpool.ThreadPool(5)
    result_list.append(tpool.spawn(this_will_fail))

tpool.join()

for result in result_list:
    try:
        returned_object = result.get()
        print type(returned_object)

    except Exception as e:
        print 'This is not running for some reason...'

输出：

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/gevent/threadpool.py", line 193, in _worker
    value = func(*args, **kwargs)
  File "<stdin>", line 2, in this_will_fail
Exception: This will fail
(<ThreadPool at 0x107c79e50 0/1/5>, <function this_will_fail at 0x107c848c0>) failed with Exception

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/gevent/threadpool.py", line 193, in _worker
    value = func(*args, **kwargs)
  File "<stdin>", line 2, in this_will_fail
Exception: This will fail
(<ThreadPool at 0x107c99210 0/1/5>, <function this_will_fail at 0x107c848c0>) failed with Exception

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/gevent/threadpool.py", line 193, in _worker
    value = func(*args, **kwargs)
  File "<stdin>", line 2, in this_will_fail
Exception: This will fail
(<ThreadPool at 0x107c99410 0/1/5>, <function this_will_fail at 0x107c848c0>) failed with Exception

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/gevent/threadpool.py", line 193, in _worker
    value = func(*args, **kwargs)
  File "<stdin>", line 2, in this_will_fail
Exception: This will fail
(<ThreadPool at 0x107c99610 0/1/5>, <function this_will_fail at 0x107c848c0>) failed with Exception

<type 'NoneType'>
<type 'NoneType'>
<type 'NoneType'>
<type 'NoneType'>

我是否遗漏了一些明显的东西，或者这是 gevent 中的错误？

编辑： concurrent.futures 没有这个问题。虽然这对我的用例来说是一个可以接受的解决方案，但我仍然想了解为什么 gevent 没有正确 return 异常。

Answer 1

在这里，我们遇到了与 gevent 线程池相同的问题。我们正在使用一种解决方案，将函数包装到 return 异常作为结果，并在我们想要使用 AsynResult 的结果时再次引发它。

class _ExceptionWrapper:
    def __init__(self, exception, error_string, tb):
        self.exception = exception
        self.error_string = error_string
        self.tb = tb

class wrap_errors(object):
    def __init__(self, func):
        self.func = func

    def __call__(self, *args, **kwargs):
        func = self.func
        try:
            return func(*args, **kwargs)
        except:
            return _ExceptionWrapper(*sys.exc_info())

    def __str__(self):
        return str(self.func)

    def __repr__(self):
        return repr(self.func)

    def __getattr__(self, item):
        return getattr(self.func, item)

def get_with_exception(g, block=True, timeout=None):
    result = g._get(block, timeout)
    if isinstance(result, _ExceptionWrapper):
        # raise the exception using the caller context
        raise result.error_string, None, result.tb
    else:
        return result

def spawn(fn, *args, **kwargs):
    # wrap the function
    fn = wrap_errors(fn)
    g = threadpool.spawn(fn, *args, **kwargs)
    # and the asynresult
    g._get = g.get
    g.get = types.MethodType(get_with_exception, g)
    return g

gevent 中的 ThreadPool 异常不会返回

ThreadPool exceptions in gevent do not get returned

python

multithreading

exception-handling

exception

gevent