如何从提升线程调用 Python？

Question

我有一个调用 C++ boost python 库的 Python 应用程序，它一切正常。但是，我有一个回调 C++ 到 Python 的场景，其中来自 boost 线程的 C++ 调用 python 并且我在 C++ 端遇到访问冲突。如果我使用 python 线程执行完全相同的回调，它会完美运行。因此，我怀疑我不能简单地使用 boost 线程从 C++ 回调 Python，而是需要做一些额外的事情才能让它工作？

Answer 1

最有可能的罪魁祸首是 Global Interpreter Lock (GIL) 在调用 Python 代码时未被线程占用，从而导致未定义的行为。验证所有进行直接或间接 Python 调用的路径，在调用 Python 代码之前获取 GIL。

GIL 是围绕 CPython 解释器的互斥体。此互斥锁可防止对 Python 个对象执行并行操作。因此，在任何时间点，最多允许一个线程（已获得 GIL 的线程）对 Python 个对象执行操作。当存在多个线程时，在不持有 GIL 的情况下调用 Python 代码会导致未定义的行为。

C 或 C++ 线程有时在 Python 文档中称为外来线程。 Python 解释器没有能力控制外来线程。因此，外来线程负责管理 GIL 以允许与 Python 个线程并发或并行执行。必须慎重考虑：

堆栈展开，因为Boost.Python可能会抛出异常。
间接调用 Python，例如复制构造函数或析构函数

一种解决方案是使用可识别 GIL 管理的自定义类型包装 Python 回调。

使用 RAII-style class 来管理 GIL 提供了一个优雅的异常安全解决方案。例如，对于以下 with_gil class，当创建 with_gil 对象时，调用线程获取 GIL。当 with_gil 对象被销毁时，它恢复 GIL 状态。

/// @brief Guard that will acquire the GIL upon construction, and
///        restore its state upon destruction.
class with_gil
{
public:
  with_gil()  { state_ = PyGILState_Ensure(); }
  ~with_gil() { PyGILState_Release(state_);   }

  with_gil(const with_gil&)            = delete;
  with_gil& operator=(const with_gil&) = delete;
private:
  PyGILState_STATE state_;
};

及其用法：

{
  with_gil gil;                      // Acquire GIL.
  // perform Python calls, may throw
}                                    // Restore GIL.

在能够通过 with_gil 管理 GIL 之后，下一步是创建一个能够正确管理 GIL 的仿函数。以下 py_callable class 将包装一个 boost::python::object 并为调用 Python 代码的所有路径获取 GIL：

/// @brief Helper type that will manage the GIL for a python callback.
///
/// @detail GIL management:
///           * Acquire the GIL when copying the `boost::python` object
///           * The newly constructed `python::object` will be managed
///             by a `shared_ptr`.  Thus, it may be copied without owning
///             the GIL.  However, a custom deleter will acquire the
///             GIL during deletion
///           * When `py_callable` is invoked (operator()), it will acquire
///             the GIL then delegate to the managed `python::object`
class py_callable
{
public:

  /// @brief Constructor that assumes the caller has the GIL locked.
  py_callable(const boost::python::object& object)
  {
    with_gil gil;
    object_.reset(
      // GIL locked, so it is safe to copy.
      new boost::python::object{object},
      // Use a custom deleter to hold GIL when the object is deleted.
      [](boost::python::object* object)
      {
        with_gil gil;
        delete object;
      });
  }

  // Use default copy-constructor and assignment-operator.
  py_callable(const py_callable&) = default;
  py_callable& operator=(const py_callable&) = default;

  template <typename ...Args>
  void operator()(Args... args)
  {
    // Lock the GIL as the python object is going to be invoked.
    with_gil gil;
    (*object_)(std::forward<Args>(args)...);
  }

private:
  std::shared_ptr<boost::python::object> object_;
};

通过在 free-space 上管理 boost::python::object，可以自由复制 shared_ptr 而无需持有 GIL。这允许我们安全地使用默认生成的复制构造函数、赋值运算符、析构函数等

可以按如下方式使用 py_callable：

// thread 1
boost::python::object object = ...; // GIL must be held.
py_callable callback(object);       // GIL no longer required.
work_queue.post(callback);

// thread 2
auto callback = work_queue.pop();   // GIL not required.
// Invoke the callback.  If callback is `py_callable`, then it will
// acquire the GIL, invoke the wrapped `object`, then release the GIL.
callback(...);

这是一个完整的示例 demonstrating，它具有 Python 扩展调用 Python 对象作为 C++ 线程的回调：

#include <memory>  // std::shared_ptr
#include <thread>  // std::this_thread, std::thread
#include <utility> // std::forward
#include <boost/python.hpp>

/// @brief Guard that will acquire the GIL upon construction, and
///        restore its state upon destruction.
class with_gil
{
public:
  with_gil()  { state_ = PyGILState_Ensure(); }
  ~with_gil() { PyGILState_Release(state_);   }

  with_gil(const with_gil&)            = delete;
  with_gil& operator=(const with_gil&) = delete;
private:
  PyGILState_STATE state_;
};

/// @brief Helper type that will manage the GIL for a python callback.
///
/// @detail GIL management:
///           * Acquire the GIL when copying the `boost::python` object
///           * The newly constructed `python::object` will be managed
///             by a `shared_ptr`.  Thus, it may be copied without owning
///             the GIL.  However, a custom deleter will acquire the
///             GIL during deletion
///           * When `py_callable` is invoked (operator()), it will acquire
///             the GIL then delegate to the managed `python::object`
class py_callable
{
public:

  /// @brief Constructor that assumes the caller has the GIL locked.
  py_callable(const boost::python::object& object)
  {
    with_gil gil;
    object_.reset(
      // GIL locked, so it is safe to copy.
      new boost::python::object{object},
      // Use a custom deleter to hold GIL when the object is deleted.
      [](boost::python::object* object)
      {
        with_gil gil;
        delete object;
      });
  }

  // Use default copy-constructor and assignment-operator.
  py_callable(const py_callable&) = default;
  py_callable& operator=(const py_callable&) = default;

  template <typename ...Args>
  void operator()(Args... args)
  {
    // Lock the GIL as the python object is going to be invoked.
    with_gil gil;
    (*object_)(std::forward<Args>(args)...);
  }

private:
  std::shared_ptr<boost::python::object> object_;
};

BOOST_PYTHON_MODULE(example)
{
  // Force the GIL to be created and initialized.  The current caller will
  // own the GIL.
  PyEval_InitThreads();

  namespace python = boost::python;
  python::def("call_later",
    +[](int delay, python::object object) {
      // Create a thread that will invoke the callback.
      std::thread thread(+[](int delay, py_callable callback) {
        std::this_thread::sleep_for(std::chrono::seconds(delay));
        callback("spam");
      }, delay, py_callable{object});
      // Detach from the thread, allowing caller to return.
      thread.detach();
  });
}

交互使用：

>>> import time
>>> import example
>>> def shout(message):
...     print message.upper()
...
>>> example.call_later(1, shout)
>>> print "sleeping"; time.sleep(3); print "done sleeping"
sleeping
SPAM
done sleeping

如何从提升线程调用 Python？

How to call Python from a boost thread?

c++

python

multithreading

boost-thread

boost-python