从 C 并行调用 python

Question

我需要从我的 C 代码中调用一个 Python 函数。它工作得很好，但是当我想进行并行化时，它就崩溃了。请参阅以下最小 C 代码：

#include <Python.h>
#include <stdio.h>

int main(void)
{
  double Z = 1.;
  double k = 1.;
  double l = 1.;
  double eta = -Z/k;

  Py_Initialize();

  PyObject* pName = PyString_FromString("mpmath");
  PyObject* pModule = PyImport_Import(pName);
  PyObject* pFunc = PyObject_GetAttrString(pModule, "coulombf");

  PyObject* pl = PyFloat_FromDouble(l);
  PyObject* peta = PyFloat_FromDouble(eta);

  int i;
#pragma omp parallel for private(i)
  for(i=0; i<10000; i++)
  {
    double r = 0.01*i;
    PyObject* prho = PyFloat_FromDouble(k*r);
    PyObject* pArgs = PyTuple_Pack(3, pl, peta, prho);
    PyObject* pValue = PyObject_CallObject(pFunc, pArgs);
    double value = PyFloat_AsDouble(pValue);
    printf("r=%.2f\tf=%.6f\n",r,value);
  }

  Py_Finalize();
  return 0;
}

我们把这个文件命名为testPython.c，你可以用gcc -fopenmp testPython.c -o testPython -I/usr/include/python2.7 -L/usr/lib64/python2.7/config -lpython2.7编译它。

现在运行它与 ./testPython，你会看到这样的错误：Fatal Python error: GC object already tracked。（有时，错误信息会有所不同。）

但是如果你编译它时留下 -fopenmp，程序就可以完美运行。

我该如何克服这个问题？谢谢！

编辑：

正如 Natecat、John Bollinger 和 Olaf 的回答，多线程不太可能大大加快处理速度，但多处理确实可以加快计算速度。纯 python 脚本如下所示：

import numpy
from mpmath import coulombf
from multiprocessing import Pool

Z = 1.
k = 1.
l = 1.
eta = -Z/k

def coulombF(r):
    return coulombf(l,eta,k*r)

pool = Pool(12)
result = pool.map_async(coulombF, numpy.arange(0.,100.,0.01))
print(result.get())

但我如何在 C 中做到这一点？我还没找到路

Answer 1

任何类型的真正多线程（例如，在一个进程中使用多个系统线程）在 python 中是不可能的，至少在最常见的 python 实现中是不可能的。您可以不使用任何类型的并行化，也可以切换到没有 GIL 的实现。这是一篇包含有关该主题的更多信息的文章：https://wiki.python.org/moin/GlobalInterpreterLock

Answer 2

@Natecat 的回答基本上是正确的，只是缺少细节和细微差别。 The docs of Python's C API给个更全的图。假设这是您正在使用的 Python 实现，您需要了解以下内容：

The Python interpreter is not fully thread-safe. In order to support multi-threaded Python programs, there’s a global lock, called the global interpreter lock or GIL, that must be held by the current thread before it can safely access Python objects. Without the lock, even the simplest operations could cause problems in a multi-threaded program [...].

Therefore, the rule exists that only the thread that has acquired the GIL may operate on Python objects or call Python/C API functions. In order to emulate concurrency of execution, the interpreter regularly tries to switch threads (see sys.setswitchinterval()). The lock is also released around potentially blocking I/O operations like reading or writing a file, so that other Python threads can run in the meantime.

和

when threads are created from C (for example by a third-party library with its own thread management), they don’t hold the GIL, nor is there a thread state structure for them.

注意：OpenMP 正是这种情况。

If you need to call Python code from these threads [...] you must first register these threads with the interpreter by creating a thread state data structure, then acquiring the GIL, and finally storing their thread state pointer, before you can start using the Python/C API. When you are done, you should reset the thread state pointer, release the GIL, and finally free the thread state data structure.

The PyGILState_Ensure() and PyGILState_Release() functions do all of the above automatically. The typical idiom for calling into Python from a C thread is:
PyGILState_STATE gstate;
gstate = PyGILState_Ensure();

/* Perform Python actions here. */
result = CallSomeFunction();
/* evaluate result or handle exception */

/* Release the thread. No Python API allowed beyond this point. */
PyGILState_Release(gstate);

您必须实现该模式以允许多个 OpenMP 线程安全地对同一个 CPython 解释器进行并发调用，但您不太可能从中获益并行化，因为各种 OpenMP 线程将在很大程度上被阻止并发运行。

从 C 并行调用 python

Call python in parallel from C

c

python

parallel-processing

openmp

python-c-api