Python multiprocessing+savefig 导致错误或系统死机

Question

我有一个大的 3d numpy 数组，我想将每个切片（2d 数组）写成一个类似 imshow 的图形（即值的热图）。作为一个具体的例子，假设数组的形状是 3x3x3000，所以我想要 3000 张图像，每张图像代表一个 3x3 矩阵。用一个线程循环它有点慢。由于迭代是完全独立的，我想使用多处理模块来加快速度。代码如下

def write_tensor_image(t_slice_wrapper):

    idx = t_slice_wrapper['idx']
    t_slice = t_slice_wrapper['t_slice']
    folder_path=t_slice_wrapper['folder_path']

    fig = matplotlib.pyplot.figure()
    ax = fig.add_subplot(111)    
    ax.imshow(t_slice,interpolation='none')
    fig.tight_layout()

    fname_ = os.path.join(folder_path,'tmp_%s.png'%str(idx))
    fig.savefig(fname_, bbox_inches="tight")    

def write_tensor_image_sequence(tensor, folder_path='/home/foo/numpy_cache'):

    os.system('mkdir -p %s'%folder_path)
    os.system('rm -rf %s/*'%folder_path)

    slices = [None]*tensor.shape[2]
    for i in range(0,tensor.shape[2]):
        slices[i] = {'t_slice':tensor[:,:,i], 'idx':i, 'folder_path':folder_path}

    pool = multiprocessing.Pool(processes=4)
    pool.map(write_tensor_image, slices)
    pool.close()
    pool.join()

但是这不起作用 - 单线程情况下工作正常（只需在 for 循环中调用 write_tensor_image()）但使用池要么导致机器完全锁定，要么给出如下内容错误：

XIO:  fatal IO error 11 (Resource temporarily unavailable) on X server ":0"
  after 849 requests (849 known processed) with 28 events remaining.
XIO:  fatal IO error 11 (Resource temporarily unavailable) on X server ":0"
XIO:  fatal IO error 11 (Resource temporarily unavailable) on X server ":0"
  after 849 requests (849 known processed) with 28 events remaining.
  after 849 requests (849 known processed) with 28 events remaining.
X Error of failed request:  BadPixmap (invalid Pixmap parameter)
Major opcode of failed request:  54 (X_FreePixmap)
Resource id in failed request:  0x4e0001e
Serial number of failed request:  851
Current serial number in output stream:  851

我认为我走在正确的轨道上（根据 How to fix the python multiprocessing matplotlib savefig() issue? and Matplotlib: simultaneous plotting in multiple threads），但我一定遗漏了一些东西。

Answer 1

在脚本的开头做一个matplotlib.use('agg')。 Matplotlib 似乎试图在每个子进程中建立一个 GUI，这相互冲突。

更一般地说，如果您不进行标准交互式绘图，您可能不想使用 pyplot 接口，而是使用 OOP 接口。

Python multiprocessing+savefig 导致错误或系统死机

Python multiprocessing+savefig leads to error or system lockup

python

numpy

matplotlib

figure

python-multiprocessing