Python: memmap 对象列表在 joblib parallel 中变为 'None' 类型

Question

我正在做以下事情：

我有一个 tensorflow DNN 层列表。 nn.append(tf.layers.dense(...))
上面的每个列表都附加到 np.memmap 个对象的列表中。 nnList[i] = nn
我可以访问内存映射列表并检索张量。但是当尝试访问 joblib.parallel 内部的张量时，它是 returns 'None' 类型的对象。但是，joblib.parallel.

我在下面附上了示例代码。

    import os
    import tempfile
    import numpy as np
    import tensorflow as tf
    from joblib import Parallel, delayed, load, dump

    tmpFolder = tempfile.mkdtemp()
    __nnFile = os.path.join(tmpFolder, 'nn.mmap')
    nnList = np.memmap(__nnFile, dtype=object, mode='w+', shape=(5))

    def main():
        for i in range(5):
            nn = []
            input = tf.placeholder(dtype=tf.float32, shape=(1, 8))
            nn.append(tf.layers.dense(inputs=input, units=8, activation=tf.sigmoid,  
                                        trainable=False))
            nn.append(tf.layers.dense(inputs=nn[0], units=2, activation=tf.sigmoid,  
                                        trainable=False))

            nnList[i] = nn

        print('nnList: ' + str(len(nnList)))
        for i in range(5):
            nn = nnList[i]
            print(nn)
            print(nn[-1])
            print('---------------------------  ' + str(i))

        with Parallel(n_jobs = -1) as parallel:
            parallel(delayed(func1)(i) for i in range(5))

    def func1(i):
        print('nnList: ' + str(len(nnList)))
        for x in range(5):
            nn = nnList[x]
            print(nn)
            print('---------------------------  ' + str(x))

    if __name__ == '__main__':
        main()

以上代码给出了这个输出。注意数组的长度以及张量如何变为 None.

    nnList: 5
    [<tf.Tensor 'dense/Sigmoid:0' shape=(1, 8) dtype=float32>, <tf.Tensor 'dense_1/Sigmoid:0' shape=(1, 2) dtype=float32>]
    Tensor("dense_1/Sigmoid:0", shape=(1, 2), dtype=float32)
    ---------------------------  0
    [<tf.Tensor 'dense_2/Sigmoid:0' shape=(1, 8) dtype=float32>, <tf.Tensor 'dense_3/Sigmoid:0' shape=(1, 2) dtype=float32>]
    Tensor("dense_3/Sigmoid:0", shape=(1, 2), dtype=float32)
    ---------------------------  1
    [<tf.Tensor 'dense_4/Sigmoid:0' shape=(1, 8) dtype=float32>, <tf.Tensor 'dense_5/Sigmoid:0' shape=(1, 2) dtype=float32>]
    Tensor("dense_5/Sigmoid:0", shape=(1, 2), dtype=float32)
    ---------------------------  2
    [<tf.Tensor 'dense_6/Sigmoid:0' shape=(1, 8) dtype=float32>, <tf.Tensor 'dense_7/Sigmoid:0' shape=(1, 2) dtype=float32>]
    Tensor("dense_7/Sigmoid:0", shape=(1, 2), dtype=float32)
    ---------------------------  3
    [<tf.Tensor 'dense_8/Sigmoid:0' shape=(1, 8) dtype=float32>, <tf.Tensor 'dense_9/Sigmoid:0' shape=(1, 2) dtype=float32>]
    Tensor("dense_9/Sigmoid:0", shape=(1, 2), dtype=float32)
    ---------------------------  4
    nnList: 5
    None
    ---------------------------  0
    None
    ---------------------------  1
    None
    ---------------------------  2
    None
    ---------------------------  3
    None
    ---------------------------  4

如何访问 joblib.parallel 中的张量？请帮忙

Answer 1

当时发现问题。希望对以后的人有帮助。

None 问题与张量无关。我错误地使用了 joblib.Parallel 函数。

应该将变量传递给 delayed 以便分叉进程可以访问（我怎么在文档中忽略了这一点！）。正确方法：

with Parallel(n_jobs = -1) as parallel:
    parallel(delayed(func1)(i, WHATEVER_VARIABLE_I_WANT) for i in range(5))

Python: memmap 对象列表在 joblib parallel 中变为 'None' 类型

Python: memmap list of objects become 'None' type inside joblib parallel

python-3.x

tensorflow

nonetype

joblib

numpy-memmap