如何 scatter/send 所有可能的列对到子进程并使用 python mpi4py 找到列之间的一致性？并行计算

Question

我有一个很大的 matrix/2D 数组，我需要通过并行计算在 python 中找到每个可能的列对的一致性（例如 mpi4py）。 Coherence [函数] 在各个子进程中计算，子进程应将一致性值发送到父进程，父进程将一致性值收集为列表。为此，我创建了一个小矩阵和所有可能的列对列表，如下所示：

import numpy as np
from scipy import signal
from itertools import combinations
from mpi4py import MPI


comm = MPI.COMM_WORLD
nproc = comm.Get_size()
rank = comm.Get_rank()

data=np.arange(20).reshape(5, 4)
#List of all possible column pairs
data_col = list(combinations(np.transpose(data), 2)) #list

# Function creation
def myFunc(X,Y):
..................
..................
    return Real_coh

if rank==0:
    Data= comm.scatter(data_col,root=0) #col_pair

任何人都可以建议我如何进一步进行。欢迎大家提问questions/clarifications。期待您的热心帮助。谢谢

Answer 1

检查以下脚本[与 comm.Barrier 同步。沟通]。在脚本中，我将文件作为内存效率高的 h5py 数据集块进行写入和读取。

import numpy as np
from scipy import signal
from mpi4py import MPI
import h5py as t

chunk_len = 5000 # No. of rows of a matrix
num_c = 34    # No. of column of the matrix

# Actual Dataset
data_mat = np.random.random((10000, num_c))

shape = (chunk_len, data_mat.shape[1])
chunk_size = (chunk_len, 1)
no_of_chunks = data_mat.shape[1]

with t.File('file_name.h5', 'w') as hf:
    hf.create_dataset("chunked_arr",  data=data_mat, chunks=chunk_size, compression='lzf')
del data_mat

def myFunc(dset_X, dset_Y):
    ..............
    ............
    return Real_coh

res = np.zeros((num_c, num_c))

comm = MPI.COMM_WORLD
size = comm.Get_size()
rank = comm.Get_rank()

for i in range(num_c):
    with t.File('file_name.h5', 'r', libver='latest') as hf:
        dset_X = hf['chunked_arr'][:, i]  # Chunk data reading
    if i % size == rank:
        for j in range(num_c):
            with t.File('file_name.h5', 'r', libver='latest') as hf:
                dset_Y = hf['chunked_arr'][:, j] # Chunk data reading
            res[i][j] = spac(dset_X, dset_Y)
comm.Barrier()
print('Shape of final result :', res.shape )

如何 scatter/send 所有可能的列对到子进程并使用 python mpi4py 找到列之间的一致性？并行计算

How to scatter/send all possible column pairs to the child processes and find coherence between the columns using python mpi4py? Parallel computation

python-3.x

mpi4py

numpy-slicing