Interleave/interweave 懒惰地处理数组

Question

我需要逐帧交错两个大型 HDF5 数据集，这些数据集表示来自显微测量的两个通道的视频帧。我认为 Dask 适合这项工作和下游流程。

这两个数组具有相同的形状和数据类型。基于这个 link，我可以用 NumPy 来做小于内存数组的事情： Interweaving two numpy arrays.

import numpy as np
# a numpy example of channel 1 data
ch1 = np.arange(1,5)[:,np.newaxis,np.newaxis]*np.ones((4,3,2))

# channel 2 has the same shape and dtype
ch2 = np.arange(10,50,10)[:,np.newaxis,np.newaxis]*np.ones((4,3,2))

# the interleaving starts with assigning a new array with douled size of the first dimension
ch1_2 = np.empty((2*ch1.shape[0],*ch1.shape[1:]), dtype=ch1.dtype)
# two assignments takes care of the interleaving 
ch1_2[0::2] = ch1
ch1_2[1::2] = ch2

很遗憾，它不适用于 Dask。

import dask.array as da
da_ch1 = da.from_array(ch1)
da_ch2 = da.from_array(ch2)
da_ch1_2 = da.empty((2*da_ch1.shape[0],*da_ch1.shape[1:]), dtype=da_ch1.dtype)
da_ch1_2[0::2] = da_ch1
da_ch1_2[1::2] = da_ch2

它失败了：“不支持的项目分配”。

任何人都可以用兼容 Dask 的替代方法帮助我吗？任何帮助将不胜感激。

Answer 1

下面的代码适用于您发布的小示例数据。您可能还必须准备一个类似的延迟函数来读取您的 hdf5 数据。

import dask.array as da
from dask import delayed
import numpy as np

@delayed
def interleave(x1, x2):
    x1_2 = np.empty(ch1_2_shape, dtype=ch1.dtype)
    x1_2[0::2] = x1
    x1_2[1::2] = x2
    return x1_2

# a numpy example of channel 1 data
ch1 = np.arange(1,5)[:,np.newaxis,np.newaxis]*np.ones((4,3,2))

# channel 2 has the same shape and dtype
ch2 = np.arange(10,50,10)[:,np.newaxis,np.newaxis]*np.ones((4,3,2))

# Interleave using dask delayed
ch1_2_shape = (2*ch1.shape[0],*ch1.shape[1:])
ch1_2 = interleave(ch1, ch2)

# Convert to dask array if required
ch1_2 = da.from_delayed(interleave(ch1, ch2), ch1_2_shape, dtype=ch1.dtype)

ch1_2.compute()

Answer 2

这是针对以下问题的高级 Dask Array 解决方案：

da_ch1_2=da.rollaxis(da.stack((da_ch1,da_ch2)),axis=1).reshape((-1,*da_ch1.shape[1:]))

Interleave/interweave 懒惰地处理数组

Interleave/interweave Dask Arrays lazily

python

arrays

dask

interleave