在 pandas 数据框列中有效地重塑数组

Question

我有以下问题：

让我们考虑这样一个 pandas 数据框：

Width  Height  Bitmap

67     56    <1d numpy array with length 67 * 56>
59     71    <1d numpy array with length 59 * 71>
61     73    <1d numpy array with length 61 * 73>
...    ...   ...

现在，我想对位图列中的每一行应用 numpy.reshape() 函数。结果，它应该看起来像：

Width  Height  Bitmap

67     56    <2d numpy array with shape 67x56 >
59     71    <2d numpy array with shape 59x71 >
61     73    <2d numpy array with shape 61x73>
...    ...   ...

我有一个看起来像这样的工作解决方案：

for idx, bitmap in df['bitmap'].iteritems():
    df['bitmap'][idx] = np.reshape(bitmap, (df['width'][idx], df['height'][idx]))

我的位图数据框非常大（1,200,000 行），所以我想有效地应用 np.reshape()。可能吗？

Answer 1

这行得通吗？

b2 = []
Temp = df.apply(lambda x: b2.append(x.Bitmap.reshape(x.Width,x.Height)), axis=1)
df.Bitmap = b2

Answer 2

我会保留循环，但是一旦我们进入循环，就会尝试减少计算量 precomputing/storing 数组中的宽度和高度值，然后在循环内访问它们。访问数组应该会更快。此外，我们将修改形状参数，而不是在循环中重塑形状。

因此，实施将是 -

def arr1d_2D(df):
    r = df.width.values
    c = df.height.values
    n = df.shape[0]
    for i in range(n):
        df.iloc[i,2].shape = (r[i],c[i])

我们可以在这里使用所有 NumPy 来处理 bitmap 列的基础数据，这应该快得多 -

def arr1d_2D_allNumPy(df):
    r = df.width.values
    c = df.height.values
    n = df.shape[0]
    b = df['bitmap'].values
    for i in range(n):
        b[i].shape = (r[i],c[i])

样本运行-

In [9]: df
Out[9]: 
   width  height                                bitmap
0      3       2                    [0, 1, 7, 4, 8, 1]
1      2       2                          [7, 3, 8, 6]
2      2       4              [6, 8, 6, 4, 7, 0, 6, 2]
3      4       3  [8, 6, 5, 2, 2, 2, 4, 3, 3, 3, 1, 8]
4      4       3  [3, 8, 4, 8, 6, 4, 2, 3, 8, 7, 7, 4]

In [10]: arr1d_2D_allNumPy(df)

In [11]: df
Out[11]: 
   width  height                                        bitmap
0      3       2                      [[0, 1], [7, 4], [8, 1]]
1      2       2                              [[7, 3], [8, 6]]
2      2       4                  [[6, 8, 6, 4], [7, 0, 6, 2]]
3      4       3  [[8, 6, 5], [2, 2, 2], [4, 3, 3], [3, 1, 8]]
4      4       3  [[3, 8, 4], [8, 6, 4], [2, 3, 8], [7, 7, 4]]

运行时测试

接近 -

def org_app(df):   # Original approach
    for idx, bitmap in df['bitmap'].iteritems():
        df['bitmap'][idx] = np.reshape(bitmap, (df['width'][idx], \
                                                df['height'][idx]))

计时 -

In [43]: # Setup input dataframe and two copies for testing
    ...: a = np.random.randint(1,5,(1000,2))
    ...: df = pd.DataFrame(a, columns=(('width','height')))
    ...: n = df.shape[0]
    ...: randi = np.random.randint
    ...: df['bitmap'] = [randi(0,9,(df.iloc[i,0]*df.iloc[i,1])) for i in range(n)]
    ...: 
    ...: df_copy1 = df.copy()
    ...: df_copy2 = df.copy()
    ...: df_copy3 = df.copy()
    ...: 

In [44]: %timeit org_app(df_copy1)
1 loops, best of 3: 26 s per loop

In [45]: %timeit arr1d_2D(df_copy2)
10 loops, best of 3: 115 ms per loop

In [46]: %timeit arr1d_2D_allNumPy(df_copy3)
1000 loops, best of 3: 475 µs per loop

In [47]: 26000000/475.0  # Speedup with allNumPy version over original
Out[47]: 54736.84210526316

疯狂 50,000x+ 加速，只是为了展示访问数据的更好方法，特别是 pandas 数据帧中的数组数据。

在 pandas 数据框列中有效地重塑数组

Efficiently reshaping arrays in a pandas dataframe column

python

performance

numpy

reshape

pandas