向量化创建组合 DataFrame 的最快方法是什么？

Question

如何通过矢量化过程创建 master DataFrame？如果不可能，执行此操作最省时（不关心内存）的方法是什么？

可以用更高效的东西代替 for 循环吗？

如您所见，combinations 非常快速地生成非常大的数字，因此我需要一种快速的方法来生成此 DataFrame。

请参阅下面的最小可重现示例：

%%time

import pandas as pd
import string
import numpy as np
from itertools import combinations

# create dummy data
cols = list(string.ascii_uppercase)
dummy = pd.DataFrame()
for col in cols:
    dummy = dummy.append([[col, 0] + np.random.randint(2, 100, size=(1, 10)).tolist()[0]])
    dummy = dummy.append([[col, 1] + np.random.randint(2, 100, size=(1, 10)).tolist()[0]])
    dummy = dummy.append([[col, 2] + np.random.randint(2, 100, size=(1, 10)).tolist()[0]])
dummy.columns=['name', 'id', 'v1', 'v2', 'v3', 'v4', 'v5', 'v1', 'v6', 'v7', 'v8', 'v9']

# create all possible unique combinations
combos = list(combinations(cols, 2))

# generate DataFrame with all combinations
master = pd.DataFrame()
for i, combo in enumerate(combos):
    A = dummy[dummy.name == combo[0]]
    B = dummy[dummy.name == combo[1]]
    joined = pd.merge(A, B, on=["id"], suffixes=('_A', '_B'))
    joined = joined.sort_values("id")
    joined['pair_id'] = i
    master = pd.concat([master, joined])

输出：

CPU times: total: 1.8 s
Wall time: 1.8 s

谢谢！

Answer 1

由于您的数据是结构化的，因此您可以使用 numpy 来利用向量化操作。

names = list(string.ascii_uppercase)
ids = [0, 1, 2]
columns = pd.Series(["v1", "v2", "v3", "v4", "v5", "v1", "v6", "v7", "v8", "v9"])

# Generate the random data
data = np.random.randint(2, 100, (len(names), len(ids), len(columns)))

# Pair data for every 2-combination of names
arr = [np.hstack([data[i], data[j]]) for i,j in combinations(range(len(names)), 2)]

# Assembling the data to final dataframe
idx = pd.MultiIndex.from_tuples([
    (p,a,b,i) for p, (a, b) in enumerate(combinations(names,2)) for i in ids
], names=["pair_id", "name_A", "name_B", "id"])
cols = pd.concat([columns + "_A", columns + "_B"])

master = pd.DataFrame(np.vstack(arr), index=idx, columns=cols)

原码：4s。新代码：7ms

向量化创建组合 DataFrame 的最快方法是什么？

What's the fastest way to vectorise creation of combinatorial DataFrame?

python

performance

profiling

vectorization

pandas