如何减少 numpy 数组中的行重复次数

Question

我想清理我的数据以减少重复项的数量。我不想删除所有重复项。

如何获得具有一定数量重复项的 numpy 数组？

假设，我有

x = np.array([[1,2,3],[1,2,3],[5,5,5],[1,2,3],[1,2,3]])

并且我将重复数设置为 2。

输出应该是这样的

x
>>[[1,2,3],[1,2,3],[5,5,5]]

或

x
>>[[5,5,5],[1,2,3],[1,2,3]]

它不计入我的任务

Answer 1

尽管当您已经有了 numpy 数组时，使用列表追加作为中间步骤并不总是一个好主意，但在这种情况下，它是迄今为止最简洁的方法：

def n_uniques(arr, max_uniques):
    uniq, cnts = np.unique(arr, axis=0, return_counts=True)
    arr_list = []
    for i in range(cnts.size):
        num = cnts[i] if cnts[i] <= max_uniques else max_uniques
        arr_list.extend([uniq[i]] * num)
    return np.array(arr_list)

x = np.array([[1,2,3],
              [1,2,3],
              [1,2,3],
              [5,5,5],
              [1,2,3],
              [1,2,3],])

reduced_arr = n_uniques(x, 2)

Answer 2

这有点棘手，但您实际上可以在没有循环的情况下做到这一点，并使用类似这样的方法保留原始数组中的相对顺序（在这种情况下，第一个重复被保留）：

import numpy as np

def drop_extra_repetitions(x, max_reps):
    # Find unique rows
    uniq, idx_inv, counts = np.unique(x, axis=0, return_inverse=True, return_counts=True)
    # Compute number of repetitions of each different row
    counts_clip = np.minimum(counts, max_reps)
    # Array alternating between valid unique row indices and -1 ([0, -1, 1, -1, ...])
    idx_to_repeat = np.stack(
        [np.arange(len(uniq)), -np.ones(len(uniq), dtype=int)], axis=1).ravel()
    # Number of repetitions for each of the previous indices
    idx_repeats_clip = np.stack([counts_clip, counts - counts_clip], axis=1).ravel()
    # Valid unique row indices are repetead at most max_reps,
    # extra repetitions are filled with -1
    idx_clip_sorted = np.repeat(idx_to_repeat, idx_repeats_clip)
    # Sorter for inverse index - that is, sort the indices in the input array
    # according to their corresponding unique row index
    sorter = np.argsort(idx_inv)
    # The final inverse index is the same as the original but with -1 on extra repetitions
    idx_inv_final = np.empty(len(sorter), dtype=int)
    idx_inv_final[sorter] = idx_clip_sorted
    # Return the array reconstructed from the inverse index without the positions with -1
    return uniq[idx_inv_final[idx_inv_final >= 0]]

x = [[5, 5, 5], [1, 2, 3], [1, 2, 3], [5, 5, 5], [1, 2, 3], [1, 2, 3]]
max_reps = 2

print(drop_extra_repetitions(x, max_reps))
# [[5 5 5]
#  [1 2 3]
#  [1 2 3]
#  [5 5 5]]

如果你根本不需要保留顺序，那么你可以简单地做：

import numpy as np

def drop_extra_repetitions(x, max_reps):
    uniq, counts = np.unique(x, axis=0, return_counts=True)
    # Repeat each unique row index at most max_reps
    ret_idx = np.repeat(np.arange(len(uniq)), np.minimum(counts, max_reps))
    return uniq[ret_idx]

x = [[5, 5, 5], [1, 2, 3], [1, 2, 3], [5, 5, 5], [1, 2, 3], [1, 2, 3]]
max_reps = 2

print(drop_extra_repetitions(x, max_reps))
# [[1 2 3]
#  [1 2 3]
#  [5 5 5]
#  [5 5 5]]

如何减少 numpy 数组中的行重复次数

How to reduce the number of row repetitions in a numpy array

python

numpy

distinct

duplicates

data-analysis