如何使用自适应间隔对 pandas 中的偏斜数据进行分组

Question

假设我的数据框中的一列包含此频率的数据：

>>> vals = list(range(11000,12000)) + list(range(5600,6120)) + list(range(0,40,4)) + \
       list(range(0,10000,300)) + list(range(1200,1400,3)) + list(range(0,10000,1100))
>>> df = pd.DataFrame({'freq' : vals})

我想看看他们的频率分布。我现在所做的只是，

>>> df.freq.value_counts(bins=20).sort_index()

(-12.0, 599.95]         13
(599.95, 1199.9]         3
(1199.9, 1799.85]       69
(1799.85, 2399.8]        3
(2399.8, 2999.75]        2
(2999.75, 3599.7]        3
(3599.7, 4199.65]        2
(4199.65, 4799.6]        3
(4799.6, 5399.55]        2
(5399.55, 5999.5]      403
(5999.5, 6599.45]      122
(6599.45, 7199.4]        3
(7199.4, 7799.35]        3
(7799.35, 8399.3]        2
(8399.3, 8999.25]        3
(8999.25, 9599.2]        2
(9599.2, 10199.15]       3
(10199.15, 10799.1]      0
(10799.1, 11399.05]    400
(11399.05, 11999.0]    600
Name: freq, dtype: int64

但是如您所见，它没有任何智能。有很多频率非常少的垃圾箱。如果它们低于特定阈值（例如 5），我想让它们合并。所以我想要的是：

(-12.0, 599.95]         13
(599.95, 1199.9]         3
(1199.9, 1799.85]       69
(1799.85, 5399.55]      15
(5399.55, 5999.5]      403
(5999.5, 6599.45]      122
(6599.45, 10799.1]      16
(10799.1, 11399.05]    400
(11399.05, 11999.0]    600

我想不出任何合适的东西，因为我对间隔感到不舒服。另外，如果有人可以建议一些更好的方法来获得具有智能间距的频率分布，那也很好。

注意：我不是在寻找垃圾箱数量的操纵，因为那必须是手动的，我想避免这种情况。

Answer 1

你可以试试qcut:

pd.qcut(df.freq, q=20).value_counts()

输出：

(-0.001, 1395.0]      83
(11835.0, 11917.0]    82
(1395.0, 5662.0]      82
(5662.0, 5743.0]      82
(5743.0, 5825.0]      82
(5825.0, 5907.0]      82
(5907.0, 5989.0]      82
(5989.0, 6070.0]      82
(6070.0, 11015.0]     82
(11015.0, 11097.0]    82
(11917.0, 11999.0]    82
(11179.0, 11261.0]    82
(11261.0, 11343.0]    82
(11343.0, 11425.0]    82
(11425.0, 11507.0]    82
(11507.0, 11589.0]    82
(11589.0, 11671.0]    82
(11671.0, 11753.0]    82
(11753.0, 11835.0]    82
(11097.0, 11179.0]    82
Name: freq, dtype: int64

Answer 2

您可以使用quantile()学习如何将物品平均分配到不同的篮子，例如：

>>> df.freq.quantile(0.9) # 90% of values are <= 11835
11835.0
>>> df.freq.quantile(0.5) # 50% of values are <= 11179
11179.0
>>> df.freq.quantile(0.2)  # 20% of values are <= 5825
5825.0
>>> df.freq.quantile(0.1)
5662.0

这些是我们平均分配篮子的价值

>>> df[df.freq < 5662].shape[0]
164
>>> df[(df.freq >= 5662) & (df.freq < 5825)].shape[0]
164

如何使用自适应间隔对 pandas 中的偏斜数据进行分组

How to group skewed data in pandas with adaptive intervals

python

pandas

group-by

intervals

frequency-distribution