python 屏蔽数据框中的每一天

Question

我必须对数据框进行每日求和，但前提是至少 70% 的每日数据不是 NaN。如果是，那么这一天就不必考虑了。有没有办法制作这样的面具？我的数据框是超过 17 年的每小时数据。

我的数据是这样的：

                    clear skies  all skies     Lab
2015-02-26 13:00:00     597.5259   376.1830  307.62
2015-02-26 14:00:00     461.2014   244.0453  199.94
2015-02-26 15:00:00     283.9003   166.5772  107.84
2015-02-26 16:00:00      93.5099    50.7761   23.27
2015-02-26 17:00:00       1.1559     0.2784    0.91
                         ...        ...     ...
2015-12-05 07:00:00      95.0285    29.1006   45.23
2015-12-05 08:00:00     241.8822   120.1049  113.41
2015-12-05 09:00:00     363.8040   196.0568  244.78
2015-12-05 10:00:00     438.2264   274.3733  461.28
2015-12-05 11:00:00     456.3396   330.6650  447.15

如果我分组并汇总，则无法知道在任何一天是否缺少数据，有些日子的总和会较低，因此会降低我的月均值

Answer 1

如评论中所述，使用groupby按日期对数据进行分组，然后编写适当的选择。这是一个将所有天数相加的示例（假设常规数据点，每天 24 个）少于 nan 个条目的 50%：

import pandas as pd
import numpy as np

# create a date range
date_rng = pd.date_range(start='1/1/2018', end='1/1/2021', freq='H')

# create random data
df = pd.DataFrame({"data":np.random.randint(0,100,size=(len(date_rng)))}, index = date_rng)

# set some values to nan
df["data"][df["data"] > 50] = np.nan

# looks like this
df.head(20)

# sum everything where less than 50% are nan
df.groupby(df.index.date).sum()[df.isna().groupby(df.index.date).sum() < 12]

示例输出：

            data
2018-01-01    NaN
2018-01-02    NaN
2018-01-03  487.0
2018-01-04    NaN
2018-01-05  421.0
...           ...
2020-12-28    NaN
2020-12-29    NaN
2020-12-30    NaN
2020-12-31  392.0
2021-01-01    0.0

Answer 2

替代解决方案 - 您可能会发现它有用且灵活：

# pip install convtools
from convtools import conversion as c

total_number = c.ReduceFuncs.Count()
total_not_none = c.ReduceFuncs.Count(where=c.item("amount").is_not(None))
total_sum = c.ReduceFuncs.Sum(c.item("amount"))

input_data = []  # e.g. iterable of dicts
converter = (
    c.group_by(
        c.item("key1"),
        c.item("key2"),
    )
    .aggregate(
        {
            "key1": c.item("key1"),
            "key2": c.item("key2"),
            "sum_if_70": c.if_(
                total_not_none / total_number < 0.7,
                None,
                total_sum,
            ),
        }
    )
    .gen_converter(
        debug=False
    )  # install black and set to True to see the generated ad-hoc code
)
result = converter(input_data)

python 屏蔽数据框中的每一天

python masking each day in dataframe

python

dataframe

mask