删除 X 天内未触及的 AWS 日志组

Question

有没有办法删除过去 30 天内没有任何写入的所有 AWS 日志组？

或者相反，获取在过去 30 天内没有写入任何内容的日志组的列表？

Answer 1

我不知道执行此操作的简单方法，但您可以使用 awscli（或者最好是 python/boto3）到 describe-log-groups，然后为每个日志组调用 describe-log-streams，然后对于每个日志 group/stream 对，使用 30 天前的 --start-time 调用 get-log-events。如果日志组的所有事件数组的并集为空，那么您知道可以删除日志组。

Answer 2

我一直在使用aws-cloudwatch-log-clean，可以说效果很好。

您需要安装 boto3，然后您：

./sweep_log_streams.py [log_group_name]

它有一个 --dry-run 选项供您首先检查它的预期内容。

请注意，如果您在 ECS 中有一个很长的运行进程，它在日志上是安静的，并且由于日志保留期，日志在 CW 中被截断为空。删除它的空日志流可能会中断并挂起服务，因为它无处 post 它的日志到...

Answer 3

这是我写的一些快速脚本：

#!/usr/bin/python3

# describe log groups
# describe log streams
# get log groups with the lastEventTimestamp after some time
# delete those log groups
# have a dry run option
# support profile

# https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/logs.html#CloudWatchLogs.Client.describe_log_streams
# https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/logs.html#CloudWatchLogs.Client.describe_log_groups
# https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/logs.html#CloudWatchLogs.Client.delete_log_group

import boto3
import time
millis = int(round(time.time() * 1000))

delete = False
debug = False
log_group_prefix='/' # NEED TO CHANGE THESE

days = 30
# Create CloudWatchLogs client
cloudwatch_logs = boto3.client('logs')

log_groups=[]
# List log groups through the pagination interface
paginator = cloudwatch_logs.get_paginator('describe_log_groups')
for response in paginator.paginate(logGroupNamePrefix=log_group_prefix):
    for log_group in response['logGroups']:
        log_groups.append(log_group['logGroupName'])

if debug:
    print(log_groups)

old_log_groups=[]
empty_log_groups=[]
for log_group in log_groups:
    response = cloudwatch_logs.describe_log_streams(
        logGroupName=log_group, #logStreamNamePrefix='',
        orderBy='LastEventTime',
        descending=True,
        limit=1
    )
    # The time of the most recent log event in the log stream in CloudWatch Logs. This number is expressed as the number of milliseconds after Jan 1, 1970 00:00:00 UTC.
    if len(response['logStreams']) > 0:
        if debug:
            print("full response is:")
            print(response)
            print("Last event is:")
            print(response['logStreams'][0]['lastEventTimestamp'])
            print("current millis is:")
            print(millis)
        if response['logStreams'][0]['lastEventTimestamp'] < millis - (days * 24 * 60 * 60 * 1000):
            old_log_groups.append(log_group)
    else:
        empty_log_groups.append(log_group)

# delete log group
if delete:
    for log_group in old_log_groups:
        response = cloudwatch_logs.delete_log_group(logGroupName=log_group)
    #for log_group in empty_log_groups:
    #    response = cloudwatch_logs.delete_log_group(logGroupName=log_group)
else:
    print("old log groups are:")
    print(old_log_groups)
    print("Number of log groups:")
    print(len(old_log_groups))
    print("empty log groups are:")
    print(empty_log_groups)

Answer 4

我做了相同的设置，我想从 Cloudwatch 日志流中删除早于 X 天的日志流。

remove.py

import optparse
import sys
import os
import json
import datetime

def deletefunc(loggroupname,days):
    os.system("aws logs describe-log-streams --log-group-name {} --order-by LastEventTime > test.json".format(loggroupname))
    oldstream=[]
    milli= days * 24 * 60 * 60 * 1000
    with open('test.json') as json_file:
       data = json.load(json_file)
       for p in data['logStreams']:
         subtract=p['creationTime']+milli
         sub1=subtract/1000
         sub2=datetime.datetime.fromtimestamp(sub1).strftime('%Y-%m-%d')
         op=p['creationTime']/1000
         original=datetime.datetime.fromtimestamp(op).strftime('%Y-%m-%d')
         name=p['logStreamName']
         if original < sub2:
             oldstream.append(name)

    for i in oldstream:
       os.system("aws logs delete-log-stream --log-group-name {} --log-stream-name {}".format(loggroupname,i))


parser = optparse.OptionParser()
parser.add_option('--log-group-name', action="store", dest="loggroupname", help="LogGroupName For Eg: testing-vpc-flow-logs",type="string")
parser.add_option('-d','--days', action="store", dest="days", help="Type No.of Days in Integer to Delete Logs other than days provided",type="int")
options, args = parser.parse_args()

if options.loggroupname is None:
        print ("Provide log group name to continue.\n")
        cmd = 'python3 ' + sys.argv[0] + ' -h'
        os.system(cmd)
        sys.exit(0)

if options.days is None:
        print ("Provide date to continue.\n")
        cmd = 'python3 ' + sys.argv[0] + ' -h'
        os.system(cmd)
        sys.exit(0)


elif options.days and options.loggroupname:

    loggroupname = options.loggroupname
    days = options.days
    deletefunc(loggroupname,days)

您可以使用以下命令运行此文件：

python3 remove.py --log-group-name=testing-vpc-flow-logs --days=7

删除 X 天内未触及的 AWS 日志组

Delete AWS Log Groups that haven't been touched in X days

amazon-web-services

amazon-cloudwatch

amazon-cloudwatchlogs