GCP Cloud 运行（with python）记录到云记录

Question

我正在尝试使用云运行通过 HTTPS（流式传输）接收数百万条日志消息并将它们发送到云日志记录。

但是我发现有一些数据丢失，云日志中的消息数量少于云运行接收。

这是我试过的示例代码，

# unzip data
data = gzip.decompress(request.data)

# split by lines
logs = data.decode('UTF-8').split('\n')

# output the logs
log_cnt = 0
for log in logs:
    try:
        # output to jsonPayload
        print(json.dumps(json.loads(log_str))
        log_cnt += 1
    except Exception as e:
        logging.error(F"messsage: {str(e)}")

并且如果我比较 log_cnt 和云日志中的日志数量，log_cnt 更多。所以一些打印没有完成传递消息。

我尝试使用logging API代替print()，但是使用logging API发送的日志数量太多（12,000次调用限制为一分钟），因此导致延迟非常糟糕，无法稳定处理请求。

我猜想实例的移动数量可能会导致它，所以我在活动实例未更改时进行测试，但仍然有 3-5% 的消息丢失。

有什么办法可以将所有消息无任何损失地发送到云日志记录吗？

（更新）

这行数据看起来像这样，（大约 1kb）

{"key1": "ABCDEFGHIJKLMN","key2": "ABCDEFGHIJKLMN","key3": "ABCDEFGHIJKLMN","key4": "ABCDEFGHIJKLMN","key5": "ABCDEFGHIJKLMN","key6": "ABCDEFGHIJKLMN","key7": "ABCDEFGHIJKLMN","key8": "ABCDEFGHIJKLMN","key9": "ABCDEFGHIJKLMN","key10": "ABCDEFGHIJKLMN","key11": "ABCDEFGHIJKLMN","key12": "ABCDEFGHIJKLMN","key13": "ABCDEFGHIJKLMN","key14": "ABCDEFGHIJKLMN","key15": "ABCDEFGHIJKLMN","key16": "ABCDEFGHIJKLMN","key17": "ABCDEFGHIJKLMN","key18": "ABCDEFGHIJKLMN","key19": "ABCDEFGHIJKLMN","key20": "ABCDEFGHIJKLMN","key21": "ABCDEFGHIJKLMN","key22": "ABCDEFGHIJKLMN","key23": "ABCDEFGHIJKLMN","key24": "ABCDEFGHIJKLMN","key26": "ABCDEFGHIJKLMN","key27": "ABCDEFGHIJKLMN","key28": "ABCDEFGHIJKLMN","key29": "ABCDEFGHIJKLMN","key30": "ABCDEFGHIJKLMN","key31": "ABCDEFGHIJKLMN","key32": "ABCDEFGHIJKLMN","key33": "ABCDEFGHIJKLMN","key34": "ABCDEFGHIJKLMN","key35": "ABCDEFGHIJKLMN"}

Answer 1

我可以推荐您使用 entries.write API，它允许您同时写入大量条目。在 API 资源管理器

中查看我的测试

可以解决JSON格式和同时写入多个问题。试试这个 API，如果它对你更好，请告诉我。如果没有，我会删除这个答案！

GCP Cloud 运行（with python）记录到云记录

GCP Cloud run (with python) logging to cloud logging

python-3.x

google-cloud-logging

google-cloud-run