如何将 google-api-client 用于 Google Cloud Logging

Question

我想从 python 脚本访问 Google 云平台日志记录。

我可以从 https://cloud.google.com/logging/docs/reference/v2/rest/v2/entries/list 访问此日志 --> 试试这个 API

现在我想得到相同的，但是来自 Python 脚本。我在之前的步骤中看到，自动创建了一个授权令牌。

我正在尝试使用此代码示例，但我不知道如何 POST https://logging.googleapis.com/v2/entries:list 使用发现：

from google.oauth2 import service_account
import googleapiclient.discovery

credentials = service_account.Credentials.from_service_account_file(service_account_file)
logging = googleapiclient.discovery.build('logging', 'v2', credentials=credentials)

然后我尝试了这个代码示例：

import requests
payload = {
    "projectIds": [
        "my-proyect"
    ],
    "resourceNames": [],
    "filter": "resource.type=cloudiot_device",
    "orderBy": "timestamp desc",
    "pageSize": 1
}
headers = {"Authorization": "Bearer AAAAAAA"}
r = requests.post("https://logging.googleapis.com/v2/entries:list", params=payload, headers=headers)

该代码示例工作正常，但它放置 AAAAAAA 令牌的位置我复制并粘贴了我在 https://cloud.google.com/logging/docs/reference/v2/rest/v2/entries/list 中看到的代码，但我不知道如何从 python 脚本生成此令牌.

谢谢！

Answer 1

这不太容易找到，因为许多 Google 的云 (!) 服务现在更喜欢云客户端库。

然而...

import google.auth

from googleapiclient import discovery

credentials, project = google.auth.default()

service = discovery.build("logging", "v2", credentials=credentials)

授权：https://pypi.org/project/google-auth/

现在，这使用 Google 应用程序默认凭据，我建议您创建一个服务帐户，生成一个密钥并授予该帐户所需的权限。然后，您需要在运行代码之前导出 GOOGLE_APPLICATION_CREDENTIALS。

PROJECT=[[YOUR-PROJECT]]
BILLING=[[YOUR-BILLING]]
ACCOUNT=[[YOUR-ACCOUNT]]

gcloud projects create ${PROJECT}
gcloud beta billing projects link ${PROJECT} \
--billing-account=${BILLING}

gcloud iam service-accounts create ${ACCOUNT} \
--project=${PROJECT}

EMAIL="${ACCOUNT}@${PROJECT}.iam.gserviceaccount.com"

gcloud iam service-accounts keys create ${PWD}/${ACCOUNT}.json \
--iam-account=${EMAIL} \
--project=${PROJECT}

# See: https://cloud.google.com/iam/docs/understanding-roles#logging-roles
gcloud projects add-iam-policy-binding ${PROJECT} \
--member=serviceAccount:${EMAIL} \
--role=roles/logging.viewer

export GOOGLE_APPLICATION_CREDENTIALS=${PWD}/${ACCOUNT}.json

python3 your-code.py

Answer 2

我正在使用以下代码示例从 Google Cloud Logging 中提取日志信息。

import os
from google.cloud import logging
from google.cloud.logging import DESCENDING

os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "my-service-account-file"

def list_entries(logger_name):
    """Lists the most recent entries for a given logger."""
    logging_client = logging.Client()
    logger = logging_client.logger(logger_name)
    print("Listing entries for logger {}:".format(logger.name))
    filter_str = "resource.type=cloudiot_device AND resource.labels.device_num_id=00000000000 AND jsonPayload.eventType=PUBLISH"
    for entry in logger.list_entries(filter_=filter_str, order_by=DESCENDING, page_size=10):
        timestamp = entry.timestamp.isoformat()
        print(" {}: {}".format(, timestamp, entry.payload))

list_entries("cloudiot.googleapis.com%2Fdevice_activity")

我的目标是每 5 分钟运行这个 python 脚本并从日志记录中获取最后 5 个条目。我的问题是此代码示例开始提取条目，但它永远不会停止。如何限制条目数？

谢谢！

Answer 3

好的，感谢 Google Engineer，解决方案的第一部分是禁用 SDK 对 gRPC 的使用并强制使用 HTTP，以便遵守 page_size：

client = logging.Client(_use_grpc=0)

或者，您可以 GOOGLE_CLOUD_DISABLE_GRPC="{{anything}}"

解决方案的第二部分是仅迭代 page_size 结果的第一页：

iterator = logger.list_entries(
    order_by=DESCENDING,
    page_size=page_size,
)
print(type(iterator))

for entry in next(iterator.pages):
    timestamp = entry.timestamp.isoformat()
    print("{}".format(timestamp))

NOTE forcing HTTP entails logger.list_entries returning an HTTPIterator instead of a (gRPC) generator hence the ability to use next() and the pages property.

NOTE The 'trick' is to only enumerate the first page of n results. There may be multiple pages but we ignore subsequent ones.

如何将 google-api-client 用于 Google Cloud Logging

How to use google-api-client for Google Cloud Logging

python-3.x

google-app-engine

google-api-client

google-cloud-logging