Google Cloud logging，Python3.8标准环境，按trace id分组请求相关日志

Question

我在 Google App Engine 标准环境中 Python3.8 的 Google 云日志设置期间遇到问题。我正在将 FastAPI 与独角兽一起使用。我的代码记录配置：

import logging.config
import sys

from google.cloud import logging as google_logging

from app.settings import ENV, _settings


if _settings.ENV == ENV.LOCAL:
    MAIN_LOGGER = 'console'
    LOGGER_CONF_DICT = {
        'class': 'logging.StreamHandler',
        'formatter': 'verbose',
        'stream': sys.stdout,
        'level': _settings.LOG_LEVEL.upper(),
    }
else:
    log_client = google_logging.Client()
    MAIN_LOGGER = 'stackdriver_logging'
    LOGGER_CONF_DICT = {
        'class': 'app.gcloud_logs.GCLHandler',
        'client': log_client,
        'name': 'appengine.googleapis.com%2Frequest_log'
        # I've tried other names: stdout, %2FA instead of / symbol, appengine.googleapis.com/stdout
        # the same result or no logs at all
    }

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'verbose': {
            'format': '%(log_color)s%(asctime)s [%(levelname)s] [%(name)s] %(message)s (%(filename)s:%(lineno)d)',
            '()': 'colorlog.ColoredFormatter',
            'log_colors': {
                'DEBUG': 'cyan',
                'INFO': 'green',
                'WARNING': 'yellow',
                'ERROR': 'red',
                'CRITICAL': 'bold_red',
            },
        }
    },
    'handlers': {
        MAIN_LOGGER: {**LOGGER_CONF_DICT},
        'blackhole': {'level': 'DEBUG', 'class': 'logging.NullHandler'},
    },
    'loggers': {
        'fastapi': {'level': 'INFO', 'handlers': [MAIN_LOGGER]},
        'uvicorn.error': {'level': 'INFO', 'handlers': [MAIN_LOGGER], 'propagate': False},
        'uvicorn.access': {'level': 'INFO', 'handlers': [MAIN_LOGGER], 'propagate': False},
        'uvicorn': {'level': 'INFO', 'handlers': [MAIN_LOGGER], 'propagate': False},
        'google.cloud.logging.handlers.transports.background_thread': {'level': 'DEBUG', 'handlers': ['blackhole'],
                                                                       'propagate': False},
        '': {
            'level': _settings.LOG_LEVEL.upper(),
            'handlers': [MAIN_LOGGER],
            'propagate': True,
        },

    }
}

logging.config.dictConfig(LOGGING)

我的日志处理程序代码：

import os
from typing import Any, Dict, Optional

from google.cloud.logging.handlers import CloudLoggingHandler
from google.cloud.logging.resource import Resource
from starlette.requests import Request
from starlette_context import context

from app.settings import _settings


class GCLHandler(CloudLoggingHandler):
    def emit(self, record):
        message = super(GCLHandler, self).format(record)
        request: Optional[Request] = None
        trace: Optional[str] = None
        span_id: Optional[str] = None
        user_id: Optional[int] = None
        resource = Resource(
            type='gae_app',
            labels={
                'module_id': os.environ['GAE_SERVICE'],
                'project_id': _settings.PROJECT_NAME,
                'version_id': os.environ['GAE_VERSION'],
                'zone': 'us16'  # tried without zone - the same result
            }
        )

        labels: Dict[str, Any] = {}

        if context.exists():  # I'm sure that it works
            request = context.get('request')  # I'm sure that it works
            user_id = context.get('user_id')  # I'm sure that it works

        if user_id is not None:
            labels['user_id'] = user_id

        if request:
            if request.headers.get('X-Cloud-Trace-Context'):
                cloud_trace = request.headers.get('X-Cloud-Trace-Context').split('/')
                if len(cloud_trace) > 1:
                    span_id = cloud_trace[1].split(';')[0]

                trace = f'projects/{_settings.PROJECT_NAME}/traces/{cloud_trace[0]}'
                labels['logging.googleapis.com/trace'] = cloud_trace[0]  # Found in some guides, not sure that its neccessary
                labels['appengine.googleapis.com/trace_id'] = cloud_trace[0]  # Found in some guides, not sure that its neccessary

        self.transport.send(
            record,
            message,
            resource=resource,
            labels=labels,
            trace=trace,
            span_id=span_id
        )

我在日志查看器中得到一些奇怪的结果，我的日志与请求日志具有相同的跟踪，但它们没有分组

有什么想法吗？

Answer 1

App Engine 中有 2 种类型的日志：

请求日志：发送到您应用的请求的日志。 App Engine 自动在请求日志中创建条目。
应用日志：您写入受支持的框架或文件的日志条目，如本页所述。

这两个日志都由 App Engine Standard 自动发送到 Cloud Logging Agent。

在第一次请求时，应用程序日志和请求日志不相关，这就是为什么没有显示在一个组中，这是 known issue stated in App Engine Official Documentation。但是在第二个请求中，您可以看到日志显示在一个组中。

Public Issue Tracker 中的功能请求已针对此行为创建，您将在其中获得有关修复的所有更新。

Answer 2

log_client = google_logging.Client()
MAIN_LOGGER = 'stackdriver_logging'
LOGGER_CONF_DICT = {
    'class': 'app.gcloud_logs.GCLHandler',
    'client': log_client,
    'name': 'app'
}

将名称更改为 app 有帮助

Google Cloud logging，Python3.8标准环境，按trace id分组请求相关日志

Google Cloud logging, Python3.8 standard environment, group request related logs by trace id

python-3.x

google-app-engine

google-cloud-logging