Promtail 多行不合并堆栈跟踪
Promtail multiline does not merge stacktrace
Promtail、Grafana、Loki 版本为 2.4.1。 运行 是 Kubernetes。
我正在关注 documentation。
- 日志中的异常与正则表达式匹配。 (ZeroWidthSpace 在日志行的开头)
- 多线阶段设置见附件配置(promtail.yaml)
我原以为错误堆栈跟踪会出现在 grafana/loki 中的单个条目中,但每一行都是一个单独的条目。我是否缺少一些配置?
# cat /etc/promtail/promtail.yaml
server:
log_level: info
http_listen_port: 3101
client:
url: http://***-loki:3100/loki/api/v1/push
positions:
filename: /run/promtail/positions.yaml
scrape_configs:
# See also https://github.com/grafana/loki/blob/master/production/ksonnet/promtail/scrape_config.libsonnet for reference
- job_name: kubernetes-pods
pipeline_stages:
- multiline:
firstline: ^\x{200B}\[
max_lines: 128
max_wait_time: 3s
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels:
- __meta_kubernetes_pod_controller_name
regex: ([0-9a-z-.]+?)(-[0-9a-f]{8,10})?
action: replace
target_label: __tmp_controller_name
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_name
- __meta_kubernetes_pod_label_app
- __tmp_controller_name
- __meta_kubernetes_pod_name
regex: ^;*([^;]+)(;.*)?$
action: replace
target_label: app
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_component
- __meta_kubernetes_pod_label_component
regex: ^;*([^;]+)(;.*)?$
action: replace
target_label: component
- action: replace
source_labels:
- __meta_kubernetes_pod_node_name
target_label: node_name
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: namespace
- action: replace
replacement:
separator: /
source_labels:
- namespace
- app
target_label: job
- action: replace
source_labels:
- __meta_kubernetes_pod_name
target_label: pod
- action: replace
source_labels:
- __meta_kubernetes_pod_container_name
target_label: container
- action: replace
replacement: /var/log/pods/*/*.log
separator: /
source_labels:
- __meta_kubernetes_pod_uid
- __meta_kubernetes_pod_container_name
target_label: __path__
- action: replace
regex: true/(.*)
replacement: /var/log/pods/*/*.log
separator: /
source_labels:
- __meta_kubernetes_pod_annotationpresent_kubernetes_io_config_hash
- __meta_kubernetes_pod_annotation_kubernetes_io_config_hash
- __meta_kubernetes_pod_container_name
target_label: __path__
事实证明,日志看起来与我们在镜头荚日志或 kubectl logs {pod}
中看到的不同。
promtail消费的原始日志可以在主机上找到:
minikube ssh
cat /var/log/pods/{namespace}_{pod}/{container}/0.log
它们看起来像这样:
{"log":"[default-nioEventLoopGroup-1-1] INFO HTTP_ACCESS_LOGGER - \"GET /health/readiness HTTP/1.1\" 200 523\n","stream":"stdout","time":"2021-12-17T12:26:29.702621198Z"}
因此,第一行正则表达式不匹配任何日志行。不幸的是,promtail 日志中没有关于此的错误。
这是 docker 日志格式,有一个管道阶段可以解析它:
- docker: {}
此外,日志中还有一个问题。多行堆栈跟踪中有额外的换行符,所以这个额外的管道阶段过滤它们:
- replace:
expression: '(\n)'
replace: ''
所以我的工作配置如下所示:
server:
log_level: info
http_listen_port: 3101
client:
url: http://***-loki:3100/loki/api/v1/push
positions:
filename: /run/promtail/positions.yaml
scrape_configs:
# See also https://github.com/grafana/loki/blob/master/production/ksonnet/promtail/scrape_config.libsonnet for reference
- job_name: kubernetes-pods
pipeline_stages:
- docker: {}
- multiline:
firstline: ^\x{200B}\[
max_lines: 128
max_wait_time: 3s
- replace:
expression: (\n)
replace: ""
#config continues below (not copied)
Promtail、Grafana、Loki 版本为 2.4.1。 运行 是 Kubernetes。
我正在关注 documentation。
- 日志中的异常与正则表达式匹配。 (ZeroWidthSpace 在日志行的开头)
- 多线阶段设置见附件配置(promtail.yaml)
我原以为错误堆栈跟踪会出现在 grafana/loki 中的单个条目中,但每一行都是一个单独的条目。我是否缺少一些配置?
# cat /etc/promtail/promtail.yaml
server:
log_level: info
http_listen_port: 3101
client:
url: http://***-loki:3100/loki/api/v1/push
positions:
filename: /run/promtail/positions.yaml
scrape_configs:
# See also https://github.com/grafana/loki/blob/master/production/ksonnet/promtail/scrape_config.libsonnet for reference
- job_name: kubernetes-pods
pipeline_stages:
- multiline:
firstline: ^\x{200B}\[
max_lines: 128
max_wait_time: 3s
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels:
- __meta_kubernetes_pod_controller_name
regex: ([0-9a-z-.]+?)(-[0-9a-f]{8,10})?
action: replace
target_label: __tmp_controller_name
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_name
- __meta_kubernetes_pod_label_app
- __tmp_controller_name
- __meta_kubernetes_pod_name
regex: ^;*([^;]+)(;.*)?$
action: replace
target_label: app
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_component
- __meta_kubernetes_pod_label_component
regex: ^;*([^;]+)(;.*)?$
action: replace
target_label: component
- action: replace
source_labels:
- __meta_kubernetes_pod_node_name
target_label: node_name
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: namespace
- action: replace
replacement:
separator: /
source_labels:
- namespace
- app
target_label: job
- action: replace
source_labels:
- __meta_kubernetes_pod_name
target_label: pod
- action: replace
source_labels:
- __meta_kubernetes_pod_container_name
target_label: container
- action: replace
replacement: /var/log/pods/*/*.log
separator: /
source_labels:
- __meta_kubernetes_pod_uid
- __meta_kubernetes_pod_container_name
target_label: __path__
- action: replace
regex: true/(.*)
replacement: /var/log/pods/*/*.log
separator: /
source_labels:
- __meta_kubernetes_pod_annotationpresent_kubernetes_io_config_hash
- __meta_kubernetes_pod_annotation_kubernetes_io_config_hash
- __meta_kubernetes_pod_container_name
target_label: __path__
事实证明,日志看起来与我们在镜头荚日志或 kubectl logs {pod}
中看到的不同。
promtail消费的原始日志可以在主机上找到:
minikube ssh
cat /var/log/pods/{namespace}_{pod}/{container}/0.log
它们看起来像这样:
{"log":"[default-nioEventLoopGroup-1-1] INFO HTTP_ACCESS_LOGGER - \"GET /health/readiness HTTP/1.1\" 200 523\n","stream":"stdout","time":"2021-12-17T12:26:29.702621198Z"}
因此,第一行正则表达式不匹配任何日志行。不幸的是,promtail 日志中没有关于此的错误。
这是 docker 日志格式,有一个管道阶段可以解析它:
- docker: {}
此外,日志中还有一个问题。多行堆栈跟踪中有额外的换行符,所以这个额外的管道阶段过滤它们:
- replace:
expression: '(\n)'
replace: ''
所以我的工作配置如下所示:
server:
log_level: info
http_listen_port: 3101
client:
url: http://***-loki:3100/loki/api/v1/push
positions:
filename: /run/promtail/positions.yaml
scrape_configs:
# See also https://github.com/grafana/loki/blob/master/production/ksonnet/promtail/scrape_config.libsonnet for reference
- job_name: kubernetes-pods
pipeline_stages:
- docker: {}
- multiline:
firstline: ^\x{200B}\[
max_lines: 128
max_wait_time: 3s
- replace:
expression: (\n)
replace: ""
#config continues below (not copied)