Grafana 仪表板显示 JSON Loki 记录中某个键的指标

Question

我无法理解如何创建仪表板时间序列图以显示 JSON 格式的 Loki 日志中的单个 key/value。

例如：这是我在资源管理器中的查询：

{job="railsdevlogs"}|json

其中 returns 日志行，例如：

{"date":"2022-01-05T21:27:21.895Z","pool":{"Pool Size":50,"Current":5,"Active":1,"Idle":4,"Dead":0,"Timeout":"5 sec"},"puma":{"Started At":"2022-01-05T20:35:26Z","Max Threads":16,"Pool Capacity":16,"Running":1,"Backlog":0,"IO Handles":15,"File Handles":2,"Socket Handles":4,"Server Log Size":46750072},"process":[{"Name":"ruby.exe","Process ID":656,"Threads":11,"Working Set":150728704,"Virtual Size":288079872},{"Name":"mysqld.exe","Process ID":4836,"Threads":3,"Working Set":360448,"Virtual Size":4445065216},{"Name":"mysqld.exe","Process ID":5808,"Threads":49,"Working Set":69906432,"Virtual Size":4924059648},{"Name":"aaaaa.exe","Process ID":14460,"Threads":18,"Working Set":49565696,"Virtual Size":5478469632},{"Name":"bbbbb.exe","Process ID":9584,"Threads":14,"Working Set":35012608,"Virtual Size":4496551936},{"Name":"ccccc.exe","Process ID":11944,"Threads":14,"Working Set":29609984,"Virtual Size":4481880064}],"gc":{"count":242,"heap_allocated_pages":1277,"heap_sorted_length":1279,"heap_allocatable_pages":9,"heap_available_slots":869213,"heap_live_slots":464541,"heap_free_slots":404672,"heap_final_slots":0,"heap_marked_slots":411311,"heap_swept_slots":457903,"heap_eden_pages":1268,"heap_tomb_pages":9,"total_allocated_pages":1278,"total_freed_pages":1,"total_allocated_objects":74364715,"total_freed_objects":73900174,"malloc_increase_bytes":640096,"malloc_increase_bytes_limit":16777216,"minor_gc_count":131,"major_gc_count":111,"remembered_wb_unprotected_objects":57031,"remembered_wb_unprotected_objects_limit":114062,"old_objects":349257,"old_objects_limit":698512,"oldmalloc_increase_bytes":640288,"oldmalloc_increase_bytes_limit":16777216},"os":{"System Name":"xxxxx","Description":"","Organization":"","Operating System":"Microsoft Windows 10 Enterprise LTSC","OS Version":"10.0.17763","OS Serial Number":"xxxxx-xxxxx-xxxxx-xxxxx","System Time":"2022-01-05T16:27:22.000-05:00","System Time Zone":-300,"Last Boot Time":"2021-12-15T23:26:38.000-05:00","System Drive":"C:","Total Physical Memory":34204393472,"Free Physical Memory":20056260608,"Total Virtual Memory":39304667136,"Free Virtual Memory":13915041792,"Number of Processes":307,"Number of Users":2,"volumes":[{"Drive":"C:\","Type":"NTFS","Total Space":1023563264000,"Free Space":681182343168,"Block Size":4096}]},"symbol":{"size":28106},"stats_collection_time":387}

使用 |json 将自动为 json 日志行中的所有 key/value 创建动态标签：

gc_count  =  123
os_Free_Virtual_Memory  = 456789
etc.

现在我想在 grafana 时间序列图中绘制这些值之一，但我很难理解如何隔离一个动态标签并绘制它。

也许我使用 |json 不正确。到目前为止我阅读的文档和示例显示了如何使用动态标签过滤日志，但我不需要它，因为我想绘制每条日志行。

谢谢

Answer 1

我认为如果您转到第 41 分钟，这应该会对 https://grafana.com/go/observabilitycon/2020/keynote-what-is-observability/ 有所帮助。有一个示例与您要实现的目标非常相似。

您的查询应该类似于：

quantile_over_time(0,99, {job="railsdevlogs"} 
| json 
| unwrap gc_count [1m]} 
by (job)

Grafana 仪表板显示 JSON Loki 记录中某个键的指标

Grafana dashboard to display a metric for a key in JSON Loki record

grafana

grafana-loki