Prometheus 服务启动失败

Question

过去 15 天我都在使用 Prometheus。从昨天开始，服务因内存不足而失败。

我已将内存从 4GB 增加到 8GB，但现在我无法启动 Prometheus 服务并因以下错误而失败：

Nov 15 10:25:20 prometheus: level=info ts=2019-11-15T18:25:20.393Z caller=head.go:598 component=tsdb msg="head GC completed" duration=22.375774ms
Nov 15 10:25:35 prometheus: level=info ts=2019-11-15T18:25:35.135Z caller=head.go:668 component=tsdb msg="WAL checkpoint complete" first=549 last=550 duration=14.741502221s
Nov 15 10:25:35 prometheus: level=info ts=2019-11-15T18:25:35.795Z caller=head.go:598 component=tsdb msg="head GC completed" duration=22.593533ms
Nov 15 10:25:36 prometheus: level=info ts=2019-11-15T18:25:36.796Z caller=compact.go:496 component=tsdb msg="write block" mint=1573826400000 maxt=1573833600000 ulid=01DSR50E9ZP9F1ZSY2YD30GEHK duration=732.572029ms
Nov 15 10:25:37 prometheus: level=error ts=2019-11-15T18:25:37.062Z caller=db.go:604 component=tsdb msg="compaction failed" err="reload blocks: unexpected corrupted block:map[01DSR50E9ZP9F1ZSY2YD30GEHK:mmap files: mmap: cannot allocate memory]"
Nov 15 10:25:38 prometheus: level=info ts=2019-11-15T18:25:38.824Z caller=compact.go:496 component=tsdb msg="write block" mint=1573826400000 maxt=1573833600000 ulid=01DSR50G8EYTYBPWWHWZ1RJV5A duration=761.592662ms
Nov 15 10:25:39 prometheus: level=error ts=2019-11-15T18:25:39.087Z caller=db.go:604 component=tsdb msg="compaction failed" err="reload blocks: unexpected corrupted block:map[01DSR50G8EYTYBPWWHWZ1RJV5A:mmap files: mmap: cannot allocate memory]"
Nov 15 10:25:41 prometheus: level=info ts=2019-11-15T18:25:41.816Z caller=compact.go:496 component=tsdb msg="write block" mint=1573826400000 maxt=1573833600000 ulid=01DSR50K6ZJFJQ4217JW2E5XBV duration=729.049353ms
Nov 15 10:25:42 prometheus: level=error ts=2019-11-15T18:25:42.087Z caller=db.go:604 component=tsdb msg="compaction failed" err="reload blocks: unexpected corrupted block:map[01DSR50K6ZJFJQ4217JW2E5XBV:mmap files: mmap: cannot allocate memory]"
Nov 15 10:25:46 prometheus: level=info ts=2019-11-15T18:25:46.814Z caller=compact.go:496 component=tsdb msg="write block" mint=1573826400000 maxt=1573833600000 ulid=01DSR50R37245ZE56EZNH0335G duration=727.559705ms
Nov 15 10:25:47 prometheus: level=error ts=2019-11-15T18:25:47.084Z caller=db.go:604 component=tsdb msg="compaction failed" err="reload blocks: unexpected corrupted block:map[01DSR50R37245ZE56EZNH0335G:mmap files: mmap: cannot allocate memory]"
Nov 15 10:25:55 prometheus: level=info ts=2019-11-15T18:25:55.807Z caller=compact.go:496 component=tsdb msg="write block" mint=1573826400000 maxt=1573833600000 ulid=01DSR510WDE18F41GJBRVXBEDE duration=722.624112ms
Nov 15 10:25:55 prometheus: level=error ts=2019-11-15T18:25:56.080Z caller=db.go:604 component=tsdb msg="compaction failed" err="reload blocks: unexpected corrupted block:map[01DSR510WDE18F41GJBRVXBEDE:mmap files: mmap: cannot allocate memory]"

普罗米修斯服务：


[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file /etc/prometheus/prometheus.yml \
--storage.tsdb.path /var/lib/prometheus/ \
--web.console.templates=/etc/prometheus/consoles \
--web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

我已经在同一台服务器上安装了 influxdb grafana。

请告诉我问题到底是什么？

提前致谢

Answer 1

你是运行 32 位 prometheus 吗？您可能运行没有地址 space。我认为 Prometheus 不打算在 32 位系统上实现这项工作，它只是用完了 4GB 地址 space，所有 mmaping 都在发生。

更多详情请见https://github.com/prometheus/prometheus/issues/4392#issuecomment-433721793

Prometheus 服务启动失败

Prometheus service is failing to start

prometheus