通知警报：超过上下文截止日期（可能是代理问题）

Question

我正在尝试将 EC2 AWS 中 ubuntu 上的 Alermanager 服务器连接到 Slack，但我收到此错误：

Apr 14 18:22:26 prometheus-db-v01-01a.myserver.com alertmanager[5854]: level=error ts=2019-04-14T18:22:26.658601495Z caller=dispatch.go:177 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post <redacted>: context deadline exceeded"

我的 alertmanager.yum 看起来像：

templates:
- '/etc/alertmanager/template/slack.tmpl'

route:
  receiver: slack_general
  repeat_interval: 5m
  group_by: [alertname]
  routes:
# severity=info alerts will not try to match to any other rule
    - match:
        severity: info
      receiver: slack_general

receivers:
- name: slack_general
  slack_configs:
  - api_url: https://hooks.slack.com/services/ID
    send_resolved: true
    username: 'Prometheus-bot'
    channel: '#errors'
    title: '{{ template "slack.my.title" . }}'
    text: '{{ template "slack.my.text" . }}'

我可以看到警报触发了我的测试警报：

root@prometheus-db-v01-01a:~# amtool alert --alertmanager.url=http://localhost:9093 -v
Alertname         Starts At                Summary
Cassandra_yellow  2019-04-14 18:11:56 UTC  The cassandra  cluster is in yellow state

我用 systemd 启动了 prometheus：

root@prometheus-db-v01-01a:~# cat /etc/systemd/system/alertmanager.service
[Unit]
Description=Prometheus Alertmanager Service
Wants=network-online.target
After=network.target

[Service]
Environment=https_proxy=http://proxy:80/
Environment=http_proxy=http://proxy:80/
User=alertmanager
Group=alertmanager
Type=simple
ExecStart=/usr/local/bin/alertmanager \
    --config.file /etc/alertmanager/alertmanager.yml \
    --storage.path /var/lib/alertmanager/data
Restart=always

[Install]
WantedBy=multi-user.target
root@prometheus-db-v01-01a:~#

我尝试使用 2 Environment 变量设置代理，但结果相同

服务器使用代理与 Slack 通信，我可以用它进行测试：

root@prometheus-db-v01-01a:~# curl -X POST --data-urlencode "payload={\"channel\": \"#errors\", \"username\": \"webhookbot\", \"text\": \"This is posted to #errors and comes from a bot named webhookbot.\", \"icon_emoji\": \":ghost:\"}" https://hooks.slack.com/services/ID

有效，而

root@prometheus-db-v01-01a:~# curl -X POST --noproxy "*" --data-urlencode "payload={\"channel\": \"#errors\", \"username\": \"webhookbot\", \"text\": \"This is posted to #errors and comes from a bot named webhookbot.\", \"icon_emoji\": \":ghost:\"}" https://hooks.slack.com/services/ID
curl: (7) Failed to connect to hooks.slack.com port 443: Connection timed out

以上所有命令均来自alertmanger服务器。

我认为这是代理问题是否正确？我该如何设置它？

提前致谢

Answer 1

问题已解决：是代理，需要在alertmanager.yml文件中设置

templates:
- '/etc/alertmanager/template/slack.tmpl'

global:
  http_config:
    proxy_url: 'http://proxy:80/'

route:
  receiver: slack_general
  repeat_interval: 5m
  group_by: [alertname]
  routes:

    - match:
        severity: minor
      receiver: slack_general

receivers:
- name: slack_general
  slack_configs:
[...]

通知警报：超过上下文截止日期（可能是代理问题）

Notify for alerts : context deadline exceeded (possible proxy issue)

amazon-web-services

amazon-ec2

proxy

prometheus-alertmanager