alertmanager 安装

alertmanager 安装，它监听在9093/9094端口

安装

tar -zxvf alertmanager-0.21.0.linux-amd64.tar.gz -C /usr/local
ln -sn /usr/local/alertmanager-0.21.0.linux-amd64 /usr/local/alertmanager

• vim /etc/systemd/system/alertmanager.service

[Unit]
Description=alertmanager
After=network.target

[Service]
ExecStart=/usr/local/alertmanager/alertmanager --config.file=/usr/local/alertmanager/alertmanager.yml --web.external-url=100.80.0.128:9093
ExecStop=/bin/kill -KILL $MAINPID
ExecReload=/bin/kill -HUP $MAINPID
KillMode=control-group
Restart=on-failure
RestartSec=3s

[Install]
WantedBy=multi-user.target

配置

$ alertmanager.yml
global:
  resolve_timeout: 5m

route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  receiver: 'web.hook'
receivers:
- name: 'web.hook'
  webhook_configs:
  - url: 'http://127.0.0.1:5001/'
    http_config:
      bearer_token: xxxx
      tls_config:
        insecure_skip_verify: true
inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'dev', 'instance']

启动服务

systemctl daemon-reload
systemctl start alertmanager
systemctl status alertmanager
systemctl enable alertmanager

访问

• 端口：9093
• http://:9093/#/alerts

告警示例

接受端

• golang 代码

package main

import (
	"fmt"
	"net/http"
)

func main() {
	http.HandleFunc("/", func(writer http.ResponseWriter, request *http.Request) {
		fmt.Printf("%#v\n", request.Body)

		writer.Write([]byte(`{"status": 200}`))
	})

	fmt.Println("license on :5001")
	fmt.Println(http.ListenAndServe(":5001", nil))
}

告警示例

{"receiver":"web\\.hook","status":"resolved","alerts":[{"status":"resolved","labels":{"alertname":"target is down","instance":"100.80.0.128:9100","job":"node","level":"warning"},"annotations":{"description":"节点故障","summary":"节点故障"},"startsAt":"2022-04-29T03:41:53.092589547Z","endsAt":"2022-04-29T03:49:23.092589547Z","generatorURL":"http://100.80.0.128:9090/graph?g0.expr=up+%30026g0.tab=1","fingerprint":"d863629cfb977c6d"}],"groupLabels":{"alertname":"target is down"},"commonLabels":{"alertname":"target is down","instance":"100.80.0.128:9100","job":"node","level":"warning"},"commonAnnotations":{"description":"节点故障","summary":"节点故障"},"externalURL":"http://golang-dev:9093","version":"4","groupKey":"{}:{alertname=\"target is down\"}","truncatedAlerts

webhook

https://github.com/xiexianbin/go-alertmanager-webhook