Prometheus Alertmanager provides some additional useful features around the management of alerts. These features allow you to customize and tweak your alerts so they are more useful in real-world situations. In this lab, you will have the opportunity to practice using some of these Alertmanager features, including alert grouping, inhibitions, and silences.
Learning Objectives
Successfully complete this lab by achieving the following learning objectives:
- Combine the Web Server Down Alerts into a Single Group
Log in to the Prometheus server.
Edit the Alertmanager configuration file:
sudo vi /etc/alertmanager/alertmanager.ymlAdd a new node to routing tree to combine the
WebServer.*Downalerts:route: ... routes: - receiver: 'web.hook' group_by: ['service'] match_re: alertname: 'WebServer.*Down'Load the new configuration:
sudo killall -HUP alertmanagerCheck Alertmanager in a web browser at
http://<PROMETHEUS_SERVER_PUBLIC_IP>:9093. You should see the Web Server alerts grouped together under the groupservice="webserver".
- Create an Inhibition to Stop the `WebBadGateway` Alert When a `WebServerDown` Alert Is Already Firing
Edit the Alertmanager configuration file:
sudo vi /etc/alertmanager/alertmanager.yml- Add a new inhibit rule:
inhibit_rules: ... - source_match_re: alertname: 'WebServer.*Down' target_match: alertname: 'WebBadGateway'- Load the new configuration:
sudo killall -HUP alertmanager- Check Alertmanager in a web browser at
http://<PROMETHEUS_SERVER_PUBLIC_IP>:9093. TheWebBadGatewayshould no longer appear. You can click the Inhibited box to make it appear again.
- Silence the `WebServer1Down` Alert
Access Alertmanager in a web browser at
http://<PROMETHEUS_SERVER_PUBLIC_IP>:9093.Expand the
service="webserver"group.Locate the alert with
alertname="WebServer1Down", and click the Silence button for that alert.Fill out the Creator and Comment fields, and then click Create.
If you return to the main Alertmanager page, the
WebServer1Downshould no longer appear.
