Configuring Prometheus Alerts

Prometheus alerts allow you to issue real-time notifications triggered by your Prometheus metric data. In this lab, you will work directly with Prometheus alerts. You will create an alert in an existing Prometheus Server by modifying the Prometheus configuration.

Learning Objectives

Successfully complete this lab by achieving the following learning objectives:

Create an Alerting Rule in Prometheus to Alert When the Server Goes Down

Log in to the Prometheus server.
Edit the Prometheus config file:
```
sudo vi /etc/prometheus/prometheus.yml
```
Add a path for rules files to the Prometheus config:
```
rule_files:
  - "/etc/prometheus/rules/*.yml"
```
Create the rules directory:
```
sudo mkdir -p /etc/prometheus/rules/
```
Create a new rules file for your alerting rule:
```
sudo vi /etc/prometheus/rules/limedrop-web.yml
```

Implement an alerting rule to issue an alert when the server goes down:

groups:
- name: limedrop-web
  rules:
  - alert: WebServerDown
    expr: up{instance="limedrop-web:9100"} == 0
    labels:
      severity: critical
    annotations:
      summary: Web Server Down

Restart Prometheus to reload the configuration:
```
sudo systemctl restart prometheus
```
Access Prometheus in a browser at http://<PROMETHEUS_SERVER_PUBLIC_IP>:9090.
Click Alerts. You should see your WebServerDown alert listed.

Configure a Routing Tree and Receiver in Alertmanager

Edit the Alertmanager configuration file:

sudo vi /etc/alertmanager/alertmanager.yml

Add a new node to the routing table:

route:

  ...

  routes:
  - receiver: 'web.hook'
    group_wait: 30s
    match_re:
      alertname: WebServerDown

Restart Alertmanager to reload the configuration:
```
sudo systemctl restart alertmanager
```
You can test your alert by shutting down the limedrop-web exporter to simulate the server going down. To do this, stop the limedrop_web_exporter service:
```
sudo systemctl stop limedrop_web_exporter
```
After a few moments, check the Alertmanager page in your browser: http://<PROMETHEUS_SERVER_PUBLIC_IP>:9093. If the alert is firing, you should see the WebServerDown alert appear.
If you wish, you can start limedrop_web_exporter again and watch the alert disappear in Alertmanager:
```
sudo systemctl start limedrop_web_exporter
```

What are Hands-on Labs

Hands-on Labs are real environments created by industry experts to help you learn. These environments help you gain knowledge and experience, practice without compromising your system, test without risk, destroy without fear, and let you learn from your mistakes. Hands-on Labs: practice your skills before delivering in the real world.

Configuring Prometheus Alerts

About this Hands-on Lab

Learning Objectives

Additional Resources

What are Hands-on Labs

Newsletter