Configuring Prometheus Alerts

45 minutes
  • 2 Learning Objectives

About this Hands-on Lab

Prometheus alerts allow you to issue real-time notifications triggered by your Prometheus metric data. In this lab, you will work directly with Prometheus alerts. You will create an alert in an existing Prometheus Server by modifying the Prometheus configuration.

Learning Objectives

Successfully complete this lab by achieving the following learning objectives:

Create an Alerting Rule in Prometheus to Alert When the Server Goes Down
  1. Log in to the Prometheus server.

  2. Edit the Prometheus config file:

    sudo vi /etc/prometheus/prometheus.yml
  3. Add a path for rules files to the Prometheus config:

      - "/etc/prometheus/rules/*.yml"
  4. Create the rules directory:

    sudo mkdir -p /etc/prometheus/rules/
  5. Create a new rules file for your alerting rule:

    sudo vi /etc/prometheus/rules/limedrop-web.yml
  6. Implement an alerting rule to issue an alert when the server goes down:

    - name: limedrop-web
      - alert: WebServerDown
        expr: up{instance="limedrop-web:9100"} == 0
          severity: critical
          summary: Web Server Down
  7. Restart Prometheus to reload the configuration:

    sudo systemctl restart prometheus
  8. Access Prometheus in a browser at http://<PROMETHEUS_SERVER_PUBLIC_IP>:9090.

  9. Click Alerts. You should see your WebServerDown alert listed.

Configure a Routing Tree and Receiver in Alertmanager
  1. Edit the Alertmanager configuration file:

    sudo vi /etc/alertmanager/alertmanager.yml
  2. Add a new node to the routing table:

      - receiver: 'web.hook'
        group_wait: 30s
          alertname: WebServerDown
  3. Restart Alertmanager to reload the configuration:

    sudo systemctl restart alertmanager
  4. You can test your alert by shutting down the limedrop-web exporter to simulate the server going down. To do this, stop the limedrop_web_exporter service:

    sudo systemctl stop limedrop_web_exporter
  5. After a few moments, check the Alertmanager page in your browser: http://<PROMETHEUS_SERVER_PUBLIC_IP>:9093. If the alert is firing, you should see the WebServerDown alert appear.

  6. If you wish, you can start limedrop_web_exporter again and watch the alert disappear in Alertmanager:

    sudo systemctl start limedrop_web_exporter

Additional Resources

Your company, LimeDrop, is using Prometheus to monitor some systems and applications. They have a web server called limedrop-web that is having some stability issues. They would like to issue an alert to Alertmanager whenever this server goes down. Configure Prometheus and Alertmanager to handle this alert.

You will need to:

  • Configure an alerting rules directory in Prometheus.
  • Add an alerting rule to fire an alert when the server goes down. You can use the expression up{instance="limedrop-web:9100"} == 0.
  • Add a routing tree node to the Alertmanager config.

You can simulate the server going down to fire the alert by stopping its exporter:

sudo systemctl stop limedrop_web_exporter

What are Hands-on Labs

Hands-on Labs are real environments created by industry experts to help you learn. These environments help you gain knowledge and experience, practice without compromising your system, test without risk, destroy without fear, and let you learn from your mistakes. Hands-on Labs: practice your skills before delivering in the real world.

Sign In
Welcome Back!

Psst…this one if you’ve been moved to ACG!

Get Started
Who’s going to be learning?