Using Grafana with Prometheus for Alerting and Monitoring

For the last six months, the Acme Anvil Corporation has been migrating some of their bare metal infrastructure to Docker containers. The team has set up Prometheus using cAdvisor to help monitor containers. After running Prometheus in the production environment for a few weeks, the team has decided to add Grafana for visualization and alerting.

Learning Objectives

Successfully complete this lab by achieving the following learning objectives:

Configure Docker.

Open port 9323 in the firewall.

firewall-cmd --zone=public --add-port=9323/tcp

Open /etc/docker/daemon.json and add the following.

{
  "metrics-addr" : "0.0.0.0:9323",
  "experimental" : true
}

Update `prometheus.yml`.

Edit the prometheus.yml file in the /root directory.

vi ~/prometheus.yml

Change the contents to reflect the following:

scrape_configs:
  - job_name: prometheus
    scrape_interval: 5s
    static_configs:
    - targets:
      - prometheus:9090
      - node-exporter:9100
      - pushgateway:9091
      - cadvisor:8080

  - job_name: docker
    scrape_interval: 5s
    static_configs:
    - targets:
      - PRIVATE_IP_ADDRESS:9323

Update `docker-compose.yml`.

Edit docker-compose.yml in the /root directory.

vi ~/docker-compose.yml

Change the contents to reflect the following:

version: '3'
services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    ports:
      - 9090:9090
    command:
      - --config.file=/etc/prometheus/prometheus.yml
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
    depends_on:
      - cadvisor
  cadvisor:
    image: google/cadvisor:latest
    container_name: cadvisor
    ports:
      - 8080:8080
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
  pushgateway:
    image: prom/pushgateway
    container_name: pushgateway
    ports:
      - 9091:9091
  node-exporter:
    image: prom/node-exporter:latest
    container_name: node-exporter
    restart: unless-stopped
    expose:
      - 9100
  grafana:
    image: grafana/grafana:5.3.0
    container_name: grafana
    ports:
      - 3000:3000
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=password
    depends_on:
      - prometheus
      - cadvisor

Install the Docker and System Monitoring dashboard.

Click the plus sign (+) on the left side of the Grafana interface.
Click Import.
Copy the contents of the JSON file included in the lab instructions.

Paste the contents of the JSON file into the Import screen of the Grafana interface, and click Load.
In the upper right-hand corner, click on Refresh every 5m and select Last 5 minutes.

Additional Resources

In this learning activity, you are tasked with setting up Prometheus and Grafana.

You will need to run the following command to make sure the Private IP is accessible for the Prometheus data source.
firewall-cmd --add-port 9090/tcp

The environment has 4 containers running: prometheus, cadvisor, nginx, and redis.

Configure Docker:

Create a daemon.json file for Docker.
- Add "metrics-addr" and set it to "0.0.0.0:9323".
- Set "experimental" to true.
- Restart the Docker service.

Update Prometheus:

Update the prometheus.yml file that is located in /root.
- Rename the cadvisor job to prometheus.
- Under targets, add the following:
- prometheus:9090
- node-exporter:9100
- pushgateway:9091
- Create a new job called docker.
- Set the scrape interval to 5 seconds.
- Create a static config and targets.
- Add the private IP with port 9323 as a target.

Update Docker Compose:

Open docker-compose.yml.
- Add a new service called pushgateway.
- Use the prom/pushgateway image.
- Name the container pushgateway.
- Map port 9091 to port 9091 on the container.
- Add a new service called node-exporter.
- Use prom/node-exporter:latest for the image.
- Name the container node-exporter.
- Map port 9100 to port 9100 on the container.
- Create a new service called grafana.
- Use the grafana/grafana image.
- Name the container grafana.
- Map port 3000 to port 3000 on the container.
- Create an environment variable called GF_SECURITY_ADMIN_PASSWORD and set it to password.
- Make sure that it depends on prometheus and the cadvisor containers.
- Execute /usr/local/bin/docker-compose up -d to rebuild the environment.
- Verify that all of your Prometheus targets are healthy by going to http://<PUBLIC_IP>:9090/targets.

Create a New Grafana Datasource:

Log in to Grafana at http://<PUBLIC_IP>:3000/login with the username admin and the password password.
- Add a new datasource called Prometheus and set it to the prometheus type.
- Set the URL to http://<PRIVATE_IP>:9090.

Add the Docker Dashboard to Grafana:

Import the new dashboard using the JSON file located here.
Set Prometheus to the datasource you created.
Set the quick range to Last 5 minutes.

Add an Email Notification Channel:

Create a new Email Notification Channel.
- Name the channel Email.
- Add your email address to the notification.

Create an Alert for CPU Usage:

Edit the CPU Usage graph.
- On the Metrics tab, create a new query: sum(rate(process_cpu_seconds_total[1m])) * 100
- Hide it from view.
- Set the alert condition to use the new query. It should be E in the dropdown menu.
- Set the Is Above condition to 75.
- In the Notification section, set the alert to Email Alerts.
- The message should say “CPU usage is over 75%”.
- Save your changes.

What are Hands-on Labs

Hands-on Labs are real environments created by industry experts to help you learn. These environments help you gain knowledge and experience, practice without compromising your system, test without risk, destroy without fear, and let you learn from your mistakes. Hands-on Labs: practice your skills before delivering in the real world.