Lab
A Cloud Guru

Using Grafana with Prometheus for Alerting and Monitoring

For the last six months, the Acme Anvil Corporation has been migrating some of their bare metal infrastructure to Docker containers. The team has set up Prometheus using cAdvisor to help monitor containers. After running Prometheus in the production environment for a few weeks, the team has decided to add Grafana for visualization and alerting.

Try for free Contact sales

Path Info

Level

Intermediate

Duration

1h 30m

Published

Nov 07, 2018

Challenge

Configure Docker.

Open port 9323 in the firewall.

firewall-cmd --zone=public --add-port=9323/tcp

Open /etc/docker/daemon.json and add the following.

{
  "metrics-addr" : "0.0.0.0:9323",
  "experimental" : true
}

Challenge

Update `prometheus.yml`.

Edit the prometheus.yml file in the /root directory.

vi ~/prometheus.yml

Change the contents to reflect the following:

scrape_configs:
  - job_name: prometheus
    scrape_interval: 5s
    static_configs:
    - targets:
      - prometheus:9090
      - node-exporter:9100
      - pushgateway:9091
      - cadvisor:8080

  - job_name: docker
    scrape_interval: 5s
    static_configs:
    - targets:
      - PRIVATE_IP_ADDRESS:9323

Challenge

Update `docker-compose.yml`.

Edit docker-compose.yml in the /root directory.

vi ~/docker-compose.yml

Change the contents to reflect the following:

version: '3'
services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    ports:
      - 9090:9090
    command:
      - --config.file=/etc/prometheus/prometheus.yml
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
    depends_on:
      - cadvisor
  cadvisor:
    image: google/cadvisor:latest
    container_name: cadvisor
    ports:
      - 8080:8080
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
  pushgateway:
    image: prom/pushgateway
    container_name: pushgateway
    ports:
      - 9091:9091
  node-exporter:
    image: prom/node-exporter:latest
    container_name: node-exporter
    restart: unless-stopped
    expose:
      - 9100
  grafana:
    image: grafana/grafana:5.3.0
    container_name: grafana
    ports:
      - 3000:3000
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=password
    depends_on:
      - prometheus
      - cadvisor

Challenge

Install the Docker and System Monitoring dashboard.

Click the plus sign (+) on the left side of the Grafana interface. Click Import. Copy the contents of the JSON file included in the lab instructions.

Paste the contents of the JSON file into the Import screen of the Grafana interface, and click Load. In the upper right-hand corner, click on Refresh every 5m and select Last 5 minutes.

Author

A Cloud Guru

The Cloud Content team comprises subject matter experts hyper focused on services offered by the leading cloud vendors (AWS, GCP, and Azure), as well as cloud-related technologies such as Linux and DevOps. The team is thrilled to share their knowledge to help you build modern tech solutions from the ground up, secure and optimize your environments, and so much more!

What's a lab?

Hands-on Labs are real environments created by industry experts to help you learn. These environments help you gain knowledge and experience, practice without compromising your system, test without risk, destroy without fear, and let you learn from your mistakes. Hands-on Labs: practice your skills before delivering in the real world.