Using Prometheus Recording Rules

45 minutes
  • 2 Learning Objectives

About this Hands-on Lab

Prometheus is a great way to collect a variety of metrics for your servers and applications. However, sometimes you need to perform calculations on the raw data Prometheus collects in order to gain actionable information. While queries can help you do this, recording rules provide an additional layer of optimization by allowing you to periodically pre-calculate query results and save them. In this lab, you will have the opportunity to implement some basic recording rules. This will provide you with some hands-on familiarity with the process of creating and managing recording rules.

Learning Objectives

Successfully complete this lab by achieving the following learning objectives:

Implement a Recording Rule to Pre-Calculate CPU Usage
  1. Log in to the Prometheus server.

  2. Create a directory to store rules files:

    sudo mkdir -p /etc/prometheus/rules
  3. Edit the Prometheus config and add a rules location:

    sudo vi /etc/prometheus/prometheus.yml
    ...
    
    rule_files:
      - "/etc/prometheus/rules/*.yml"
    
    ...
  4. Create a rules file for limedrop-gateway server metrics:

    sudo vi /etc/prometheus/rules/limedrop-gateway.yml
  5. Implement a recording rule to pre-calculate CPU usage for the server:

    groups:
    - name: limedrop_gateway
      rules:
      - record: limedrop_gateway:cpu_usage
        expr: sum(rate(node_cpu_seconds_total{instance='limedrop-gateway:9100',mode!='idle'}[5m])) * 100 / 2
  6. Restart Prometheus to load the new configuration:

    sudo systemctl restart prometheus
  7. Access the expression browser (http://<PROMETHEUS_SERVER_PUBLIC_IP>:9090/graph) and run a query to verify you can see the pre-calculated data:

    limedrop_gateway:cpu_usage[5m]
Add a Recording Rule to Pre-Calculate Memory Usage
  1. Edit the rules file:

    sudo vi /etc/prometheus/rules/limedrop-gateway.yml
  2. Add a recording rule to pre-calculate memory usage for the server:

    groups:
    - name: limedrop_gateway
      rules:
      - record: limedrop_gateway:cpu_usage
        expr: sum(rate(node_cpu_seconds_total{instance='limedrop-gateway:9100',mode!='idle'}[5m])) * 100 / 2
      - record: limedrop_gateway:memory_usage
        expr: 100 - (node_memory_MemAvailable_bytes{instance='limedrop-gateway:9100'} / node_memory_MemTotal_bytes{instance='limedrop-gateway:9100'}) * 100
  3. Restart Prometheus to load the new configuration:

    sudo systemctl restart prometheus
  4. Access the expression browser (http://<PROMETHEUS_SERVER_PUBLIC_IP>:9090/graph) and run a query to verify you can see the pre-calculated data:

    limedrop_gateway:memory_usage[5m]
  5. You can also check out the rules interface (http://<PROMETHEUS_SERVER_PUBLIC_IP>:9090/rules) to see the status of your rules.

Additional Resources

Your company, LimeDrop, is using Prometheus to monitor several servers. One such server, limedrop-gateway, is running an API gateway application. The admin team has a dashboard set up to view performance statistics for this server.

This means the queries calculating these statistics are executed every time the dashboard refreshes. You have been asked to optimize your usage of Prometheus resources by building some recording rules to pre-calculate some of the performance statistics for the limedrop-gateway server.

Implement the requested recording rules and query their data to make sure they are working.

Here are the details:

  • The instance name for the server is limedrop-gateway:9100.

  • Build a recording rule to pre-calculate CPU usage data for the server. Store the results in a metric called limedrop_gateway:cpu_usage. You can use this query to get CPU usage:

    sum(rate(node_cpu_seconds_total{instance='limedrop-gateway:9100',mode!='idle'}[5m])) * 100 / 2
  • Build a recording rule to pre-calculate memory usage data for the server. Store the results in a metric called limedrop_gateway:memory_usage You can use this query to get memory usage:

    100 - (node_memory_MemAvailable_bytes{instance='limedrop-gateway:9100'} / node_memory_MemTotal_bytes{instance='limedrop-gateway:9100'}) * 100

What are Hands-on Labs

Hands-on Labs are real environments created by industry experts to help you learn. These environments help you gain knowledge and experience, practice without compromising your system, test without risk, destroy without fear, and let you learn from your mistakes. Hands-on Labs: practice your skills before delivering in the real world.

Get Started
Who’s going to be learning?

How many seats do you need?

  • $499 USD per seat per year
  • Billed Annually
  • Renews in 12 months

Ready to accelerate learning?

For over 25 licenses, a member of our sales team will walk you through a custom tailored solution for your business.


$2,495.00

Checkout
Sign In
Welcome Back!

Psst…this one if you’ve been moved to ACG!