Prometheus is a great way to collect a variety of metrics for your servers and applications. However, sometimes you need to perform calculations on the raw data Prometheus collects in order to gain actionable information. While queries can help you do this, recording rules provide an additional layer of optimization by allowing you to periodically pre-calculate query results and save them. In this lab, you will have the opportunity to implement some basic recording rules. This will provide you with some hands-on familiarity with the process of creating and managing recording rules.
Learning Objectives
Successfully complete this lab by achieving the following learning objectives:
- Implement a Recording Rule to Pre-Calculate CPU Usage
Log in to the Prometheus server.
Create a directory to store rules files:
sudo mkdir -p /etc/prometheus/rulesEdit the Prometheus config and add a
ruleslocation:sudo vi /etc/prometheus/prometheus.yml... rule_files: - "/etc/prometheus/rules/*.yml" ...Create a
rulesfile forlimedrop-gatewayserver metrics:sudo vi /etc/prometheus/rules/limedrop-gateway.ymlImplement a recording rule to pre-calculate CPU usage for the server:
groups: - name: limedrop_gateway rules: - record: limedrop_gateway:cpu_usage expr: sum(rate(node_cpu_seconds_total{instance='limedrop-gateway:9100',mode!='idle'}[5m])) * 100 / 2Restart Prometheus to load the new configuration:
sudo systemctl restart prometheusAccess the expression browser (
http://<PROMETHEUS_SERVER_PUBLIC_IP>:9090/graph) and run a query to verify you can see the pre-calculated data:limedrop_gateway:cpu_usage[5m]
- Add a Recording Rule to Pre-Calculate Memory Usage
Edit the rules file:
sudo vi /etc/prometheus/rules/limedrop-gateway.ymlAdd a recording rule to pre-calculate memory usage for the server:
groups: - name: limedrop_gateway rules: - record: limedrop_gateway:cpu_usage expr: sum(rate(node_cpu_seconds_total{instance='limedrop-gateway:9100',mode!='idle'}[5m])) * 100 / 2 - record: limedrop_gateway:memory_usage expr: 100 - (node_memory_MemAvailable_bytes{instance='limedrop-gateway:9100'} / node_memory_MemTotal_bytes{instance='limedrop-gateway:9100'}) * 100Restart Prometheus to load the new configuration:
sudo systemctl restart prometheusAccess the expression browser (
http://<PROMETHEUS_SERVER_PUBLIC_IP>:9090/graph) and run a query to verify you can see the pre-calculated data:limedrop_gateway:memory_usage[5m]You can also check out the rules interface (
http://<PROMETHEUS_SERVER_PUBLIC_IP>:9090/rules) to see the status of your rules.
