Creating a Recording Rule

1 hour
  • 3 Learning Objectives

About this Hands-on Lab

One of our Prometheus expressions in Grafana is running a bit slow. After a careful diagnosis, we’ve decided that the expression would be more effective if it was turned into a recording rule. We will need to configure our Prometheus environment to be able to use recording rules.

Learning Objectives

Successfully complete this lab by achieving the following learning objectives:

Create the Recording Rule
  1. Add the job:process_cpu_seconds:rate5m recording rule to prometheus-rules-config-map.yml. The file should look like this when we’re done editing it:
apiVersion: v1
kind: ConfigMap
metadata:
  creationTimestamp: null
  name: prometheus-rules-conf
  namespace: monitoring
data:
  process_rules.yml: |
    groups:
    - name: process_rules
      rules:
      - record: job:process_cpu_seconds:rate5m
        expr: (rate(process_cpu_seconds_total[5m]))
  1. Apply the changes with this command:
kubectl apply -f prometheus-rules-config-map.yml
Update the Prometheus Configuration
  1. Edit prometheus-config-map.yml

    vi prometheus-config-map.yml
  2. Add the rule files to prometheus-config-map.yml:

    rule_files:
    - /var/prometheus/rules/*_rules.yml
  3. It should look like this when we’re done:

    apiVersion: v1
    kind: ConfigMap
    metadata:
    name: prometheus-server-conf
    labels:
    name: prometheus-server-conf
    namespace: monitoring
    data:
    prometheus.yml: |-
    global:
      scrape_interval: 5s
      evaluation_interval: 5s
    
    rule_files:
    - /var/prometheus/rules/*_rules.yml
    
    scrape_configs:
      - job_name: 'kubernetes-apiservers'
    
        kubernetes_sd_configs:
        - role: endpoints
        scheme: https
    
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    
        relabel_configs:
        - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
          action: keep
          regex: default;kubernetes;https
    
      - job_name: 'kubernetes-nodes'
    
        scheme: https
    
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    
        kubernetes_sd_configs:
        - role: node
    
        relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
        - target_label: __address__
          replacement: kubernetes.default.svc:443
        - source_labels: [__meta_kubernetes_node_name]
          regex: (.+)
          target_label: __metrics_path__
          replacement: /api/v1/nodes/${1}/proxy/metrics
    
      - job_name: 'kubernetes-pods'
    
        kubernetes_sd_configs:
        - role: pod
    
        relabel_configs:
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
          action: keep
          regex: true
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
          action: replace
          target_label: __metrics_path__
          regex: (.+)
        - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
          action: replace
          regex: ([^:]+)(?::d+)?;(d+)
          replacement: $1:$2
          target_label: __address__
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)
        - source_labels: [__meta_kubernetes_namespace]
          action: replace
          target_label: kubernetes_namespace
        - source_labels: [__meta_kubernetes_pod_name]
          action: replace
          target_label: kubernetes_pod_name
    
      - job_name: 'kubernetes-cadvisor'
    
        scheme: https
    
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    
        kubernetes_sd_configs:
        - role: node
    
        relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
        - target_label: __address__
          replacement: kubernetes.default.svc:443
        - source_labels: [__meta_kubernetes_node_name]
          regex: (.+)
          target_label: __metrics_path__
          replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
    
      - job_name: 'kubernetes-service-endpoints'
    
        kubernetes_sd_configs:
        - role: endpoints
    
        relabel_configs:
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
          action: keep
          regex: true
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
          action: replace
          target_label: __scheme__
          regex: (https?)
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
          action: replace
          target_label: __metrics_path__
          regex: (.+)
        - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
          action: replace
          target_label: __address__
          regex: ([^:]+)(?::d+)?;(d+)
          replacement: $1:$2
        - action: labelmap
          regex: __meta_kubernetes_service_label_(.+)
        - source_labels: [__meta_kubernetes_namespace]
          action: replace
          target_label: kubernetes_namespace
        - source_labels: [__meta_kubernetes_service_name]
          action: replace
          target_label: kubernetes_name
  4. Apply the changes with this command:

    kubectl apply -f prometheus-config-map.yml
Add `prometheus-rules-conf` to the Prometheus Deployment
  1. Edit prometheus-deployment.yml and add the prometheus-rules-volume volume to it. When we’re finished, it should look like this:

    
    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
    name: prometheus-deployment
    namespace: monitoring
    spec:
    replicas: 1
    template:
    metadata:
      labels:
        app: prometheus-server
    spec:
      containers:
        - name: prometheus
          image: prom/prometheus:v2.2.1
          args:
            - "--config.file=/etc/prometheus/prometheus.yml"
            - "--storage.tsdb.path=/prometheus/"
            - "--web.enable-lifecycle"
          ports:
            - containerPort: 9090
          volumeMounts:
            - name: prometheus-config-volume
              mountPath: /etc/prometheus/
            - name: prometheus-storage-volume
              mountPath: /prometheus/
            - name: prometheus-rules-volume
              mountPath: /var/prometheus/rules
      volumes:
        - name: prometheus-config-volume
          configMap:
            defaultMode: 420
            name: prometheus-server-conf
    
        - name: prometheus-rules-volume
          configMap:
            name: prometheus-rules-conf
    
        - name: prometheus-storage-volume
          emptyDir: {}
    ---
    apiVersion: v1
    kind: Service
    metadata:
    name: prometheus-service
    namespace: monitoring
    annotations:
      prometheus.io/scrape: 'true'
      prometheus.io/port:   '9090'

spec:
selector:
app: prometheus-server
type: NodePort
ports:

  • port: 8080
    targetPort: 9090
    nodePort: 8080

  1. Apply the changes with this command:
    kubectl apply -f prometheus-deployment.yml

Additional Resources

  1. Become root, then execute the /root/bootstrap.sh script to complete the setup of the environment.

  2. Edit the Configuration Map prometheus-rules-config-map.yml, which is located in /root/prometheus.

    • Create a new file called process_rules.yml that will be managed by the configuration map.
    • Create a group called process_rules.
    • Create a rule called job:process_cpu_seconds:rate5m.
    • Use the rate function as the expression to calculate the process_cpu_seconds_total metric's per-second average rate of increase over a span of 5 minutes.
  3. Apply the changes to process_rules.yml.

  4. Now edit the ConfigMap prometheus-config-map.yml to include the rule files. The rules files section should include all rule files located in /var/prometheus/rules/ that end with _rules.yml without having to hard code the file.

  5. Apply the changes to prometheus-config-map.yml.

  6. Last of all, edit prometheus-deployment.yml. For your Prometheus container, add a new volume called prometheus-rules-volume that mounts /var/prometheus/rules.

    • In the volumes section, add a new volume called prometheus-rules-volume that maps to the prometheus-rules-conf Config Map.
  7. Apply the changes to prometheus-deployment.yml.

What are Hands-on Labs

Hands-on Labs are real environments created by industry experts to help you learn. These environments help you gain knowledge and experience, practice without compromising your system, test without risk, destroy without fear, and let you learn from your mistakes. Hands-on Labs: practice your skills before delivering in the real world.

Get Started
Who’s going to be learning?

How many seats do you need?

  • $499 USD per seat per year
  • Billed Annually
  • Renews in 12 months

Ready to accelerate learning?

For over 25 licenses, a member of our sales team will walk you through a custom tailored solution for your business.


$2,495.00

Checkout
Sign In
Welcome Back!

Psst…this one if you’ve been moved to ACG!