Configuring Prometheus to Use Service Discovery

1 hour
  • 3 Learning Objectives

About this Hands-on Lab

Recently, your team has deployed Prometheus to the companies Kubernetes cluster. Now it is time to use service discovery to find targets for cAdvisor and the Kubernetes API. You have been tasked with modifying the Prometheus Config Map that is used to create the `prometheus.yml` file. Create the scrape config and add the jobs for `kubernetes-apiservers` and `kubernetes-cadvisor`. Then, propagate the changes to the Prometheus pod.

Learning Objectives

Successfully complete this lab by achieving the following learning objectives:

Configure the Service Discovery Targets

Edit prometheus-config-map.yml and add in the two service discovery targets:

vi prometheus-config-map.yml

When we’re done, the whole file should look like this:

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-server-conf
  labels:
    name: prometheus-server-conf
  namespace: monitoring
data:
  prometheus.yml: |-
    global:
      scrape_interval: 5s
      evaluation_interval: 5s

    scrape_configs:
      - job_name: 'kubernetes-apiservers'

        kubernetes_sd_configs:
        - role: endpoints
        scheme: https

        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

        relabel_configs:
        - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
          action: keep
          regex: default;kubernetes;https

      - job_name: 'kubernetes-cadvisor'

        scheme: https

        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

        kubernetes_sd_configs:
        - role: node

        relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
        - target_label: __address__
          replacement: kubernetes.default.svc:443
        - source_labels: [__meta_kubernetes_node_name]
          regex: (.+)
          target_label: __metrics_path__
          replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
Apply the Changes to the Prometheus Configuration Map

Now, apply the changes that were made to prometheus-config-map.yml:

kubectl apply -f prometheus-config-map.yml
Delete the Prometheus Pod
  1. List the pods to find the name of the Prometheus pod:

    kubectl get pods -n monitoring
  2. Delete the Prometheus pod:

    kubectl delete pods <POD_NAME> -n monitoring
  3. Open up a new web browser tab, and navigate to the Expression browser. This will be at the public IP of the lab server, on port 8080:

    http://<IP>:8080
  4. Click on Status, and select Target from the dropdown. We should see two targets in there.

Additional Resources

  1. Elevate your permissions to root. Execute ./bootstrap.sh to complete the setup of the environment.
  2. Edit prometheus-config-map.yml to create two service discovery targets.
  3. Create a job called kubernetes-apiservers.
    • The role should be set to endpoint and the scheme should be set to https.
    • Configure tls_config to use /var/run/secrets/kubernetes.io/serviceaccount/ca.crt as the CA file, and /var/run/secrets/kubernetes.io/serviceaccount/token on the bearer token file.
    • Relabel __meta_kubernetes_namespace, __meta_kubernetes_service_name, and __meta_kubernetes_endpoint_port_name.
    • Make sure these source labels are kept, and set default, kubernetes, and https for the RegEx.
  4. Create a second job called kubernetes-cadvisor.
    • Set the scheme to https.
    • Configure tls_config to use /var/run/secrets/kubernetes.io/serviceaccount/ca.crt as the CA file and /var/run/secrets/kubernetes.io/serviceaccount/token on the bearer token file.
    • Set the role to node.
    • Configure three relabel settings:
    • Create a labelmap that will remove __meta_kubernetes_node_label_ from the label name.
    • Create a target label that will replace the address with kubernetes.default.svc:443.
    • Finally, create a target label that will replace the metrics path with /api/v1/nodes/${1}/proxy/metrics/cadvisor and set the __meta_kubernetes_node_name source label as the value of ${1}.
  5. Reload the Prometheus pod by deleting it.
  6. Verify that two service discovery endpoints are appearing as targets.
What are Hands-on Labs

Hands-on Labs are real environments created by industry experts to help you learn. These environments help you gain knowledge and experience, practice without compromising your system, test without risk, destroy without fear, and let you learn from your mistakes. Hands-on Labs: practice your skills before delivering in the real world.

Get Started
Who’s going to be learning?

How many seats do you need?

  • $499 USD per seat per year
  • Billed Annually
  • Renews in 12 months

Ready to accelerate learning?

For over 25 licenses, a member of our sales team will walk you through a custom tailored solution for your business.


$2,495.00

Checkout
Sign In
Welcome Back!
Thanks for reaching out!

You’ll hear from us shortly. In the meantime, why not check out what our customers have to say about ACG?