Monitoring in Kubernetes with Prometheus and Grafana

1.5 hours
  • 2 Learning Objectives

About this Hands-on Lab

Monitoring is an essential part of the overall CI/CD picture. To deploy frequently, you need to be able to have confidence that if a deployment breaks something, you will be able to identify the problem and respond quickly to minimize the impact on users. In this activity, you will learn to install and configure Prometheus and Grafana in a Kubernetes cluster, and you will set up some basic Grafana dashboards to give you insight into the performance of the cluster and the applications running in it.

Learning Objectives

Successfully complete this lab by achieving the following learning objectives:

Install Prometheus in the Kubernetes cluster.

To do this, make sure you have cloned the Kubernetes charts repo:

cd ~/
git clone https://github.com/kubernetes/charts
cd charts
git checkout efdcffe0b6973111ec6e5e83136ea74cdbe6527d
cd ../

Create a prometheus-values.yml with this content:

  alertmanager:
    persistentVolume:
      enabled: false
  server:
    persistentVolume:
      enabled: false

Use helm to install Prometheus with prometheus-values.yml:

helm install -f ~/prometheus-values.yml ~/charts/stable/prometheus --name prometheus --namespace prometheus
Install Grafana in the Kubernetes cluster.

To do this, make sure you have cloned the kubernetes charts repo:

cd ~/
git clone https://github.com/kubernetes/charts

Create a grafana-values.yml with this content (you will use this password to log in to Grafana):

adminPassword: somePassword

Use helm to install Grafana with grafana-values.yml:

helm install -f ~/grafana-values.yml ~/charts/stable/grafana --name grafana --namespace grafana

Additional Resources

Your team is building the train schedule app. They currently have it running on a Kubernetes cluster, but they need to monitor the performance of the cluster and the applications running on it. You have been tasked with installing and setting up Prometheus to aggregate data and Grafana to display this data. Both can be installed on the Kubernetes cluster itself. To make sure everything is working, you will need to create two dashboards in Grafana:

  1. Import the Kubernetes All Nodes community dashboard to display basic metrics about the Kubernetes cluster.
  2. Create a new Dashboard and add a graph showing requests per minute for the train-schedule app.

To accomplish this, you will need to:

  • Initialize helm with: helm init --wait
  • Clone the Kubernetes standard charts git repo and checkout a specific commit:
    git clone https://github.com/kubernetes/charts
    cd charts
    git checkout efdcffe0b6973111ec6e5e83136ea74cdbe6527d
    cd ../
  • Create a prometheus-values.yml for prometheus to turn off persistent storage:
    alertmanager:
    persistentVolume:
      enabled: false
    server:
    persistentVolume:
      enabled: false
  • Use helm to install prometheus in the prometheus namespace:
    helm install -f ~/prometheus-values.yml ~/charts/stable/prometheus --name prometheus --namespace prometheus
  • Create a grafana-values.yml for grafana to set an admin password:
    adminPassword: password
  • Use helm to install grafana in the grafana namespace:
    helm install -f ~/grafana-values.yml ~/charts/stable/grafana --name grafana --namespace grafana
  • Deploy a NodePort service to provide external access to grafana. Make a file called grafana-ext.yml:
    kind: Service
    apiVersion: v1
    metadata:
    namespace: grafana
    name: grafana-ext
    spec:
    type: NodePort
    selector:
      app: grafana
    ports:
    - protocol: TCP
      port: 3000
      nodePort: 8081

    And deploy the service:

    kubectl apply -f ~/grafana-ext.yml
  • Log in to grafana at <Kubernetes Node Public IP>:8081.
  • Add a datasource for prometheus. The type should be set to Prometheus and the url is http://prometheus-server.prometheus.svc.cluster.local.
  • Add the Kubernetes All Nodes community dashboard with id 3131.
  • Create a new dashboard and add a requests per minute graph for the train-schedule app. You can use the following query:
    sum(rate(http_request_duration_ms_count[2m])) by (service, route, method, code)  * 60

What are Hands-on Labs

Hands-on Labs are real environments created by industry experts to help you learn. These environments help you gain knowledge and experience, practice without compromising your system, test without risk, destroy without fear, and let you learn from your mistakes. Hands-on Labs: practice your skills before delivering in the real world.

Get Started
Who’s going to be learning?

How many seats do you need?

  • $499 USD per seat per year
  • Billed Annually
  • Renews in 12 months

Ready to accelerate learning?

For over 25 licenses, a member of our sales team will walk you through a custom tailored solution for your business.


$2,495.00

Checkout
Sign In
Welcome Back!

Psst…this one if you’ve been moved to ACG!