Lab
A Cloud Guru

Monitoring a Batch Job with Prometheus Pushgateway

Prometheus Pushgateway provides a way to provide metrics to Prometheus with a push-based model. This is particularly useful for monitoring short-lived job processes. In this lab, you will have the opportunity to work with the Pushgateway API by pushing metrics to it. You will modify a simple job to implement monitoring for the job by pushing metrics to Pushgateway every time it runs.

Try for free Contact sales

Path Info

Level

Intermediate

Duration

45m

Published

Jun 12, 2020

Challenge

Modify the Cleanup Job to Push a Metric to Pushgateway to Signal When It Runs
1. Log in to the Job server.
2. Edit the cleanup job script:
```
sudo vi /etc/jobs/cleanup.sh
```
3. Implement a call to the Pushgateway API at the end of the script to signal that the job ran:
```
num_files=$(rm -vrif /etc/debug_data/* | wc -l)

cat << EOF | curl --data-binary @- http://prometheus:9091/metrics/job/debug_cleanup/instance/10.0.1.102
  # TYPE job_executed_successful gauge
  job_executed_successful 1
EOF
```
4. Access the Prometheus expression browser (http://<PROMETHEUS_SERVER_PUBLIC_IP>:9090/graph), and run a query to verify you can see metric data pushed to Pushgateway by the cleanup job script. Note you may have to wait a minute or so for the job to execute so you can see your changes.
```
job_executed_successful[5m]
```
Challenge

Modify the Cleanup Job to Push a Metric to Pushgateway Representing the Number of Files Deleted in Each Execution
1. Edit the cleanup job script again:
```
sudo vi /etc/jobs/cleanup.sh
```
2. Implement a call to the Pushgateway API at the end of the script to signal that the job ran:
```
num_files=$(rm -vrif /etc/debug_data/* | wc -l)

cat << EOF | curl --data-binary @- http://prometheus:9091/metrics/job/debug_cleanup/instance/10.0.1.102
  # TYPE job_executed_successful gauge
  job_executed_successful 1
  # TYPE job_num_files_deleted gauge
  job_num_files_deleted $num_files
EOF
```
3. Access the expression browser again and verify you can see the new metric in Prometheus. Note you may have to wait a minute or so for the job to execute so you can see your changes.
```
job_num_files_deleted[5m]
```

Author

A Cloud Guru

The Cloud Content team comprises subject matter experts hyper focused on services offered by the leading cloud vendors (AWS, GCP, and Azure), as well as cloud-related technologies such as Linux and DevOps. The team is thrilled to share their knowledge to help you build modern tech solutions from the ground up, secure and optimize your environments, and so much more!

What's a lab?

Hands-on Labs are real environments created by industry experts to help you learn. These environments help you gain knowledge and experience, practice without compromising your system, test without risk, destroy without fear, and let you learn from your mistakes. Hands-on Labs: practice your skills before delivering in the real world.