Application Monitoring with Prometheus

30 minutes
  • 7 Learning Objectives

About this Hands-on Lab

When it comes to monitoring, keeping track of our CPU, memory, and other infrastructure metrics often isn’t enough. Beyond the common infrastructure-based metrics we expose via tools such as the Node Exporter, we also want to instrument our application to send out metrics related to the app itself. This is done by using the Prometheus client library available to our application’s language.

Once we have the client library set up in our application, we can then define any metrics we wish to collect and write the code to support them. We can also have the Prometheus client record a number of default metrics already included in the library and available to us. Once finished, we’ll have added metrics for our to-do application that track added and removed tasks, as well as response times across our app.

Learning Objectives

Successfully complete this lab by achieving the following learning objectives:

Add the Prometheus client and track any default metrics.
  1. Move into the forethought directory:

    $ cd forethought

  2. Install the prom-client via npm, Node.js’s package manager:

    $ npm install prom-client –save

  3. Open the index.js file, where we’ll be adding all of our metrics code:

    $ $EDITOR index.js

  4. Require the use of the prom-client by adding it to our variable list:

    var express = require(‘express’);
    var bodyParser = require(‘body-parser’);
    var app = express();
    const prom = require(‘prom-client’);

    With prom being the name we’ll use when calling the client library.

  5. Enable default metrics scraping:

    const collectDefaultMetrics = prom.collectDefaultMetrics;
    collectDefaultMetrics({ prefix: ‘forethought_’ });

  6. Use Express to create the /metrics endpoint and call in the Prometheus data:

    app.get(‘/metrics’, function (req, res) {
    res.set(‘Content-Type’, prom.register.contentType);
    res.end(prom.register.metrics());
    });

Add the `forethought_tasks_added` metric.
  1. Define the metric:

    const tasksadded = new prom.Counter({
      name: 'forethought_tasks_added',
      help: 'The number of items added to the to-do list, total'
    });
  2. Call the new metric in the addtask post function so it increases by one every time the function is called while adding a task:

    // add a task
    app.post("/addtask", function(req, res) {
      var newTask = req.body.newtask;
      task.push(newTask);
      res.redirect("/");
      tasksadded.inc();
    });
Add the `forethought_tasks_complete` metric.
  1. Define the metric:

    const tasksdone = new prom.Counter({
      name: 'forethought_tasks_complete',
      help: 'The number of items completed'
    });
  2. Call the new metric in the /removetask post function so it increases by one every time the function is called:

    // remove a task
    app.post("/removetask", function(req, res) {
      var completeTask = req.body.check;
      if (typeof completeTask === "string") {
        complete.push(completeTask);
        task.splice(task.indexOf(completeTask), 1);
      }
      else if (typeof completeTask === "object") {
        for (var i = 0; i < completeTask.length; i++) {
          complete.push(completeTask[i]);
          task.splice(task.indexOf(completeTask[i]), 1);
          tasksdone.inc()
        }
      }
      res.redirect("/");
    });
Add the `forethought_current_tasks` metric.
  1. Define the metric:

    const taskgauge = new prom.Gauge({
      name: 'forethought_current_tasks',
      help: 'Amount of incomplete tasks'
    });
  2. Add an increase to the /addtask method:

    // add a task
    app.post("/addtask", function(req, res) {
      var newTask = req.body.newtask;
      task.push(newTask);
      res.redirect("/");
      tasksadded.inc();
      taskgauge.inc();
    });
  3. Add a decrease to the /removetask method:

    // remove a task
    app.post("/removetask", function(req, res) {
      var completeTask = req.body.check;
      if (typeof completeTask === "string") {
        complete.push(completeTask);
        task.splice(task.indexOf(completeTask), 1);
      }
      else if (typeof completeTask === "object") {
        for (var i = 0; i < completeTask.length; i++) {
          complete.push(completeTask[i]);
          task.splice(task.indexOf(completeTask[i]), 1);
          tasksdone.inc();
          taskgauge.dec();
        }
      }
      res.redirect("/");
    });
  4. Save and exit.

Add the `forethought_response_time_summary` metric.
  1. Add the response-time module:

    $ npm install response-time --save
  2. Reopen the index.js file.

  3. Define the metric:

    const responsetimesumm = new prom.Summary ({
      name: 'forethought_response_time_summary',
      help: 'Latency in percentiles',
    });
  4. Add response-time:

    var responseTime = require('response-time');
  5. Write the code to retrieve the response time:

    app.use(responseTime(function (req, res, time) {
      responsetimesumm.observe(time);
    }));
Add the `forethought_response_time_histogram` metric.
  1. Define the metric:

    const responsetimehist = new prom.Histogram ({
      name: 'forethought_response_time_histogram',
      help: 'Latency in history form',
      buckets: [0.1, 0.25, 0.5, 1, 2.5, 5, 10]
    });
  2. Call the histogram in the existing response-time code:

    app.use(responseTime(function (req, res, time) {
      responsetimesumm.observe(time);
      responsetimehist.observe(time);
    }));
Reimage the application and redeploy the container; add to Prometheus.
  1. Stop the current Docker container for our application:

    $ docker stop ft-app
  2. Remove the container:

    $ docker rm ft-app
  3. Remove the image:

    $ docker image rm forethought
  4. Rebuild the image:

    $ docker build -t forethought .
  5. Deploy the new container:

    $ docker run --name ft-app -p 80:8080 -d forethought
  6. Switch to the monitoring server.

  7. Add our application endpoint to Prometheus (replacing PRIVATE_IP with the private IP for the Application server listed on the lab page):

    $ sudo $EDITOR /etc/prometheus/prometheus.yml
    
      - job_name: 'forethought'
        static_configs:
        - targets: ['PRIVATE_IP:80']

    Save and exit.

  8. Restart Prometheus:

    $ sudo systemctl restart prometheus

Additional Resources

Your company has recently set up Prometheus to work as the monitoring solution, and you, specifically, have been assigned to create the metrics needed to monitor the company's flagship application, Forethought, a task-tracking application written in Node.js. The application itself is located on the application server, within the home directory, where you'll be doing most of the work. Prometheus itself is located on the monitoring server.

The metrics needed have already been decided — you just need to write the code to implement them based on the provided information:

Goals

General

  • All tasks must have the prefix forethought_.
  • A /metrics endpoint needs to be exposed.
  • Enable the collection of default metrics.
  • When finished, the application needs to be reimaged and Dockerized.
  • Add the new applicaton as an endpoint for Prometheus.

Metrics

forethought_tasks_added

  • Counter
  • Tracks the number of added tasks

forethought_tasks_complete

  • Counter
  • Tracks the number of completed tasks

forethought_current_tasks

  • Gauge
  • Tracks the amount of ACTIVE, INCOMPLETE tasks
  • Goes up when a task is added
  • Goes down when a task is completed

forethought_response_time_summary

  • Summary
  • Observes the response time of the application

forethought_response_time_histogram

  • Histrogram
  • Observes the response time of the application
  • Buckets: 0.1, 0.25, 0.5, 1, 2.5, 5, 10

What are Hands-on Labs

Hands-on Labs are real environments created by industry experts to help you learn. These environments help you gain knowledge and experience, practice without compromising your system, test without risk, destroy without fear, and let you learn from your mistakes. Hands-on Labs: practice your skills before delivering in the real world.

Sign In
Welcome Back!

Psst…this one if you’ve been moved to ACG!

Get Started
Who’s going to be learning?