In this hands-on lab, we get our monitoring solution up and running by installing Prometheus, Alertmanager, and Grafana, as well as ensuring metrics from all three services are being fed into Prometheus. This gives us a full monitoring platform so we can start performing both infrastructure and application monitoring and everything in between. As we install Prometheus and Alertmanager, we’ll also configure our environment so we can easily and effectively change our configurations without any added work.
Learning Objectives
Successfully complete this lab by achieving the following learning objectives:
- Set up Prometheus
- On the
monitoring
server, create a user for Prometheus. - Create the needed directories:
/etc/prometheus
and/var/lib/prometheus
. - Set the ownership of the
/var/lib/prometheus
directory. - Download Prometheus (here’s the link).
- Extract the files.
- Move the configuration file to
/etc/prometheus
and set the owner to theprometheus
user. - Move the binaries and set the owner.
- Create the service file.
- Start Prometheus and make sure it automatically starts on boot.
- On the
- Set up Alertmanager
- On the
monitoring
server, create thealertmanager
system user. - Create the needed directory:
/etc/alertmanager
. - Download Alertmanager (here’s the link).
- Extract the files.
- Move the binaries.
- Set the ownership of the binaries.
- Move the configuration file into the
/etc/alertmanager
directory. - Set the ownership of the
/etc/alertmanager
directory. - Create the
alertmanager.service
file for systemd. - Stop Prometheus, and then update the Prometheus configuration file to use Alertmanager.
- Reload systemd, and then start the
prometheus
andalertmanager
services. - Make sure
alertmanager
starts on boot.
- On the
- Set up Grafana
- On the
grafana
server, install the prerequisitelibfontconfig
package. - Download and install Grafana using the
.deb
package provided on the Grafana download page. - Start Grafana.
- Ensure Grafana starts at boot.
- Using the Grafana IP address listed on the lab page, access Grafana’s web UI by navigating to
<GRAFANA_IP_ADDRESS>:3000
in a new browser tab. - Log in with the username
admin
and the passwordadmin
. Reset the password when prompted. - Click Add data source on the home page.
- Select Prometheus.
- In the HTTP section, set the URL to
http://<MONITORING_IP_ADDRESS>:9090
(using the public IP of themonitoring
server provided on the lab page). - Click Save & Test.
- On the
- Add Alertmanager and Grafana Endpoints to Prometheus
- On the
monitoring
server, open the Prometheus configuration file. - Add the Alertmanager endpoint.
- Add the Grafana endpoint.
- Restart Prometheus.
- Using the public IP address of the
monitoring
server, navigate to the Prometheus web UI in a new browser tab:http://<MONITORING_IP_ADDRESS>:9090
- Ensure all three endpoints are listed on the Targets page.
- On the