A single Prometheus server can provide a great deal of value in terms of monitoring. However, it can also become a single point of failure if something goes wrong. Luckily, the simple architecture employed by Prometheus makes it relatively easy to set up additional Prometheus servers in a highly-available configuration. In this lab, you will see what this looks like as you build your own highly-available, multi-instance Prometheus setup.
Learning Objectives
Successfully complete this lab by achieving the following learning objectives:
- Copy the Prometheus configuration from the existing Prometheus server to the new instance.
Log in to Prometheus Server 1.
Copy
prometheus.ymlto Prometheus Server 2.
scp /etc/prometheus/prometheus.yml cloud_user@10.0.1.102:/home/cloud_userLog in to Prometheus Server 2.
Copy
prometheus.ymlto the appropriate location.
sudo cp ~/prometheus.yml /etc/prometheus/prometheus.yml- Copy the rules configuration from the existing Prometheus server to the new instance.
- On Prometheus Server 1, view
prometheus.ymlto determine the rules location(s) underrule_files:
cat /etc/prometheus/prometheus.yml- Copy files from the rules directory to Prometheus Server 2:
scp /etc/prometheus/rules/* cloud_user@10.0.1.102:/home/cloud_user- On Prometheus Server 2, create the rules directory, then copy the rules file to the appropriate location:
sudo mkdir -p /etc/prometheus/rules sudo cp ~/limedrop-alerts.yml /etc/prometheus/rules- On Prometheus Server 1, view
- Start the new Prometheus instance and verify that everything is working.
- On Prometheus Server 2, start and enable Prometheus:
sudo systemctl enable prometheus sudo systemctl start prometheus- Access Prometheus Server 2 in a browser at
http://<Prometheus Server 2 Public IP>:9090. Run a query to verify that it is scraping metrics fromlimedrop-web:
up{instance="limedrop-web:9100"}- You can also click
Alertsto verify that theWebServerDownalert appears.
