For this lab, we will incorporate a few different services to build a simple monitoring and remediation system for accidental changes made to **production** autoscaling groups, while ignoring **development** changes. The architecture will utilize CloudTrail, EventBridge, EC2 (Autoscaling Groups), and Lambda.
We will find there are two EC2 autoscaling groups that have different `environment` tags attached, **prd** and **dev**. Using these tags in conjunction with EventBridge, we will create an Event Pattern that matches any EC2 termination changes made to any autoscaling group, which then will invoke a custom Lambda function. This Lambda function will complete some operational checks to see if the tags are set to **prd**, and if so, start resetting the desired capacity. If the tags are set to **dev**, then we will only print the instances and environment to the CloudWatch logs.
Learning Objectives
Successfully complete this lab by achieving the following learning objectives:
- Create an EventBridge Rule
We want to be able to understand the process of creating an EventBridge rule that watches for event patterns that we specify using available AWS CloudTrail API calls.
We want to look for an Auto Scaling service event called EC2 Instance Termination Successful within the account.
If you need to reference the event pattern for some help, you can find the solution pattern over in our GitHub Repo
- Configure a Lambda Target
Use the deployed Lambda function as a target for our EventBridge rule. The Lambda will parse the auto scaling event data that is sent to it and, if the group is tagged as prd, it will correct the changes immediately.
- Successfully Reset Production Changes
Once all the pieces are put together, we should now test that the system works.
When you scale down the Development auto scaling group, it should trigger the Lambda and only print logs to CloudWatch.
However, if you scale down the Production auto scaling group, it should trigger the Lambda, which will then do two things:
- Reset the baseline capacity to 1 instance
- Print the information to CloudWatch logs
