I haven’t completed the course yet so maybe it’s addressed later, but I wanted to verify that the first step in responding to a compromised EC2 instance is to stop the instance immediately.
Shouldn’t the first step be to remove all security groups from the instance and place it in a quarantined security group with zero access in or out? Wouldn’t most malicious activity that’s impacting others be contained in this way? I understand that there could be exceptions (maybe if the activity is quickly running up your bill), but this way you could take the snapshot while it’s booted in order to preserve the in-memory data.
If you stop an EC2 instance via AWS console or AWS CLI, you stop all activity on the VM.
This not only stops the network activity, but also prevents that a malicious program 1) notices the detection and 2) removes traces of the its activity (e.g., cleaning log files). The latter is important for a forensic analysis of the attack.
Therefore, certainly, you can remove SG associations from the EC2 instance, but, I would highly recommend that you do it AFTER stopping the instance.
By the way, as far as I know, changing/removing SGs of a running EC2 instance won’t have any effect on already established TCP connections. It will only prevent establishing new connections. This is because SGs are stateful. With NACLs however, you can cut the network traffic, immediately. Keep in mind that NACLs are defined for subnets and not for (individual) EC2 instances, though.
Regarding your idea of preserving in-memory data, I assume you suggest creating a memory dump and not a snapshot, correct? Because neither EBS snapshots nor AMIs preserve in-memory data. If so, again, creating a memory dumps take time and can be detected by the malicious program. Therefore, it can remove its traces before the memory dump has been completed.
Without being forensic expert, I would think that losing in-memory data is acceptable, if you can prevent loosing the traces of the malicious activity (e.g., log files, executables, etc.)
Hey Reisberg and Rene,
I am also not a forensic expert, but I am reading through AWS Security Incident Response Guide (June2020) (Link: https://d1.awsstatic.com/whitepapers/aws_security_incident_response.pdf) and on page 40, under "InfrastructureDomain Incidents," AWS describes a 7 step process if "your monitoring solution notified you of a potential security anomaly on your Amazon EC2 instance." There is no mention of shutting down the EC2 instance.
Again, I do not know the correct answer, but maybe this information can add to the conversation.