I haven’t completed the course yet so maybe it’s addressed later, but I wanted to verify that the first step in responding to a compromised EC2 instance is to stop the instance immediately.
Shouldn’t the first step be to remove all security groups from the instance and place it in a quarantined security group with zero access in or out? Wouldn’t most malicious activity that’s impacting others be contained in this way? I understand that there could be exceptions (maybe if the activity is quickly running up your bill), but this way you could take the snapshot while it’s booted in order to preserve the in-memory data.
Thanks!
2 Answers
Hi Reisberg,
If you stop an EC2 instance via AWS console or AWS CLI, you stop all activity on the VM.
This not only stops the network activity, but also prevents that a malicious program 1) notices the detection and 2) removes traces of the its activity (e.g., cleaning log files). The latter is important for a forensic analysis of the attack.
Therefore, certainly, you can remove SG associations from the EC2 instance, but, I would highly recommend that you do it AFTER stopping the instance.
By the way, as far as I know, changing/removing SGs of a running EC2 instance won’t have any effect on already established TCP connections. It will only prevent establishing new connections. This is because SGs are stateful. With NACLs however, you can cut the network traffic, immediately. Keep in mind that NACLs are defined for subnets and not for (individual) EC2 instances, though.
Regarding your idea of preserving in-memory data, I assume you suggest creating a memory dump and not a snapshot, correct? Because neither EBS snapshots nor AMIs preserve in-memory data. If so, again, creating a memory dumps take time and can be detected by the malicious program. Therefore, it can remove its traces before the memory dump has been completed.
Without being forensic expert, I would think that losing in-memory data is acceptable, if you can prevent loosing the traces of the malicious activity (e.g., log files, executables, etc.)
Kind regards,
Rene
Hey Reisberg and Rene,
I am also not a forensic expert, but I am reading through AWS Security Incident Response Guide (June2020) (Link: https://d1.awsstatic.com/whitepapers/aws_security_incident_response.pdf) and on page 40, under "InfrastructureDomain Incidents," AWS describes a 7 step process if "your monitoring solution notified you of a potential security anomaly on your Amazon EC2 instance." There is no mention of shutting down the EC2 instance.
Again, I do not know the correct answer, but maybe this information can add to the conversation.
-Clark
@Layclark thanks for the link but it missed a _ the one that work is https://d1.awsstatic.com/whitepapers/aws_security_incident_response.pdf The paper is clear in the next page (41) that both options (offline and online) are valid and it will be a forensics investigation decision. I’d say with the increasing sophistication of the attacks, the online ( volatile data capture) is increasingly very important but it will depend on the scale of the damage being done and how big the instance memory is etc. On page 42 and 43 it discusses on how to approach memory capture including automating it via tagging, cloud watch events, lambda, shell event, SSM, and hibernation as an option. Both will need some prep work eg install mem tools like rekall or have hibernate enabled (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Hibernate.html#hibernation-enabled-AMI) . Not sure all this is relevant for the exam, just FYI. For the exam i keep in mind acloudguru’s answer for offline investigations and the isolation etc for online.