Hi, the suggested steps in this sections are missing the memory acquisition, as first step the instance should be isolated to then acquire the memory, stopping the instance will cause loosing valuable evidence that may never be written to disk/log (especially if there are no EBS volumes), also there is no mention of triage, live/offline forensics, key/privilege usage check and rotation/revocation. A dedicated vpc would be great but changing security groups to allow only access from the IR/forensic analyst workstation will suffice.
I will be good if a training on AWS_IR could be added maybe as a separate training (like ALB training)
6 Answers
Have an Incident Response/Forensic Security Group and enclave prepared and ready.
If an instance becomes compromised, change its security group to become a member of the Incident Response/Forensic Security Group; with automation associated with this change, and it will become untouched and quarantined immediately where automation can also begin forensics activities in real time.
Only allow SSH (22) or RDP (3389) ingress rules for the Incident Response/Forensic Security Group where you have a bastion host prepared to conduct forensic activities. No egress.
Only just seen this before posting my almost identical recommendation in another thread.
Using this tool automates this process and will indeed also take a memory snapshot before shutting down!
I came here to make comments about "stopping the instance" and I am glad a lot of people already called that out!
This is a great discussion, IMHO I think what you do first all depends on the situation, e.g the criticality of your application, the sensitivity of your data and the level of risk or loss you are facing. In a lot of cases you would probably want to act quickly to avoid allowing a malicious actor to continue accessing your system.
In some situations it may be more important to prevent any further damage being done. In others, the ability to perform forensics using a snapshot of the RAM may take precedent. The minutes spent creating your snapshot and copying it to another location might allow a hacker to do greater damage in your account, so personally I think you would need to weigh up the consequences, and also have a well documented and communicated process for capturing RAM so that all operations staff know the procedure, to avoid wasting valuable time discussing what the process is, which tool or commands to use etc…
However, AWS err on the side of caution and recommend:
If you are unable to identify and stop unauthorized activity on your EC2 instance, we recommend that you terminate the compromised EC2 instance and replace it with a new instance as needed.
https://docs.aws.amazon.com/guardduty/latest/ug/guardduty_remediate.html
I just noticed that there’s a fairly new technical guide which discusses incident response at length:
https://d1.awsstatic.com/whitepapers/aws_security_incident_response.pdf
They mention a few third party tools, but are careful to say that
The following links to third-party tools are external and are not endorsed by AWS. AWS
offers no guarantees or representations of any kind about these tools or pages.
_• AWSIR – Python installable command line utility for mitigation of host and key
compromises
• MargaritaShotgun – Remote Memory Acquisition Tool
• ThreatPrep – Python module for evaluation of AWS account best practices
around incident handling readiness
_• ThreatResponse Web – Web based analysis platform for use with the AWSIR
command line tool.
• GRR Rapid Response – Remote live forensics for incident response
• Linux Write Blocker – The kernel patch and user-space tools to enable Linux
software write blocking
The MargaritaShotgun might be one to consider, allowing you to capture system memory and save it to S3 – but note that they are not recommending it as such…
Great info, thanks!
Stopping the instance should NOT be the first thing you should do. I would take a snapshot, then dump the ram, copy that, and THEN stop the instance.
i agree! you stop the instanace you will lose the connection the hacker has into the machine and any other /tmp data . Ryan needs to sort that video out i think he rushed it.