Skip to content


  1. Compromised AWS IAM credentials ~ 25-40 mins
  2. Compromised EC2 instance ~ 25-40 mins

Module 3: Detect, Investigate & Respond

Unfortunately, due to a misconfiguration in your environment, an attacker may have been able to gain access to the web server. You are getting alerts from the security services you’ve put in place indicating malicious activity. These alerts include communication with known malicious IP addresses, account reconnaissance, changes to an Amazon S3 bucket configuration, and disabling security configurations. You must identify what activity the intruder may have performed and how they did it so you can block the intruder’s access, remediate the vulnerabilities, and restore the configuration to its proper state.

Part 1 - Compromised AWS IAM credentials

Detect and investigate

By now you’ve received email alerts from the security services you enabled. Now what? As part of your risk driven detection strategy your organization has decided to prioritize AWS IAM related findings.

  1. Sort through your email alerts and identity an alert related to an AWS IAM principal

Amazon GuardDuty Finding: UnauthorizedAccess:IAMUser/MaliciousIPCaller.Custom

  1. Copy the <Access Key ID> from the e-mail alert.

Explore findings related to the access key (Amazon GuardDuty)

Now that you have a resource identifier to pivot from you can use Amazon GuardDuty to start doing an initial investigation into these findings.

  1. Go to the Amazon GuardDuty console (us-west-2).

  2. Click in the Add filter criteria box, select Access Key ID, and then paste in the <Access Key ID> you copied from the e-mail, then select Apply.

    What findings do you see related to this Access Key ID?

  3. Click on one of the findings to see the details.

    What principal are these credentials associated with?

  4. Examining User type under Resource affected you can see that the access key referenced in this finding is from an IAM assumed role.

  5. Examining Principal ID under Resource affected you will find two strings separated by a colon. The first is the unique ID for the IAM role and the second is the EC2 instance ID.

You may have to resize your screen by dragging the middle vertical scrollbar to the left to see the entire text

  1. The Principal ID contains a unique ID for the entity making the API request, and when the request is made using temporary security credentials (which is what happens for an assume role call) it also includes a session name. In this case the session name is the EC2 instance ID since the assume role call was done using an IAM role for EC2.

  2. Copy the full Principal Id which contains both the unique ID of the role and the session name: "principalId": "< unique ID >:< session name >"

  3. Examine the User Name under Resource affected and copy it down. This corresponds to the name of the IAM role involved since the temp creds used to make the API call came from EC2 instance with an IAM role attached.


Now that you have identified that a temporary security credential from an IAM role for EC2 is being used by an attacker, the decision has been made to rotate the credential immediately to prevent any further misuse or potential privilege escalation.

Revoke the IAM role sessions (IAM)

  1. Browse to the AWS IAM console.

  2. Click Roles and find the role you identified in the previous section using the User Name you copied down earlier (this is the role attached to the compromised instance), and click on that Role Name.

  3. Click on the Revoke sessions tab.

  4. Click on Revoke active sessions.

  5. Click the acknowledgement check box and then click Revoke active sessions.

    What is the mechanism that is put in place by this step to actually prevent the use of the temporary security credentials issued by this role?

Restart the EC2 instance to rotate the access keys (EC2)

All active credentials for the compromised IAM role have been invalidated. This means the attacker can no longer use those access keys, but it also means that any applications that use this role can't as well. You knew this going in but decided it was necessary due to the high risk of a compromised IAM access key. In order to ensure the availability of your application you need to refresh the access keys on the instance by stopping and starting the instance. A simple reboot will not change the keys. If you waited the temporary security credential on the instance would be refreshed but this procedure will speed things up. Since you are using AWS Systems Manager for administration on your EC2 instances you can use it to query the metadata to validate that the access keys were rotated after the instance restart.

  1. In the EC2 console Stop the Instance named threat-detection-wksp: Compromised Instance.

Check the box next to the instance, select the Actions menu, Instance State, Stop, confirm by pressing Yes, Stop

  1. Wait for the Instance State to say stopped under Instance State (you may need to refresh the EC2 console) and then Start the instance.

    You will need to wait until all Status Checks have passed before continuing.

Verify the access keys have been rotated (Systems Manager)

  1. Go to AWS Systems Manager console and click on Session Manager on the left navigation and then click Start Session.

    You should see an instance named threat-detection-wksp: Compromised Instance with a Instance state of running.

  2. To see the credentials currently active on the instance, click on the radio button next to threat-detection-wksp: Compromised Instance and click Start Session.

  3. Run the following command in the shell and compare the access key ID to the one found in the email alerts to ensure it has changed:


Why would this scenario be a good use case for auto-scaling groups?

At this point you've successfully revoked all the active sessions from AWS IAM role and rotated the temporary security credentials on the EC2 instance.

Part 2 - Compromised EC2 instance

Detect and investigate

Now that you've addressed the compromised IAM credential you need focus in on how the attacker was able to compromise the EC2 instance. It's this compromise which allowed them to query the instance metadata and steal the credentials.

Explore findings related to the instance ID (AWS Security Hub)

When investigating the compromised IAM credential you discovered that it was from an IAM role for EC2 and identified the EC2 instance ID from the principal ID of the finding. Using the instance ID (that you previously copied, it starts with an ‘i’, such as i-08fa26ffb15a66f5a) you can use AWS Security Hub to start investigating the findings. To start, you are going to research the GuardDuty findings related to the EC2 instance.

  1. Go to the AWS Security Hub console.
  2. The link should take you to the Findings section (if not, click on Findings in the navigation on the left).

    • Add a filter by clicking in the Add filter box and scrolling down to Product Name, and paste in the word GuardDuty.

    • Use your browser's find function Control-F and paste in the <Instance ID> you copied earlier (from the principal ID you gathered in the GuardDuty finding).

    • Now copy the Amazon Resource Name (ARN) from the Resource ID for the first match. The ARN will look something like this arn:aws:ec2:us-west-2:166199753942:instance/i-0efc5172a5d7ecc6b

    • Add one more filter by clicking the Add filter box again and selecting Resource ID and paste in the ARN from the previous step

    What GuardDuty findings do you see related to this instance ID?

One of the findings should indicate that the EC2 instance is communicating with an IP address on a threat list (disallowed IP) which adds further evidence to the conclusion that the instance has been compromised. The other finding should indicate that a system at a particular IP address is performing an SSH brute force attack against your instance. You now need to investigate if the SSH brute force attack was successful and if that is what allowed the attacker to gain access to the instance.

Determine if ssh password authentication is enabled on the EC2 instance (AWS Security Hub)

Automated responses to threats can do many things. For example, you could have an trigger that helps gather information about the threat that could then be used in the investigation by the security team. With that option in mind, we have a CloudWatch event rule in place that will trigger an Amazon Inspector scan of an EC2 instance when GuardDuty detects a particular attack. We will use AWS Security Hub to view the findings from Inspector. We want to determine if the SSH configuration adheres to best practices.

  1. Go to the AWS Security Hub console.
  2. The link should take you to the Findings section (if not, click on Findings in the navigation on the left).
    • Add a filter by clicking in the Add filter box and scrolling down to Product Name, and paste in the word Inspector.
    • Use your browser's find function Control-F and paste in password authentication over SSH
    • The finding may not be on the first page of findings, use the > to move to the next page.

Click on the finding regarding SSH and password authentication for the instance that experienced the SSH brute force attack and review.

If you do not see any findings after a while, there may have been an issue with your Inspector agent. Go to the Inspector console, click on Assessment Templates, check the template that starts with threat-detection-wksp, and click Run. Please allow 15 minutes for the scan to complete. You can also look in Assessment runs and check the status. Feel free to continue through this module and check the results later on.

After review you should see that password authentication over SSH is configured on the instance. In addition, if you examine some of the other Inspector findings you will see that there are no password complexity restrictions. This means the instance is more susceptible to an SSH brute force attack.

Determine if the attacker was able to login to the EC2 instance (CloudWatch logs)

Now that we know that the instance was more susceptible to an SSH brute force attack, let’s look at the CloudWatch logs and create a metric to see if there were any successful SSH logins (to finally answer the question of whether the SSH brute force attack was successful.) Your corporate policy is to send security certain logs from EC2 instances to CloudWatch.

  1. Go to CloudWatch logs.
  2. Click on the log group /threat-detection-wksp/var/log/secure
  3. If you have multiple log streams, filter using the Instance ID you copied earlier and click on the stream.
  4. Within the Filter Events text box put the following Filter Pattern: [Mon, day, timestamp, ip, id, msg1= Invalid, msg2 = user, ...]

    Do you see any failed (invalid user) attempts to log into the instance? Would that be consistent with an SSH brute force attack?

  5. Now replace the Filter with one for successful attempts: [Mon, day, timestamp, ip, id, msg1= Accepted, msg2 = password, ...]

    Do you see any successful attempts to log into the instance? Which linux user was compromised?


Modify the EC2 security group (EC2)

The active session from the attacker was automatically stopped by an update to the NACL on the subnet where the instance resides. This was done by a CloudWatch event rule trigger that is invoked based on certain GuardDuty findings. You've decided that all administration on EC2 Instances will be done through AWS Systems Manager so you no longer need administrative ports open so a good next step would be to modify the security group associated with the EC2 instance to prevent the attacker or anyone else from connecting.

  1. Go to the Amazon EC2 Console.

  2. Find the running instances with the name threat-detection-wksp: Compromised Instance.

  3. Under the Description tab, click on the Security Group for the compromised instance.

  4. View the rules under the Inbound tab.

  5. Click Edit and delete the inbound SSH rule.

    The SSM Agent was installed on your EC2 Instance during the initial configuration.

  6. Click Save

Congratulations! You have successfully remediated the incident and further hardened your environment. This is obviously a simulation and we can not cover every aspect of the response function in the short time allotted but hopefully this gave you an idea of the capabilities available on AWS to detect, investigate and respond to threats and attacks.

Here is a diagram of the attack you just investigated. Numbers 1 & 2 show the SSH brute force attack and successful SSH login. Number 3 shows the S3 bucket changes the attacker made. Number 4 shows the API calls the attacker made with the IAM temporary credentials stolen from the compromised EC2 instance. 03-diagram-attack

If you are going through this workshop in a classroom setting then the instructor should start the module 4 presentation soon.