Security and IAM

Capital One Breach: Key Takeaways to Better Secure Your AWS Environment

Several clients have asked us what they can do to help avoid a breach similar to the very public and recent Capital One breach.
Josh von Schaumburg Featured
Josh von Schaumburg | Aug 14 2019

Wed, 14 Aug 2019

Here at Trek10, several clients have asked us what they can do to help avoid a breach similar to the very public and recent Capital One breach. After researching the topic and gathering the data points over the last couple of weeks, we have pulled together some recommendations. Before jumping in, note that this blog post is not a deep dive on what happened in the actual attack. We will do a quick review, but this post is intended to serve as a guide on what actions enterprises can take, in their AWS environments, to mitigate the risk of this type of attack.

As always, Brian Krebs provides a great overview and more detail in a follow up. The indictment itself is also a great read!

As is often the case with a breach, there are two components to the attack — there is the initial attack on the perimeter, followed by lateral movement across the environment for the data exfiltration. In this case, the initial attack occurred when a public-facing EC2 instance, hosting the open source Modsec WAF, was breached. The vulnerability that allowed this initial breach is unknown, which means a virtually infinite number of possible vulnerabilities could have allowed the attacker in. Based on the coverage, it’s very unlikely that this was a zero day vulnerability, which means it probably could have been avoided with appropriate security controls.

Once the attacker has breached the EC2 instance, all AWS services that the EC2 instance has access to via the IAM policy of the IAM role that is attached to the EC2 instance profile (known as “*****-WAF-Role” in the indictment) are now available to the attacker via AWS’s metadata service.

This could easily break off into an entirely separate blog post, but we’ll try to make it quick! Since the attack, much controversy has surrounded AWS’s metadata service (which allows any user with terminal access to an instance to query to gain access to temporary IAM credentials.

The metadata service provides AWS IAM credentials for EC2 instances to access other AWS services without having to manually create IAM access keys and hardcode them into your application -- this is a huge security win. Due to this attack, EC2 roles and the AWS metadata service have been widely criticized as major AWS vulnerabilities; we feel this is far from the truth.

In reality, this service allows for automated key rotation and removes the requirement of hard-coding access keys into application code in order to access AWS services. Are there ways that Amazon could improve the security of the metadata service? Of course, security should always be improving, but the removal of this service altogether would lead to far less secure applications.

These AWS services are accessible both from the terminal of the EC2 instance, as well as from the attacker’s local environment outside the instance… but the keys are temporary, rotated automatically multiple times per day, and will quickly expire if used locally).

Interestingly, the data exfiltration was completed from a VPN provider’s IP address using the S3 Sync command. GuardDuty, AWS’s threat detection service, provides a high severity notification if the credentials associated with an EC2 instance profile are used outside of the EC2 instance. At Trek10, we integrate GuardDuty into our ticketing system (both internally and for clients) such that high severity alerts create a ticket for review.

One of the key discussion points related to the breach is around whether or not the IAM policy that was attached to *-WAF-Role was overly permissive. It is not clear why the Modsec instance required access to S3. We have not found any public details related to this, but given that we know the attacker ran the ListBuckets command and then the Sync command, we can assume that the IAM policy of the role looks something like this (though it could have very well been scoped down to specific S3 buckets):

{ "Effect": "Allow", "Actions": [ "s3:ListBuckets", "s3:ListAllMyBuckets", "s3:ListBuckets", "s3:GetObject" ], "Resource": "*"}

How to Mitigate Risk from this Attack

This section outlines an overview of recommendations for your AWS environment, based on the attack, as well as some details on how Trek10 implements best security practices into its CloudOps offering. The recommendations are broken down based on whether the suggestion mitigates risk from the initial attack on the perimeter or on the lateral attack that allowed the S3 data exfiltration. We will start with the lateral attack, as that is far more interesting due to the public information available!

Preventing Lateral Attacks on S3 Data

Remember that AWS IAM is an extremely powerful service (one that we take very seriously at Trek10 — just check out our approach to IAM escalation to understand how seriously!). The key takeaway from this breach is to understand that the power of AWS IAM means that organizations need to take the principle of least privilege very seriously and ensure that all IAM entities are scoped appropriately. Below are some recommendations based on the breach:

  • Conduct an IAM audit.
    • Review the IAM policies attached to all IAM entities
    • Trek10 pulls a wide variety of data with its custom IAM enumeration scripts to help our clients implement the principle of least privilege across all IAM entities.
    • We are also always developing additional tooling and integrations to support security, like our Excess Access Exorcist.

You may even find that there are numerous IAM entities that are not being used, remove them by first deactivating any keys to make sure nothing breaks, then DELETE THEM!

  • Enable AWS GuardDuty, which has numerous detection capabilities. One capability that is quite relevant to this breach, as discussed previously, is its ability to identify if credentials obtained as part of the EC2 metadata service have been used outside of the EC2 host.

GuardDuty should be integrated into your ticketing system, as we do for our clients. We also provide the option of notifying on high severity results only, for example, to reduce noise.

  • Use Amazon Macie
    • From the docs: The fully managed service continuously monitors data access activity for anomalies, and generates detailed alerts when it detects risk of unauthorized access or inadvertent data leaks. Today, Amazon Macie is available to protect data stored in Amazon S3, with support for additional AWS data stores coming later this year.

Because Amazon Macie is quite expensive, we are experimenting with some different approaches to detecting and preventing S3 data exfiltration for our clients (for example, if the S3 API calls a certain percentage of all S3 objects in an account, cut off access to the authenticating IAM entity until a manual approval is provided).

Preventing Initial Attack on EC2 Instance

Because the public knows so little about that the initial attack on the EC2 instance, these “mitigations” are just general good EC2 security practices — with a particular focus on how an organization can utilize AWS’s vast arsenal of security services to protect EC2 instances:

  • Conduct a firewall audit
    • Configure hardened security groups / network ACLs to close all unnecessarily opened ports.
    • Trek10 uses its custom scripting to enumerate security groups into a report for validation.
  • Better yet, use infrastructure as code everywhere with security testing integrated into your CI/CD pipeline to detect any drift. With our AWS DevOps Competency, Trek10 has world renowned expertise in building CI/CD pipelines.
  • Regularly scan your EC2 instances with AWS Inspector to detect any vulnerabilities and conduct remediation.
    • Trek10 integrates AWS Inspector reports into its ticketing system for remediation.
  • Enable AWS GuardDuty, already mentioned previously, for intrusion detection.
    • Trek10 integrates GuardDuty alerts into its ticketing system and even automates the blocking of attacking IP addresses with AWS WAF.
  • Ensure your systems are up to date with AWS Patch Manager via Systems Manager.
    • Trek10 applies patching best practices to Patch Manager, which integrates with its ticketing system.

Trek10’s CloudOps service integrates a wide variety of AWS security services into its ticketing system and automates the deployment with pre-built templates. Additionally, Trek10 is in the process of building an advanced, cloud native security detection and prevention service. We are combining AWS native security services and the absolute best 3rd party commercial and open source security tools for full protection of EC2 instances and cloud native AWS services. We are partnering with Sophos Cloud Optix for this offering, and we are excited to release this service later this year. Be on the look out for a future blog post!

Josh von Schaumburg Featured
Josh von Schaumburg