A nightmare scenario for any engineering team is discovering that a misconfigured, vulnerable piece of cloud infrastructure enabled an attacker to exfiltrate sensitive customer data. The potential consequences are endless—lost revenue, fines, increased insurance costs, and legal settlements—and the nightmare is exponentially worse if the breach is only discovered a significant time after the actual compromise.
High-profile breaches can echo painfully far into the future. Perform a Google search for “Capital One,” and a top result, almost a year later, is still likely to be their 2019 breach, where an attacker utilized the company’s misconfigured infrastructure to access the personal data of over 100 million people.
How was the attacker able to steal vast reams of highly sensitive data? It was later determined that they apparently used a Server-Side-Request-Forgery (SSRF) attack to obtain the necessary access. SSRF attacks use a company’s own infrastructure against itself, imitating valid requests from within the network to access sensitive data. As more enterprises move to take advantage of the natural flexibility, scaling, and power offered by cloud platforms like AWS, the proliferation of these types of attacks will only increase. That’s why detecting and ultimately defeating SSRF attacks against cloud infrastructure should be a paramount priority for any organization.
What Is SSRF?
Server-Side-Request-Forgery (SSRF) attacks use a server to read or change internal resources such as APIs and databases. By “forging” what would otherwise be a normal request from the server, a hostile request can masquerade as a safe activity, thus injecting modified URLs or making requests from private data stores.
The target for an SSRF is a backend resource—for example, a database—that is not directly accessible to the attacker. However, the attacker can access a server which can communicate with the target. An SSRF attack is an attempt to abuse the server so that it sends a hostile request to the target on behalf of the attacker. Usually, the target has some level of implicit trust for the server, and will accept the request as being legitimate. Thus, the attacker gains access to a resource that normally is private.
Any server that accepts a URI as an input and returns the result might be vulnerable to an SSRF attack. Here’s a simplified illustration:
- Many internet-facing applications will accept a URI, retrieve data from it, and return the data to the requestor somehow. For example, social media platforms usually allow users to create posts based on external content. To do this, they must retrieve data from the external content’s URI, import it into a post, and make the post available to its users.
- An attacker wishes to wage an SSRF attack, and discovers (or brute-forces) a URI for a backend resource in the application’s internal network.
- The attacker abuses the application by submitting the backend URI (perhaps including a query string) as if it were for an external resource. If the application and backend network are not sufficiently secured against SSRF, the application might obediently retrieve private data from the resource and return it to the attacker.
As with many attacks, the initial entry into private infrastructure is enabled by a vulnerable piece of code running on a server that is exposed to the internet. Applications that do not sanitize or validate URL parameters are a prime target for this type of attack. Once an attacker discovers an application that will parse unsanitized values, they can then pivot that application and the underlying server to make forged requests on their behalf.
Services Vulnerable to SSRF Attacks
Often, backend services and APIs in cloud infrastructure are configured by default to implicitly trust requests from other parts of the infrastructure that exist within the bounds of the private network. Some examples of services that are generally not secure by default include:
- Elasticsearch SSL/TLS is not enabled by default, and data can be read in plaintext from HTTP REST APIs.
- Consul allows reading and writing to its data via HTTP API.
- MongoDB comes with an HTTP REST API by default.
- EC2 Instance Metadata Service (IMDS) is the vector that ultimately led to the Capital One breach. EC2 Instances have access to a universal, link-local metadata service.
It’s easy to imagine the depth of valuable information an attacker could glean from access to any one of these services. Elasticsearch and MongoDB often store critical customer and infrastructure data. Access to Consul means the attacker could learn in-depth information about the infrastructure and potentially cause destructive changes, like deleting data.
EC2 IMDS: Potential for disaster
For EC2 IMDS, AWS provides a unique WAF feature in a REST interface available at http://169.254.169.254, which provides access to configuration and authentication information on all EC2 instances. IMDS is a powerful piece of functionality, enabling configuration and management capabilities. It also can provide access to authentication data via the AWS IAM service and STS.
In the IAM authentication model, access is granted based on “roles,” which can be assigned to servers. In this way, engineering teams can grant the necessary access to other resources for applications running on EC2 instances, without having to cope with the additional complexity of managing secure credential data in the code itself. For example, a Java application might need to read and write to CSV files in an S3 bucket. Instead of having to load sensitive credentials at runtime, the underlying EC2 server can be given a role that allows access to the required S3 bucket, and the running application implicitly shares this access.
On a compromised server, with access to the IMDS endpoint, an attacker could forge this same request to the S3 bucket and be granted access to potentially sensitive data there. Any other service-access granted via the assigned role would potentially be at risk as well.
IMDS: An actual disaster
The discussion above isn’t just theoretical. The Capital One breach is a real-life example of how a savvy attacker can abuse EC2 IMDS to cause massive damage.
In this case, the attacker took advantage of a misconfigured WAF to relay a request to IMDS. The request was for the WAF’s credentials to backend S3 storage, which were returned to the attacker. The attacker then used the credentials to list the contents of the S3 storage, and then systematically exfiltrated it.
For this attack to succeed, several security holes had to exist.
- First, judging by the way that the attack was discovered, it seems that Capital One did not have adequate SSRF detection set up. (Someone stumbled across the exfiltrated data in a Github gist, four months after the attack was complete.)
- During the attack, the WAF relayed the SSRF attempt to IMDS, and returned its own credentials to the attacker.
- The WAF had been granted overly permissive credentials. If the WAF’s IAM role had been appropriately restricted, the attacker could not have used them to do much. But it wasn’t, so the attacker had broad latitude to iterate through S3 storage and remove data from it.
Let’s discuss the various ways to close these types of holes, and prevent SSRF attacks from succeeding.
Detecting and Preventing SSRF attacks on AWS
What framework should be put in place to deal with SSRF attacks? Ideally, any good security approach will be holistic and multi-pronged. It should include both detection and prevention.
SSRF detection with GuardDuty
AWS GuardDuty offers threat-detection as a service, with minimal technical investment to get up and running. Unfortunately, it’s not likely to detect an SSRF attack until after an attacker has already compromised credentials and used them in an attempt to access data. By that point, the damage has already been done.
Drilling down further, there are two general approaches to consider: Per-host detection and infrastructure-wide traffic mirroring.
Per-host detection methods generally involve deploying host-based intrusion detection systems, or some kind of agent that will forward or mirror network traffic to a centralized collector or aggregator. However, there are cost and performance implications for this approach. One of the primary drivers of cloud-cost growth is network transfer costs, and per-host mirroring has the potential to effectively double the cost surface.
An additional possibility is to utilize tools that are already present in the Linux toolchain, such as auditd or iptables. Most engineers should be familiar with iptables, the packet filter/firewall available on most Linux distributions. It can be configured to log and take actions against packet traffic according to text-based rules defined by the system administrator. Using knowledge about which user and process IDs should and should not be able to send traffic to the IMDS endpoint, rules could be applied that would log unauthorized access attempts. At that point, existing log aggregation can be parsed for events indicating a potential attack.
Auditd is a daemon for logging low-level system calls, and with proper configuration can log calls made to the underlying sockets that represent the IMDS.
Detection via infrastructure-wide traffic mirroring
Infrastructure-wide traffic mirroring involves enabling VPC traffic mirroring in an AWS account. You can then set up an IDS with VXLAN decapsulation capabilities, such as Suricata or Zeek, to monitor the ENI interface, at which point you can configure alerts for potential attack event signatures in the relevant logs.
To borrow an old saying: “An ounce of prevention is worth a pound of cure.” The triage, cleanup, and assessment of a successful attack is a time-consuming and expensive process. Taking the time to architect preventative measures early on will pay off big time down the road.
Prevention starts with the software itself. Infosec should be coordinating with development and DevOps teams to help ensure there is a strong framework of resources to help enable well-architected and secure software. Web-facing architecture in particular should receive heavy focus. Sanitizing inputs with a focus on URL parsing can help block an important vector for SSRF attacks.
That last point is crucial, and it bears repeating with emphasis:
Ensure that your web-facing applications sanitize their inputs.
Many frameworks include validation libraries. Use them! Also ensure that validation is done by the server application; even if client-side validation is performed, this can be bypassed by an attacker, so don’t rely on it.
Sanitization should account for possible obfuscation/encoding of hostile inputs. And it goes without saying that the rulesets should be able to detect and block requests that include the IMDS address (169.254.169.254).
Consider your infrastructure’s design
If a planned service or stack is still in the planning phase, there may be an opportunity to make design choices around the infrastructure itself that will assist in reducing the attack surface. Stateless, web-facing compute workloads could be migrated from EC2 instances to Lambda functions, eliminating an entire class of OS-based vulnerabilities.
Audit (and if necessary, correct) your IAM structure
The Capital One breach took advantage of an overly generous set of permissions for a WAF.
AWS has a rich IAM featureset that allows fine-grained control over identity and access permissions. Every application and service in your network should have an appropriate role assigned, and every role should have only the least amount of permissions necessary.
AWS has developed Version 2 of the Instance Metadata System, which uses a session token to ensure that requests for data are secure and authorized. This prevents many common vectors for SSRFs which rely on open WAFs, open reverse proxies, and similar vulnerabilities. If you aren’t already using IMDSv2, you should map out a strategy to adopt it soon.
Adopt a robust web security solution
AWS includes AWS WAF and AWS Shield, which have some ability to block some types of hostile traffic. However, they are not comprehensive security solutions, nor are they meant to be. Furthermore, SSRF attacks are merely one among a broad spectrum of threats on today’s Internet.
A complete web security solution such as Reblaze includes a next-generation WAF, multilayer DDoS protection, advanced bot management, API security, a mobile/native client SDK, scraping and data theft prevention, real-time traffic analysis, and other features to secure web applications, services, and APIs. This is an important part of security for AWS workloads.
Plan and take action
There isn’t a single magic bullet for either detection or prevention of SSRF attacks. A multi-layered, vigilant approach has the best chance of helping an enterprise maintain a solid security posture.
Yes, this requires some detailed planning and tedious work, especially in the beginning. But consider this question:
If your organization does have an SSRF vulnerability, wouldn’t you prefer to discover it yourself now… instead of learning about it later on Twitter? Food for thought.
Questions or comments? Feel free to contact us.