What is rate limiting? Why is it such a vital part of effective web security, and what’s important to know about its configuration within a security solution? Those questions, and more, are discussed in this article.
What is Rate Limiting?
Rate limiting is the identification of hostile traffic sources by the rates at which they are attempting to access resources within a network. A web security solution monitors the frequency and timing of incoming requests from each requestor. When a given requestor exceeds the specified rate limit, that requestor is blocked from further access for a specified length of time.
For example, it is common to set up rate limits on a login page. Almost all legitimate users will be able to complete the login process in one, two, or possibly three attempts. However, when one specific “user” tries to login dozens of times within a few minutes, and fails each time, that user is probably not legitimate. Instead, it is probably an illicit actor trying to stuff or discover login credentials, and attempting to gain unauthorized access to a user account. Rate-limiting the login page is a straightforward way to block this malicious activity.
Why is Rate Limiting So Important?
Rate limiting is crucially important in the modern threat environment. There are numerous forms of attack which are primarily detectable through monitoring the rates of incoming requests. Here are five of them.
DDoS Attacks: DDoS is an attack which attempts to make the targeted system unavailable to its intended users. The attacker hopes to overwhelm the victim’s network with a massive amount of incoming traffic and take the targeted servers down.
DDoS is the most obvious situation that rate limiting can mitigate. When a specific traffic source submits too many requests in a given time, then the web security solution should ban that source from future network access.
However, DDoS attacks can be challenging in a way that other attacks are not. The first “D” in DDoS stands for “Distributed”; requests will come from a large number (potentially millions) of different IP addresses, so any given requestor will not necessarily violate the rate limit. In this situation, the security solution must discern that despite the variety of originating geolocations, these requests are nevertheless part of the same attack. They must be treated as if they originated from a single requestor, and all of these incoming requests must be blocked.
Credential Stuffing: When threat actors breach a large site, they often harvest a significant trove of credential sets. Soon afterwards, they will send credential stuffing bots to other sites, attempting to use the stolen credentials to login elsewhere. This can have a high success rate, because unfortunately, many Internet users re-use their username/password combinations across different sites.
By their nature, these bots will access a login page over and over, as they try different credential sets. Rate limiting is an effective way to detect and block them, thus preventing this form of ATO (account takeover) attack.
Brute force attacks: These are cruder forms of credential attacks. Here the attacker tries to guess a login credential by systematically submitting different variations until one is found that works. A well-secured network will have credential requirements (e.g., password length and composition) that makes a brute-force attack unlikely to succeed. Nevertheless, a large attack can consume a lot of network resources, and so it’s important to block them. Rate limiting works well for this.
Site scraping and data theft: In some verticals, scraping and data theft are rampant, and blocking these activities is crucially important. For example, in ecommerce, competitors often try to steal pricing data from each other. Content aggregators who sell access to data must prevent unauthorized access, as this is the basis of their business model. And so on.
Scraper bots attempt to access and copy as much data from the targeted application as they can. Rate limiting can detect this, and thwart the scraping.
Inventory Denial (a.k.a. Inventory Hoarding). Threat actors use inventory-denial bots to begin transactions (e.g., purchasing products from ecommerce sites, making reservations in travel applications, etc.), without ever completing them. This removes the items from available inventory, and prevents legitimate customers from buying. Rate limiting can detect these activities and block the bots.
Configuring Effective Rate Limiting
As with any process that can filter incoming traffic, rate limiting must be used correctly.
If the limits are set too low, this will create false positive alarms. Some legitimate users will be excluded from the protected application.
If the limits are set too high, the consequences can be even worse; it will create false negative alarms. Instead of being excluded from the network, some attackers will be allowed access. The outcome of this will depend on the type of attack: everything from extra resource consumption to loss of private data to compromise of user accounts.
Setting correct rate limits requires careful analysis of historical data, the selection of an appropriate algorithm for that analysis, making the results available to the user for monitoring and manual fine-tuning if desired, and more.
Reblaze includes mechanisms to do all of this automatically, along with other important features (such as granular event conditions, combinable rate limits for a single URL, autobanning of malicious traffic sources who violate rate limits after they reset, and more).
There is much more that could be said about selecting and optimizing rate limits for different situations. This will be the topic of an upcoming white paper from Reblaze.
Meanwhile, we’d be happy to give you a demo of the platform, including not only intelligent rate limiting (and dedicated DDoS protection) but also a next-generation WAF, advanced bot management, API security, and more. Contact us for a demo.
