Frontiers of Bot Detection
Previous articles in this series described:
- The high amount (almost 40 percent) of malicious bots in web traffic today, and the damage that they can do.
- The most common types of bot attacks.
- The worst bot threats for different industries.
- Why traditional detection methods are increasingly ineffective.
- State-of-the-art bot detection methods that can identify even the latest, most sophisticated forms of automated traffic.
As hostile bots continue to grow more powerful, security solutions must also evolve. In this article, we’ll discuss the frontiers of bot detection: some of the current research that we are doing at Reblaze, and new techniques we’re developing to achieve optimal business outcomes in bot management.
Among our areas of investigation, there are three that will be discussed below:
- Performance optimization
- Business outcome optimization
- New sources of data
Over the last few years, bot detection methods have evolved considerably. The approaches used today can be categorized as follows:
- Static rules
- Statistical analysis
- Machine Learning (ML)
- Combination of ML and statistical analysis
This list represents a rough progression from less-effective approaches that are fast and inexpensive, to more-powerful analysis methods that are slower and resource-intensive. For an IDS (Intrusion Detection System) to have optimal performance in bot detection, it must apply the correct mix of these four approaches. It should use as much as possible of the earlier methods for fast detection, with sufficient use of the latter methods to ensure high accuracy during bot mitigation.
The simplest form of detection relies on the enforcement of static rulesets. These can be based on ACLs (Access Control Lists), sizing limits, time limits, blacklisting of IPs, and so on.
This approach requires minimal processing, and thus it is fast and cheap. However, static rules can be evaded fairly easily by the latest bots. As discussed previously, bots can change IP addresses, reduce the speed at which they submit requests, and so on.
An additional disadvantage is that overreliance on static rule enforcement will tend to increase the rate of FP (False Positive) alarms. As rulesets become more stringent, legitimate users will begin to violate them inadvertently. For example, a reduction in the allowable number of requests per session will flag users with longer sessions, and they will be incorrectly identified and blocked as bots.
Nevertheless, static rules still play a useful role in bot detection. Moderately-strict rulesets can still detect a large percentage of hostile bots with a very low rate of FPs. And there’s no reason to expend excessive computing workloads on catching these bots when it’s still possible to detect them using low-overhead methods.
Statistical analysis of incoming traffic requires more processing than static rule enforcement, but it is still straightforward to implement.
Applying widespread statistical tests such as Student’s t-test, Z-test, and analysis of variance (ANOVA) can often give transparent results. The base of these methods is a calculation of standard deviations and determination of borders between normal and abnormal numbers of chosen metrics.
These analyses are usually run on aggregated features or raw parameters extracted from traffic logs. Unlike static rule enforcement, statistical traffic analysis does not occur in real time. However, statistical analysis can be used to create ‘local’ or custom rules, which are then enforced in real time along with static rulesets. An ideal IDS will have minimal lag between the receipt of each request and its subsequent analysis, and it will frequently update its ‘local’ rules as needed.
Although applying ML to bot detection is not a new idea, many security solutions still are not doing this. Nevertheless, it is a necessity today due to the increasing sophistication of hostile bots. Today’s ‘smart’ attackers adapt to static limitations, minimize statistical anomalies, and otherwise program their bots to avoid detection.
Earlier ML algorithms were based on a search for anomalies and baseline plotting. The conventional method for outliers detection is RANSAC (i.e., “random sample consensus.” RANSAC is an iterative method to estimate parameters of a mathematical model from a set of observed data that contains outliers, when outliers are to be accorded no influence on the values of the estimates). The general criteria for dataset separation were abnormal and normal behaviors.
Today, anomalies have much more complex foundations. Bot attacks are difficult to detect because many bots have adopted algorithms to imitate human behavior. In other words, an IDS needs to detect bots which mask themselves as legitimate users.
As with statistical approaches, ML analysis is performed on aggregated results and then translated into rules for one-step usage in real time. However, ML requires larger datasets; every extracted feature (x) requires 10x observations. This makes ML a bit less flexible than purely statistical approaches. ML is also significantly more technical; it can be difficult for users to make optimal choices for feature significance, data distribution, and so on.
Some Machine Learning algorithms can see hidden interconnections (nonobvious features) in the data. In the case of unsupervised techniques, they can help to do a customized selection of significant features connected to dataset-specific characteristics.
This makes ML more effective than statistical methods for detecting sophisticated bots which are masquerading as human users.
Combining ML and Statistical Techniques
When choosing an approach for bot detection, it’s important to understand that a perfect, universally applicable method cannot exist. Even if theoretical results would give a perfect correlation with real-life datasets, threat actors would develop a new generation of bots which would leverage techniques to circumvent any static rules. (Indeed, as the next section will discuss, we shouldn’t even expect a universally applicable standard for assessing bot detection methods.)
In general, a combination of approaches such as static rules, statistical analysis, and ML will be the best solution for real-world applications. Some bots can still be blocked with fast, inexpensive methods. More sophisticated bots can then be detected with more intensive analysis. The challenge for IDS providers is to build this flexibility into their platforms with an accessible interface that doesn’t require too much ML expertise from users.
Optimizing for Business Outcomes
When evaluating a bot detection solution, what metric(s) should be used?
Accuracy is the most intuitive performance measure, and it is straightforward to calculate. It is the ratio of correctly predicted observations to the total observations.
It is tempting to seek an “ultimate” bot detection algorithm that is the most accurate of all possible algorithms, with the intent of applying it across all possible situations. However, such an algorithm does not exist. Different use cases are best addressed with different approaches, and customizing an algorithm for one use case makes it less effective in others.
Further, accuracy as described above is an incomplete metric. False Positive (FP) and False Negative (FN) events must also be considered.
When an FP alarm occurs, a legitimate user request is misidentified by the IDS as being hostile, and is therefore blocked. An FN is the inverse of this: a malicious request is incorrectly identified as being legitimate, and it is allowed through.
Thus, a complete measurement of accuracy becomes:
Accuracy = (TP+TN) / (FP+TP+FN+TN)
(Where TP and TN represent True Positive and True Negative events, respectively.)
However, even this improved metric is not necessarily enough to produce optimal business outcomes. One might think that a higher level of accuracy is always better, but this is not always the case.
FP and FN events are both undesirable, and an IDS should minimize them both. However, they are not necessarily equal; one can have much harsher consequences than the other.
For example, in retail, FPs can create direct losses of revenue when customers are prevented from buying. Meanwhile, for other organizations, FNs might have worse consequences. For example, if they facilitate the exposure of sensitive customer data from an organization in a highly-regulated industry, this can result in punitive fines.
Therefore, when individual organizations fine-tune their IDS performance, they must consider more than just overall accuracy.
For example, it’s possible that the highest-possible accuracy is accompanied with minimal FPs but a higher level of FNs, as compared to a slightly lower accuracy which includes higher FPs but lower FNs. In an industry where FNs are a serious matter, the slightly lower accuracy might be more desirable.
We see then that for an IDS to produce optimal results for an organization, it should be customized for each use case. The definition of “optimal” will vary from organization to organization, and indeed, even within the same organization it can vary across web applications.
Historically, the ability to customize an IDS to this level has been rare. For a web security solution to offer full customization, it must expose a variety of algorithmic parameters to users, while providing in-depth feedback about the resulting performance changes, and do it all in an understandable UI that doesn’t overwhelm users with details.
This task is obviously not easy. But as bot detection becomes more technically challenging, and as the relative impacts of FPs and FNs become potentially more variable (with the still-rising commercial importance of the web, an increasingly restrictive regulatory environment surrounding consumer privacy, etc.), web security solution providers will need to add this type of customization to their products.
Leveraging New Sources of Data
Traditional bot detection relies on data captured from the incoming traffic stream. However, this is not the only potential source of useful information.
Conversion rate (CR) is a rich source of insights into the success of a web application. It is the ratio of conversions to visitors or requests, where a “conversion” represents the attainment of a desired goal (e.g., a purchase, registration, download, etc.)
Conversions occur when an incoming request corresponds to a specified ‘target’ action within the web application (clicking on a “checkout” button, submitting a filled-in lead-generation form, and so on). Each time a target action occurs, the application’s conversion rate is positively affected.
Malicious bots usually do not perform typical target actions. Therefore, when an IDS correctly identifies and blocks bots, the web application’s conversions will not go down.
However, when an IDS generates FP alarms, then some legitimate users are being filtered as bots, and they are prevented from taking actions they would otherwise have taken. This decreases the CR.
This provides an opportunity to use application CR for feedback and algorithmic improvement. In other words, as potential “improvements” to the IDS are made (via adjustments to its algorithms, parameters, and so on), the CR can be monitored. If the adjustments caused the CR to decrease, then it’s quite possible that the FP rate increased (as shown in the example below). Further investigation should be done immediately.
Comparison of blocked requests to conversion rate. For hour 15, as more requests were blocked, conversion rate fell to near-zero. This indicates a probable high rate of FPs being generated by the IDS.
On the other hand, if IDS adjustments resulted in a higher percentage of requests being blocked, but the CR was not affected, then it would seem that the rate of false alarms has not changed, and the IDS adjustments were successful.
Comparison of blocked requests to conversion rate. There appears to be little correlation between blocked/passed requests and CR. The fluctuations in CR are thus attributable to other factors (e.g., normal variations in buyer behavior throughout the day), rather than being the result of FPs and FNs.
A brief aside: The discussion above assumes that conversion rate is calculated based upon all incoming requests, before the IDS determines which “visitors” are bots. If instead CR is calculated using legitimate visitors (i.e., the visitors remaining after the removal of requestors that were identified as bots), then the relationships described above become somewhat more complicated. For example, a change in the rate of FNs will not affect CR under the “all requests” approach, but it will affect CR under the “only legitimate requests” approach. There are other complicating factors as well. In any case, the overall point remains: CR is a useful metric for improving the effectiveness of an IDS.
Summary: network traffic logs are no longer the only source of data to analyze. The accuracy of an IDS’ bot detection can also be improved by integrating its UEBA with analytical platforms such as Google Analytics. Furthermore, this can provide insights into the rates of FP and FN generation.
As discussed previously in this article series, effective bot protection is crucial for any organization with significant web assets. However, IDS solutions which rely on older detection methods are becoming increasingly incapable of identifying today’s hostile bots. Modern approaches such as ML-based UEBA and granular Biometric Behavioral Profiling are essential tools for reliable bot control.
Further, as discussed in this article, there are many promising ways in which bot detection can evolve beyond its current forms. Reblaze is actively integrating these capabilities into its platform, while pushing forward into new areas of research.
This concludes our series on Bot Protection in 2019. There’s much more that could be said. If you’d like to discuss your organization’s situation, we’re happy to help. You can contact us here.
This article was the sixth and final part of a six-article series. You can download the complete report here: 2019 State of Bot Protection.