False Positive (FP) alarms are an important topic within cybersecurity. When FPs occur, legitimate users and customers are denied access to your systems. However, it’s obviously very difficult to identify FP alarms; if the IDS (Intrusion Detection System) could recognize them, the IDS wouldn’t be generating them in the first place.

The first article in this series (Using Machine Learning to Reduce False Positives While Combating Attacks, Part 1) discussed accuracy metrics for FP identification, and how to use Machine Learning (ML) to increase the accuracy of an IDS. Now in Part 2, we’ll build upon this further, and discuss some specific techniques for identifying and correcting FP alarms.

Threat Analysis and Detection

An IDS can use a variety of approaches for identifying hostile requests within incoming traffic. They include static rulesets, statistical anomaly detection, and ML-based analysis.

Static rulesets—ACLs (Access Control Lists), blocklists, rate limits, and so on—have many advantages. They are the simplest approach, and they require the least amount of security expertise from administrators. In fact, an IDS can rely upon public external sources for a large portion of its rulesets (e.g., the Spamhaus DROP lists can be used to filter traffic coming from netblocks controlled by cybercriminals). However, they are not capable of detecting the most sophisticated threats today, especially the latest hostile bots which mimic human users. Further, static rulesets have a tendency to increase FP alarms; as administrators add more rulesets, it becomes more likely that some legitimate traffic will match the specified characteristics.

Statistical anomaly detection is a more powerful approach than static rulesets, and usually it is more customized to the specific use case. It typically relies upon calculations of standard deviations and determination of borders between normal and abnormal numbers of chosen metrics. However, it is far more demanding to use, requiring significant time and expertise from the administrators.

ML-based analysis can provide the power of statistical anomaly detection, combined with ease of use and a high level of automation. Some ML algorithms can discover connections and relationships among traffic parameters that a human analyst would not notice. UEBA (User and Entity Behavioral Analytics) can be used to measure outcomes and feed back the metrics into the analysis. to modify its parameters and increase its performance over time. However this last point raises a question: how should performance be measured?

False Positive Alarms and Accuracy

At first glance, it would seem that optimal performance is achieved by a minimal rate of FP alarms. This rate can be easily calculated:

False Positive Rate = FP / (FP+TN)

…where TN is the number of True Negatives (legitimate events correctly identified as such).

However, this rate doesn’t provide a complete understanding. We should also consider the number of FNs (False Negatives), when malicious requests are incorrectly identified as being legitimate.

Thus, a better metric is accuracy. This is the ratio of correctly predicted observations to the total observations:

Accuracy = (TP+TN) / (FP+TP+FN+TN)

…which is calculated from TPs (True Positives, i.e. correctly identified hostile requests), TNs (correctly identified legitimate requests), FPs (legitimate requests incorrectly identified as being hostile), and FNs (hostile requests incorrectly identified as being legitimate).

It might seem that accuracy is the best metric to use for the performance of an IDS. Indeed, this would be true… if 100 percent accuracy could be achieved with a high rate of certainty.

In the real world, this is difficult to accomplish. Today’s threat actors are, in many cases, quite sophisticated, and they do their best to outwit their targets’ security systems. A modern IDS can achieve impressive rates of accuracy, but there can still be uncertainty about some of the decisions being made for edge cases.

Thus, it is best to acknowledge that perfect accuracy is probably not achievable with 100 percent certainty. In turn, organizations should carefully consider their best approach for optimizing IDS performance. In many cases, this means setting aside a quest for perfect accuracy, and optimizing instead for their particular situation, by choosing to emphasize the reduction of either FP or FN events.

Optimizing for Specific Use Cases

FP and FN events are both undesirable, and an IDS should minimize them both. However, it’s often more important to minimize one, even if it means accepting a potentially higher rate of the other.

For example, in online retail, FPs can prevent customers from buying, and create direct losses of revenue. Conversely, for other organizations in highly-regulated industries, FNs might have worse consequences; they might allow the exposure of sensitive customer data, which could result in punitive fines from regulators.

Therefore, when individual organizations fine-tune their IDS performance, they must consider their priorities. Often, rather than pursue the highest possible rate of accuracy, they will choose instead to stringently limit one type of false alarm, even if it could mean a slightly higher rate of the other type.

We see then that for an IDS to produce optimal results for an organization, it should be customized for each use case. The definition of “optimal” will vary from organization to organization, and indeed, even within the same organization it can vary across web applications.

Which Security Solutions Can Do This?

Historically, the ability to customize an IDS to this level has been rare. For a web security solution to offer full customization, it must expose a variety of algorithmic parameters to users, while providing in-depth feedback about the resulting performance changes, and do it all in an understandable UI that doesn’t overwhelm users with details.

Reblaze has been actively researching this area for years, and our platform includes different amounts of customization for these purposes. However, we also recognize that although many organizations want this level of detail and customization in their security solution, others do not.

Many of our customers ask a more straightforward question: “Setting aside all the details about FPs, FNs, and so on, how can we get the best possible result for us? How can this help our bottom line?”

This question seems simple, but it’s actually quite profound. It introduces a new and practical way to achieve optimal security, by using business outcomes as a security metric.

Optimizing for Business Outcomes

Conversion rate (CR) is a rich source of insights into the success of a web application. It is the ratio of conversions to visitors or requests, where a “conversion” represents the attainment of a desired goal (e.g., a purchase, registration, download, etc.)

Attackers usually do not “convert” or perform the desired actions for a web application. Therefore, when an IDS correctly blocks an attacker, the number of conversions will not be affected.

However, if an IDS generates an FP alarm, then a legitimate user is being prevented from accessing the application. If that user would otherwise have performed the desired action, then the number of total conversions is being negatively affected. Therefore, FPs will, over time, affect the CR.

This provides an opportunity to use application CR for feedback and algorithmic improvement. A web security solution that uses UEBA and ML can use CR to automatically adjust and optimize its performance.

In other words, as potential “improvements” to the IDS are made (via adjustments to its algorithms, parameters, and so on), the CR can be monitored. If the adjustments caused the CR to decrease, then it’s quite possible that the FP rate increased (as shown in the example below). Further investigation should be done immediately.

On the other hand, if IDS adjustments resulted in a higher percentage of requests being blocked, but the CR was not affected, then it would seem that the rate of false alarms has not changed, and the IDS adjustments were successful.

We see then that a business-outcome metric such as CR can be used directly to improve the performance of an IDS. In fact, it should be acknowledged that ultimately, business outcomes are the most important factor.

Security is a vital requirement on the web today, but it is not a goal for its own sake. A modern security solution should provide optimal security for the purpose of achieving optimal business outcomes.

Conclusion

Machine Learning is a powerful tool for achieving better protection for sites, applications, and APIs. It can provide automated analysis and continual improvement, which is especially important in the ever-evolving threat environment today.

The Reblaze platform includes ML/UEBA capabilities, and we are continuing to push forward into new areas of research. The next article in this series will discuss a specific security mechanism in our platform that protects against a wide variety of attack types.