Are you currently experiencing an attack?

Are you currently experiencing an attack?

Full Traffic Visibility: Why You Need It

When comparing web security solutions, a vital aspect is often overlooked: traffic visibility.

For effective security in the modern threat environment, complete visibility is required: a full understanding of every incoming request (not just the ones that were blocked), along with their contents and context.

When security events occur, or even during normal traffic conditions, partial visibility can hinder your ability to understand what is happening. Only complete visibility can help you to achieve and maintain an optimal security posture.

Unfortunately, web security solutions vary in the amount of traffic data they make available to users. In this article, we’ll discuss why full visibility is a vital requirement for effective security, and the important features to look for when evaluating a solution.

But first, we should ask the question: what constitutes “full visibility”?

Full Visibility: What It Includes

Visibility includes a number of different aspects:

  • Contents and metadata for each HTTP request
  • What happened to the request, and why
  • User-friendly access to current and historical data, both inclusive and exclusive
  • The ability to view data and control traffic in real time
  • The ability to do all the above not only for blocked requests, but also for requests that were passed. As we’ll see, this is an especially important differentiator when evaluating security solutions.

Let’s discuss these requirements, and some example situations that illustrate why each one is important.

Contents and metadata for each HTTP request

For a solution to provide full visibility, it needs to expose all the data that an HTTP request includes: the source IP, the URL that was requested, HTTP method, and so on. All the headers should be shown, along with the payload.

The solution should also display contextual metadata; whatever information the solution can discern that isn’t contained in the request itself. For example, the request might have originated from a VPN, or a cloud IP, or by someone using a Tor browser, and so on.

Why this is important: Maintaining a robust security posture requires a thorough understanding of incoming traffic. Modern attacks can be quite subtle, and you need the ability to examine all parameters of incoming requests. For example, merely tracking requestors by their IP addresses won’t be useful when threat actors are rotating IPs; to understand (and then block) the full scope of their activities, sometimes you’ll want to dig into the data and find patterns based on a variety of characteristics.

What happened to the request, and why

The security solution should display the result of its processing, i.e. was the request:

  • Immediately blocked by the solution
  • Subjected to extra inspection, and then blocked
  • Subjected to extra inspection, and then passed
  • Passed, but only because the solution was in “monitoring-only” mode (perhaps for initial setup, testing, or other purposes), and otherwise it would have blocked the request
  • Passed by the solution but rejected by the upstream server
  • Passed by the solution and accepted by the upstream server

Also, for every request that was blocked or subjected to extra inspection, the solution should explain why. In other words, it should display the aspect(s) of the request that were deemed to be suspicious or unacceptable.

Further, the solution should include the responses that were issued to incoming requests: not only the ones produced by the solution itself, but also by the upstream server. Many solutions do not include these in their reporting, but without them, it’s impossible to fully understand how your system will respond to various traffic conditions.

Why this is important: When you regularly review your incoming traffic and verify your system’s response, you can discover and correct anomalies. Over time, this will increase the accuracy and precision of your web security.

This has many uses, including overall site hygiene (such as a regular review of HTTP status codes being returned, especially 4xx and 5xx). Another recommended practice is to see if any recent security event was an FP (i.e., a False Positive alarm: a non-malicious activity that was misinterpreted as an attack). If an FP is found, you can then revise the policy that produced it.

Here’s an example. Most WAFs include default protection against SQL injection attacks, but sometimes users disable it and replace it with specific filtering of their own. So let’s say an admin has added a filter to your WAF to block all requests that match this regex:

(?i)(select|create|drop|<|>|alert)W

The admin intended to block all attempts to inject SELECT, CREATE, and DROP commands.

Most of the time, this filter works well. But once in a while, it can produce a False Positive. For example, let’s say one of your customers attempts to share one of your pages on Pinterest. This will produce a request which includes a URL like this:

https://www.pinterest.com/pin/create/button/?url=https://yoursite.com/your-page/

The regex shown earlier will match this URL (because it contains the word “create”), and thus the request would be incorrectly blocked as an injection attempt. This is a contrived example, but you understand the point; occasionally, security policies can have unintended consequences.

A regular review of your system’s reactions to incoming requests will allow you to notice errors such as these, and correct the conditions that produce them. Therefore, a good security solution must display full traffic data and your system’s responses, so that these reviews are possible.

User-friendly access to current and historical data, both inclusive and exclusive

It is challenging to present large volumes of data in an accessible, understandable-at-a-glance format. Therefore, many solutions don’t even try. Nevertheless, this is an essential feature.

A web security solution should allow you to see data across specified periods of time. You should be able to use regex filters, boolean operators, and so on, to quickly construct queries and get results. The solution should support both inclusive (‘show me requests with these characteristics’) and exclusive (‘exclude data with these characteristics’) queries.

Why this is important: Obviously, during the stress of an attack, an interface that is awkward or difficult to use is unacceptable.

However, even during normal conditions, robust query capabilities are important. As mentioned above, it’s important to regularly look at your incoming traffic and see what insights can be gained.

For example, you might want to begin each day with a review of what happened over the previous 24 hours. But looking at all requests would be overwhelming; this type of review is much more feasible when you can include filters such as:

  • Exclude routine requests, such as load-balancer calls to health monitoring URLs.
  • Exclude passed requests, and show only blocked requests.
  • Exclude requests rejected by the upstream server, displaying only those blocked by the security solution.
  • Exclude requests from banned/quarantined IPs.
  • Exclude requests that are obviously invalid, e.g. those with unrecognized host headers.

When a security solution offers comprehensive query capabilities, including flexible and robust filtering options, you can display only the most interesting recent events in your traffic stream. This makes it much easier to regularly verify that the solution is reacting appropriately to the traffic it receives, and to fine-tune your system’s security posture as the threat environment evolves.

The ability to view data and control traffic in real time

Some solutions only offer historical traffic data. This is inadequate; although it’s often useful to know what happened in the past, sometimes you need to know what is happening now.

Similarly, a solution must provide tools that can be used in real time. Security rulesets are necessary for long-term traffic filtering, but sometimes an admin must take immediate and direct actions that would not be triggered by existing policies.

Why this is important: Clearly, when an attack occurs, it’s crucial to know about it immediately, so corrective measures can be taken. It’s also important that a solution offers remedial actions that can be applied at once, while a longer-term resolution is prepared. For example, if requests start arriving that appear hostile, but which are somehow evading being filtered by the current security policies, an admin should be able to quickly ban the traffic source (e.g., by quarantining its IP address). This will block the requests and shut down the attack; then, with the immediate problem solved, there will be time for a thorough analysis and construction of new ruleset(s) to prevent this type of activity in the future.

The ability to do all the above for all requests

Many solutions do not provide full traffic data. They display metrics for blocked requests, but they do not show any data for requests that were passed.

These solutions restrict your ability to control your traffic, and you should not accept this.

Why this is important: If you have the ability to view passed requests as well as those which were blocked, you will find it useful in many situations.

For example, let’s say there is a traffic source with behavior that is increasingly anomalous, but not (yet) overtly hostile in a way that will trigger a security policy. You might want to scrutinize this requestor, and maybe even restrict its access to your environment. However, without the ability to view passed requests, you would have no way of even knowing that this was occurring.

Here’s another example. Let’s say you notice that one IP address has sent several requests for the same URL, and all the requests were blocked. This raises a question: are the blocks occurring because of the requestor, or because of the targeted URL?

If you can view passed requests, you can look and see if this same block is happening to multiple users. If you discover that the answer is no—that this specific IP is being blocked, but other requestors for the same URL are being allowed through—then the block is more likely to be the correct response to this IP’s request. (This is especially true when that IP has attempted other questionable activities.) Conversely, if multiple requestors are being blocked, and they seem to be innocuous otherwise, this indicates the problem might be a FP, and you should investigate the security policies that are triggering these events.

Viewing passed requests is often a valuable method for discovering FP alarms. This is important, because FPs will often go unnoticed otherwise, but they can do a lot of damage. For example, an e-commerce site might incorrectly exclude online shoppers. Or, it might reject “good” web crawlers, which would reduce its Internet visibility. Many other examples could be cited; in short, when FPs occur, it’s important to correct the conditions that produced them.

Conclusion

Along with the topics discussed above, there are many other reasons why full traffic visibility is a must-have for a web security solution.

If your current solution does not provide it, consider Reblaze. The Reblaze platform includes full, real-time traffic visibility as part of its all-in-one unified security suite. Users can easily see a broad overview of their incoming traffic, for any specified time period, and then drill down quickly into shorter periods or individual requests as needed. Reblaze clearly shows the details of incoming traffic, the decisions that were made (not only those originating from the WAF, but also from Reblaze’s DDoS protection, its advanced bot mitigation, its rate limiting module, its ATO (account takeover) prevention features, and more), and the enforcement actions that were taken.

To learn more or get a demo, feel free to contact us.

Get your price quote

Fill out your email below, and we will send you a price quote tailored to your needs

This website uses cookies to ensure you get the best experience on our website.