How to Make Threat Detection Better: Challenges and Solutions

The very first Intrusion Detection System (IDS) appeared in times when some security researchers weren’t even born yet – in 19871. In 2022, we could celebrate the 35th birthday of threat detection but do SOC teams feel like popping bottles? Perhaps, not all of them. Because spotting intruders early gets harder each day and no magical solution for tackling all issues at once ever seems to be looming on the horizon.

However, it’s not the time to get upset. Yes, challenges are big and they make us start moving and coming up with new exciting methods to enhance threat detection capabilities. Let’s look at some of the latest challenges and review the ways how to address them.

Main Challenges

Threat detection doesn’t come easy and there is quite a number of reasons why it is so. And the more advanced the network of the organization becomes, the more difficult it is to keep track of every single event and discern whether it falls within the organization’s security standards. Here are some main challenges that detection specialists have to face:

• Wrong threat prioritization. Running detections with strict baselines can be a headache when alerts keep popping up and it’s hard to struggle with high false positive rates and distinguish real threats from legitimate user behavior. Consequently, SOC engineers spend a lot of their valuable time sorting out these alerts and there’s a high probability they can miss something important in the meantime.

• Lack of accurate visibility. Simply installing a SIEM and getting loads of data from multiple sources isn’t enough for accurate and timely detections. The next essential step is data correlation which also needs to be done right; otherwise, the SOC team will lack the much-needed visibility. They need to see data artifacts connected to a certain event, as well as what happened before, what happened somewhere else that is alike, etc.

• Lack of context-enriched detection content. When it comes to unknown threats, there is never enough information about them. Hence, it’s impossible to write a simple rule that detects something, when the Blue Team doesn’t even know what they are looking for.

Moreover, certain behavioral patterns of users and systems are unique to every other organization, so even detections that worked out perfectly somewhere else need precise tailoring to make them work in a completely different environment. Businesses like MSSPs also need a lot of custom content for multiple security solutions to meet their clients’ needs but it’s challenging to keep up with all the variety of vendor-specific formats.

• Lack of time, expertise, and resources. When it comes to staying ahead of modern threats, the research and development of new detection rules have to be performed on a continuous basis for the SOC to continue effective operation. However, organizations do not always possess the right amount of funding and talent, and things get even worse when there’s a lack of time.

All in all, threat detection issues are hard but extremely interesting to tackle. There is no single recipe for success. More likely, it’s about the art of mixing up people, processes, and technologies in the right proportions.


Now, when we reviewed the main challenges of threat detection, let’s explore some of the latest solutions that might be helpful for making detection faster, easier, and more efficient.

Visibility and Context

Good detection starts with good visibility paired with insightful analytics. To maximize the quality of insight, security analysts need to employ the best practices of machine learning, data science, and behavioral analytics.

To ensure visibility, it is necessary to collect data from all the sources that are crucial when it comes to determining the probable vectors of attacks. For example, a global organization in any sector of the economy normally wants to see telemetry like:

• System logs
• User activity
• Network activity & traffic
• Processes
• File hashes & operations
• Denied connections
• Peripheral device activity
• Persistence activity

However, just seeing all these vast amounts of information is barely enough. The next step is to ensure that the data correlates right. That’s when local context comes into play. Ideally, SOC engineers want to automate the contextualization and correlation of events and write detections only for the right context.

For example, in SentinelOne multiple elements are grouped together into storylines that display the whole story of what happened on a device and what was the cause for it to happen.

Once the proper correlation is in place, it is possible to implement the latest Sigma rules from expert sources like SOC Prime’s Detection as Code platform, which delivers thousands of content pieces on a continuous basis, context-enriched with relevant behavioral security intelligence, CVE, and MITRE ATT&CK® links.

Custom Baselines

Setting baselines for the normal and outlier behavior is also crucial for making threat detection more efficient. However, this task may not be as easy as it sounds. Modern cyber-attacks keep gaining increased sophistication, that’s why an average hacker would certainly like to blend in with legitimate users, making it challenging to detect suspicious activity.

As a result, the most common baseline for threat detection concentrates on specific signs of typical behavior. From there, security analysts can investigate deeper in search of anomalous events. Naturally, for every business, the set of baselines can be different.

Behavior-Based Detections

Further investigation might imply forming hypotheses and checking them through penetration testing or threat hunting. When doing this, SOC engineers try to think like attackers, forming the right questions like “If I were to attack this network, what path would I choose?” or “Which users have privileged access to the systems that I want to invade?”

When they know more about the possibly vulnerable points, it is easier to write more accurate behavior-based detections. They also might want to implement such rules in various environments. That is when speeding up the process with a content translation engine like Uncoder.IO, which instantly converts behavior-based SIGMA rules into a variety of SIEM, EDR, and NTDR formats, comes in handy.

Detection Mindset

Because of big amounts of data that need to be reviewed for possible intrusion patterns, SOC engineers employ ML tools like User Entity and Behavior Analytics (UEBA). Such solutions may help reduce the mean time to detect (MTTD), thus increasing the speed and efficiency of threat detection in general. However, they still cannot entirely eliminate the noise, i.e. false positives and false negatives rates.

To deal with the latter, the local context and uncertain conditions need to be managed by people. The issue here is that it’s hard to scale teams, capable of delivering timely and accurate detection solutions. Even mature IT specialists might have trouble dealing with the “often unknown part”2. To boot, the rapid technology development has caught many organizations in the state when multiple solutions pile up on top of each other.

As a result, the internal infrastructure is so intricate that simple rules like “block this” or “allow that” simply don’t work properly. That’s why any alerts that pop up from detection algorithms need to be carefully triaged and confirmed by humans anyway.

It turns out that modern threat detection looks less like an exact science and more like a probability theory mixed up with intuition that nevertheless requires top technical skills. Automation and the latest software solutions are necessary but they are like the basis for investigation, and not like something that you can install once and just let it do its job.

Ultimately, the best recipe for good threat detection is playing by your own rules (local context & baselines), gathering and correlating the accurate datasets, and adding some human touch to every other detection rule.