The amount of data that your organization produces is massive. In addition, the amount of data is increasing exponentially, which makes traditional threat detection in today’s complex IT environment difficult. The primary issue is that today’s cyber security tools are incapable of ingesting and analyzing all the data in a reasonable amount of time. Accurate data analysis takes hours, days, and even weeks to process massive amounts of data. When multiplied day by day, these tools quickly fall behind and the results of the data analysis become inaccurate, because there is constantly new, more relevant data to try and analyze.
To combat the data problem, many tools and security vendors utilize a common practice called “data reduction”. The purpose of data reduction is to:
- Break up the large datasets into more manageable pieces for analysis
Keep licensing and storage costs from escalating
Allow SIEM and log management systems to find some threats
Essentially, data reduction produces a smaller subset of the original data set, keeping only the components of the data deemed important, while discarding the rest of the data. This introduces a new problem, however, the data reduction process unwittingly removes potential threat indicators and by doing so, introduces bias into the data.
By eliminating threat indicators and the rest of the data, this bias ultimately reduces data fidelity, eliminates context, and weakens threat detection capabilities of behavioral and machine learning models.
Data Access Problem
The ability to access the entirety of the data improves query performance. However, the slow processing speed of the database query and the scale of raw data ingestion can bottleneck the entire threat platform.
Data access is a significant factor in using machine learning. Machine learning can detect hidden patterns within the log data to find underlying threats, not previously known or discoverable by other means. The machine learning algorithms use various fields within the parsed event logs to facilitate “feature selection”, used by the model to identify anomalies.
With more features to analyze, the machine learning will become more accurate in finding threats, however, with more features to analyze, the time to detection will slow if data access is poor. The data access problem is a significant constraint that currently limits detection abilities using traditional database query and SIEM technologies. Solving the data access problem can significantly improve time to detection by lessening the dwell time of a threat, thereby saving the organization from financial losses, reputational impact, and business interruption.
Data Interpretation Problem
Data interpretation is the ultimate design of any threat management system. Each organization’s data contains all of the potential answers about attackers, compromised systems, data exfiltration, reconnaissance activities, insider threats, lateral movement and other activities worth investigating.
The organization’s data is an important and critical asset to finding and facilitating threat analysis. However, it is not actually the raw data that is most valuable; the analysis and its interpretation is the key outcome desired.
Most threat detection today is based upon SIEM technologies that produce excessive “chirps” which are false positive alerts that create noise, obfuscating the real threats that get through undetected. Since the majority of the alerts are false positives, these investigations necessitate expensive labor to validate and are often too late.
By the time the investigation is completed, the attackers have slipped through the defenses, gained access and covered their tracks. To combat these issues, data interpretation in today’s world must be presented in real-time, requiring speed, accuracy and minimal labor costs.
How Security On-Demand Solves the Data Problem
We knew that in order to create a better SIEM or Threat Detection & Response platform, we would need to solve the Big Data problem and create a tool that could quickly access massive amounts of data, analyze all of it, and create accurate alerts that would detect threats.
Our team of brilliant Data Scientists in Warsaw, Poland developed a ground-breaking Big Data engine called AQ Technology that enables query speeds of 100x any other threat detection platform. AQ Technology allows us to analyze the full set of logs without data reduction, more accurately generate alerts, and to discover new and unknown threats in the data.
Layered with multiple machine learning models and behavioral analytics, we are able to find the anomalies in the data and the advanced threats, like ransomware. Additionally, implementing the latest generational detection solution, threat analytics.
If you’re interested in learning more, please reach out to our sales team for a demo.
About Security On-Demand
Security On-Demand (SOD) provides 24×7 advanced cyber-threat detection services for mid-market companies and state or local government agencies. SOD’s patented, behavioral analytics technology platform, ThreatWatch® enables the detection of advanced threats that help protect brand value and reduce the risk of a data breach. Headquartered in San Diego, California with R&D offices in Warsaw Poland, SOD services and protects hundreds of brands globally and is the winner of multiple industry awards. Please visit us at www.securityondemand.com. Find us on LinkedIn and follow us on Twitter @SecurityOnDmand.