Even as CISOs and SOCs are under tremendous pressure to secure their businesses against the rising tide of increasingly sophisticated cyberattacks. Most enterprise Security Operations Centres (SOCs) and Chief InformationSecurity Officers (CISOs) continue to rely on Security Information and EventManagement (SIEM) solutions; however, the dynamic and changing digital environment is making SIEMs, particularly older SIEM systems, increasingly challenging for the teams managing them.

While a vast majority of enterprises (80%+) utilise SIEMs as one of the core foundational tools in their security stack, legacy SIEMs present a significant challenge to SOC efficiency. Legacy SIEMs were designed as monolithic, on-premise aggregators of security logs and can severely impede SOC operations today, which require a more flexible approach. In this piece, we shall take a critical look at the drawbacks of legacy SIEMs and propose a modular approach to future-proof SIEM and SOC operations for enterprises.

Why are legacy SIEMs a problem?

SIEMs were first developed in the early 2000s for enterprises that managed their computing infrastructure and software they owned and operated, typically within their own office space. The term “SIEM” was coined by Gartner in 2005; however, early SIEM products date back to 2000, when ArcSight was released. Legacy SIEMs focus on log collection, rule-based correlation, and manual investigation workflows.

These systems were not designed to manage cloud-based platforms where data volumes have expanded and scaled. Security logs must be collected from digital devices and endpoints that are not limited by geographic boundaries. They are restricted in their scalability, speed, and adaptability to modern cloud-native environments and threats. These limitations arise from the fundamental design of these tools, leading to various challenges. Some of these challenges are:

Cloud and Hybrid Environments

Legacy SIEMs face substantial limitations in integrating with cloud-based environments and emerging technologies, such as containers, microservices, and serverless systems, as well as with new network architectures. Most of these SIEMs lack the necessary solutions and modules, like APIs, connectors, or native support, to ingest data from modern cloud and hybrid platforms seamlessly. SOCs must invest considerable engineering resources and expenses to link legacy SIEMs to an evolving cloud-native environment and maintain those telemetry pipelines.

Connecting and Collecting Security Data and Logs

Legacy SIEMs require point-to-point integration within a vast, complex, and diverse technological ecosystem. Maintaining these integrations over time demands ongoing engineering effort. There are no pre-built integrations for connecting SIEMs to commercial off-the-shelf solutions, and integrating with and managing data ingestion from custom in-house applications and microservices is even more challenging. Additionally, they lack telemetry pipeline health tracking to detect and resolve any disruptions in the pipeline.

Data Ownership and Avoiding Lock-in

Legacy SIEMs were designed to store all your security data and continue to serve as the all-purpose repository for numerous enterprises today. Many enterprises face complex data retention regulatory requirements, which bind them to their SIEM solution until the data retention requirement period has elapsed. Legacy SIEMs lack the flexibility and analytics capabilities in data storage that modern cloud-based data lake solutions offer, creating a significant opportunity cost in utilising SIEMs for storage.

Data Formats and Standardization

Legacy SIEMs were not designed to ingest and parse logs in various formats, then route those data streams in a way and format that downstream platforms can effectively utilise. Parsing, tagging, segmenting, and transforming data into different formats and data models for routing, visibility, and analytics consumes considerable data engineering resources. SOCs inevitably determine which data elements to transform, resulting in blindspots and gaps in visibility and management.

Cost Efficiency

SOCs and CISOs face increasing challenges due to ingestion and computational costs that significantly exceed the basic infrastructure expenses for legacy SIEMs. Moreover, data ingestion licenses and storage costs exert additional pressure as data volumes grow at twice the rate compared to IT budgets year-on-year. Beyond the sticker price for legacy SIEMs, the maintenance and updates of older SIEM products involve patch management, tuning, and configuration optimization. 

CISOs and SOCs remain attached to their legacy SIEMs because these systems provide context and have been optimized over time to minimize false positive rates. They are also concerned about the uncertainty, costs, and efforts involved in migrating to a new SIEM setup. However, if CISOs and SOCs want to enjoy the right SIEM for them (whether it’s legacy or next-gen) while ensuring that it can keep up with modern security threats, a modular SIEM architecture may be the solution.

Why should SIEMs be modular?

SIEM platforms serve as comprehensive solutions, incorporating data collection, log management, security data storage, threat detection, and incident response. A modular SIEM architecture divides these functions into separate, independently scalable components, enabling each to develop or be substituted without affecting the overall system. This strategy allows organizations to utilize best-in-class tools for auxiliary functions like log management, storage, and response, while SIEMs focus on their primary strength: advanced threat detection. By separating non-core functions, SIEMs can deliver quicker, more precise threat detection, integrate more seamlessly with other security tools, and scale more effectively to address changing security and compliance demands.

Roadmap for Implementing Modular SIEM Architecture

Phase 1: “Own Your Data – Avoid Vendor Lock-Ins”

Time to deploy – 2 months

This phase separates data ingestion, data storage, and threat detection from the SIEM. This modular approach enables SOCs to select best-of-breed tools while maintaining complete control over their securityinfrastructure. Data ingestion can be managed by a Security Data Fabric orSecurity Data Pipeline Platform, which can handle schema drifts, provide visibility into data flow, connect to various systems and tools, and distributedata to multiple consumers without delays while minimizing ingress and egress costs. 

For data storage, decoupling SIEM from storage for non-security-relevant data unlocks significant cost efficiency and analytics capabilities.

1.       Event Filtering:

SIEMs only receive pertinent security data to have all the threat models and policies defined.

2.       Enterprise Data LakePartitioning

Partition tables within your Enterprise DataLake to host your Security Data Lake, requiring little or no maintenance forthreat hunting capabilities.

3.       Pre-processing Data:

Ensure data is parsed before seeding it intothe data lake, making it usable. Dumping raw data into the Security Data Lakeis ineffective; this is a key requirement of the Security Data Fabric /Security Data Pipeline Platform.

4.       Cold Storage Utilization:

Push any compliance-related data that is not immediately needed tocold storage, with the ability to replay the data if required.

With only pertinent data going into the SIEM, detection rates shouldimprove while reducing false positives. Threat hunting teams can leverage thecomplete data set in the security data lake needed to perform huntingactivities. A SOAR that integrates with your SDL and SIEM should automatemature processes where required.

Phase 2: Headless SIEMs

Future-proof architecture with lessvendor lock-in

The Next-Gen adaptive, intelligent HeadlessSIEM architecture focuses on what matters to CISOs — effective threat detection— rather than infrastructure concerns like plumbing and storage. Threatdetection content is leveraged via the power of existing data lake for threatevaluation. A robust, GenAI-enabled SOAR automates responses and streamlinesthreat management.

Phase 3: AI-powered SOC and Data Analytics

Time: POC now with production timeline in 1 year

Just as CISO organizations seemed to have secured cloud computing and hybrid environments, the emergence of Generative AI is urging security teams to leverage lessons learned from their cloud adoption process to accelerate the protection and integration of this new disruptive technology. Security teams are dedicated to safeguarding generative AI capabilities while seeking methods to harness this technology to enhance their security measures. They aim to avoid limiting their use of Gen AI capabilities to just Co-Pilots. 

In today's ever-evolving threat landscape, CISO organizations are tackling various sophisticated threats while managing extensive institutional security data. The combination of this institutional knowledge with AI capabilities transforms enterprise security data into actionable insights. Currently, AI models mainly concentrate on web data, often neglecting the vital insights found within the organization’s security data. The security data fabric, along with the information in the Security data lake, helps bridge this gap by integrating institutional security data with AI, facilitating interaction with insights rather than just presenting irrelevant raw security data. Retrieval-AugmentedGeneration (RAG) enables your security data to be included in the prompts used to query the LLM model, thereby providing valuable detection insights and strengthening your threat-hunting capabilities.

Additionally, integrating Identity, Vulnerability, and GRC data into the security data lake would enhance the efficiency and modularity of the security stack. This approach allows for the use of best-of-breed solutions, facilitating theadoption of new technologies to defend against emerging threats and comply with the fast-changing regulatory landscape. It also ensures full data control while optimizing the security budget. 

Value Delivered by Modular SIEM

Future Outlook

The proposed architecture supports an adaptive security posture for the foreseeable future. Security data fabric will become an integral part of the organization, functioning as a centralized data bus. SOAR can truly evolve into a security orchestration and automation platform by not only automating a few incident response playbooks but also streamlining vulnerability ticketing, quarterly access certifications, third-party vendor questionnaires, and GRC metrics reporting, all based on the data orchestrated by the security data fabric.

Conclusion

Today’s consolidated platform approach comes with a significant risk of vendor lock-in at the cost of innovation when the threat landscape is changing at a rapid pace. Leading security market players should embrace standards so that the data is interoperable on any SIEM platform rather than locking their customers with a proprietary data format and making it difficult to switch SIEM providers. The SIEM vendor that builds adapters to seamlessly operate on existing proprietary formatted log data from other impacted SIEM vendors due to consolidation will take the market share faster than consolidated platforms.

The current SIEM market consolidation will force CISO organizations to fast-track adoption of flexible, modular, future proof security architecture. By standardizing log data collection and ingestion, segmenting data storage based on usage leveraging standardized format and adopting innovative best-of-breed threat detection capabilities on top of standardized data, the organizations will be well positioned to take advantage of emerging technologies just as the threat actors are doing. This will help the CISO organizations to stay toe-to-toe with threat actors, if not a step ahead.

‍

DataBahn recognized as leading vendor in SACR 2025 Security Data Pipeline Platforms Market Guide

As security operations become more complex and SOCs face increasingly sophisticated threats, the data layer has emerged as the critical foundation. SOC effectiveness now depends on the quality, relevance, and timeliness of data it processes; without a robust data layer, SIEM-based analytics, detection, and response automation crumble under the deluge of irrelevant data and unreliable insights.

Recognizing the need to engage with current SIEM problems, security leaders are adopting a new breed of security data tools known as Security Data Pipeline Platforms. These platforms sit beneath the SIEM, acting as a control plane for ingesting, enriching, and routing security data in real time. In its 2025 Market Guide, SACR explores this fast-emerging category and names DataBahn among the vendors leading this shift.
‍

‍Understanding Security Data Pipelines: A New Approach

The SACR report highlights this breaking point: organizations typically collect data from 40+ security tools, generating terabytes daily. This volume overwhelms legacy systems, creating three critical problems:

First, prohibitive costs force painful tradeoffs between security visibility and budget constraints. Second, analytics performance degrades as data volumes increase. Finally, security teams waste precious time managing infrastructure rather than investigating threats.

Fundamentally, security data pipeline platforms partially or fully resolve the data volume problems with differing outcomes and performance. DataBahn decouples collection from storage and analysis, automates and simplifies data collection, transformation, and routing. This architecture reduces costs while improving visibility and analytic capabilities—the exact opposite of the traditional, SIEM-based approach.

AI-Driven Intelligence: Beyond Basic Automation

The report examines how AI is reshaping security data pipelines. While many vendors claim AI capabilities, few have integrated intelligence throughout the entire data lifecycle.

DataBahn's approach embeds intelligence at every layer of the security data pipeline. Rather than simply automating existing processes, our AI continually optimizes the entire data journey—from collection to transformation to insight generation.

This intelligence layer represents a paradigm shift from reactive to proactive security, moving beyond "what happened?" to answering "what's happening now, and what should I do about it?"

Take threat detection as an example: traditional systems require analysts to create detection rules based on known patterns. DataBahn's AI continually learns from your environment, identifying anomalies and potential threats without predefined rules.  

The DataBahn Platform: Engineered for Modern Security Demands

In an era where security data is both abundant and complex, DataBahn's platform stands out by offering intelligent, adaptable solutions that cater to the evolving needs of security teams.

Agentic AI for Security Data Engineering: Our agentic AI, Cruz, automates the heavy lifting across your data pipeline—from building connectors to orchestrating transformations.  Its self-healing capabilities detect and resolve pipeline issues in real-time, minimizing downtime and maintaining operational efficiency.

Intelligent Data Routing and Cost Optimization: The platform evaluates telemetry data in real-time, directing only high-value data to cost-intensive destinations like SIEMs or data lakes. This targeted approach reduces storage and processing costs while preserving essential security insights.

Flexible SIEM Integration and Migration: DataBahn's decoupled architecture facilitates seamless integration with various SIEM solutions. This flexibility allows organizations to migrate between SIEM platforms without disrupting existing workflows or compromising data integrity.  

Enterprise-Wide Coverage: Security, Observability, and IoT/OT: Beyond security data, DataBahn's platform supports observability, application, and IoT/OT telemetry, providing a unified solution for diverse data sources. With 400+ prebuilt connectors and a modular architecture, it meets the needs of global enterprises managing hybrid, cloud-native, and edge environments.
‍

Next-Generation Security Analytics

One of DataBahn’s standout features highlighted by SACR is our newly launched "insights layer”—Reef. Reef transforms how security professionals interact with data through conversational AI. Instead of writing complex queries or building dashboards, analysts simply ask questions in natural language: "Show me failed login attempts for privileged users in the last 24 hours" or "Show me all suspicious logins in the last 7 days"

Critically, Reef decouples insight generation from traditional ingestion models, allowing Security analysts to interact directly with their data, gain context-rich insights without cumbersome queries or manual analysis. This significantly reduces the mean time to detection (MTTD) and response (MTTR), allowing teams to prioritize genuine threats quickly.

Moving Forward with Intelligent Security Data Management

DataBahn's inclusion in the SACR 2025 Market Guide affirms our position at the forefront of security data innovation. As threat environments grow more complex, the difference between security success and failure increasingly depends on how effectively organizations manage their data.

We invite you to download the complete SACR 2025 Market Guide to understand how security data pipeline platforms are reshaping the industry landscape. For a personalized discussion about transforming your security operations, schedule a demo with our team. Click here

In today's environment, your security data should work for you, not against you.

Telemetry Data Pipelines

and how they impact decision-making for enterprises

For effective data-driven decision-making, decision-makers must access accurate and relevant data at the right time. Security, sales, manufacturing, resource, inventory, supply chain, and other business-critical data help inform critical decisions. Today’s enterprises need to aggregate relevant data from around the world and various systems into a single location for analysis and presentation to leaders in a digestible format in real time for them to make these decisions effectively.

Why telemetry data pipelines matter

Today, businesses of all sizes need to collect information from various sources to ensure smooth operations. For instance, a modern retail brand must gather sales data from multiple storefronts across different locations, its website, and third-party sellers like e-commerce and social media platforms to understand how their products performed. It also helps inform decisions such as inventory, stocking, pricing, and marketing.

For large multi-national commercial enterprises, this data and its importance get magnified. Executives have to make critical decisions with millions of dollars at stake and in an accelerated timeline. They also have more complex and sophisticated systems with different applications and digital infrastructures that generate large amounts of data. Both old and new-age companies must build elaborate systems to connect, collect, aggregate, make sense of, and derive insights from this data.

What is a telemetry data pipeline?

Telemetry data encompasses various types of information captured and collected from remote and hard-to-reach sources. The term ‘telemetry’ originates from the French word ‘télémètre’, which means a device for measuring (“mètre”) data from afar (“télé”). In the context of modern enterprise businesses, telemetry data includes application logs, events, metrics, and performance indicators which provide essential information that helps run, maintain, and optimize systems and operations.

A telemetry pipeline, as the name implies, is the infrastructure that collects and moves the data from the source to the destination. But a telemetry data pipeline doesn’t just move data; it also aggregates and processes this data to make it usable, and routes it to the necessary analytics or security destinations where it can be used by leaders to make important decisions.

Core functions of a telemetry data pipeline

Telemetry data pipelines have 3 core functions:

1. Collecting data from multiple sources;
2. Processing and preparing the data for analysis; and
3. Transferring the data to the appropriate storage destination.

DATA COLLECTION

The first phase of a data pipeline is collecting data from various sources. These sources can include products, applications, servers, datasets, devices, and sensors, and they can be spread across different networks and locations. The collection of this data from these different sources and moving them towards a central repository is the first part of the data lifecycle.

Challenges: With the growing number of sources, IT and data teams find it difficult to integrate new ones. API-based integrations can take between four to eight weeks for an enterprise data engineering team, placing significant demands on technical engineering bandwidth. Monitoring and tracking sources for anomalous behavior, identifying blocked data pipelines, and ensuring the seamless flow of telemetry data are major pain points for enterprises. With data volumes growing at ~30% Y-o-Y, being able to scale data collection to manage spikes in data flow is an important problem for engineering teams to solve, but they don’t always have the time and effort to invest in such a project.

DATA PROCESSING & PREPARATION

The second phase of a data pipeline is aggregating the data, which requires multiple data operations such as cleansing, de-duplication, parsing, and normalization. Raw data is not suitable for leaders to make decisions, and it needs to be aggregated from different sources. Data from different sources have to be turned into the same format, stitched together for correlation and enrichment, and prepared to be further refined for further insights and decision-making.

Challenges: Managing the different formats and parsing it can get complicated; and with many enterprises building or having built custom applications, parsing and normalizing that data is challenging. Changing log and data schemas can create cascading failures in your data pipeline. Then there are challenges such as identifying and masking sensitive data and quarantining it to protect PII from being leaked.

DATA ROUTING

The final stage is taking the data to its intended destination – a data lake or lakehouse, a cloud storage service, or an observability or security tool. For this, data has to be put into a specific format and has to be optimally segregated to avoid the high cost of the real-time analysis tools.

Challenges: Different types of telemetry data have different values, and segregating the data optimally to manage and reduce the cost of expensive SIEM and observability tools is high priority for most enterprise data teams. The ‘noise’ in the data also causes an increase in alerts and makes it harder for teams to find relevant data in the stream coming their way. Unfortunately, segregating and filtering the data optimally is difficult as engineers can't predict what data is useful and what data isn’t. Additionally, the increasing volume of data with the stagnant IT budget means that many teams are making sub-optimal choices of routing all data from some noisy sources into long-term storage, meaning that some insights are lost.

How can we make telemetry data pipelines better?

Organizations today generate terabytes of data daily and use telemetry data pipelines to move the data in real-time to derive actionable insights that inform important business decisions. However, there are major challenges in building and managing telemetry data pipelines, even if they are indispensable.

Agentic AI solves for all these challenges and is capable of delivering greater efficiency in managing and optimizing telemetry data pipeline health. An agentic AI can –

1. Discover, deploy, and integrate with new data sources instantly;
2. Parse and normalize raw data from structured and unstructured sources;
3. Track and monitor pipeline health; be modular and sustain loss-less data flow;
4. Identify and quarantine sensitive and PII data instantly;
5. Manage and fix for schema drift and data quality;
6. Segregate and evaluate data for storage in long-term storage, data lakes, or SIEM/observability tools
7. Automate the transformation of data into different formats for different destinations;
8. Save engineering team bandwidth which can be deployed on more strategic priorities

Curious about how agentic AI can solve your data problems? Get in touch with us to explore Cruz, our agentic AI data-engineer-in-a-box to solve your telemetry data challenges.