Dark Clouds: Why Enterprises are re-evaluating multi-cloud architecture

This October, the digital world learned that even clouds can fail. For more than a decade, a handful of technical giants have been the invisible gravity that holds the digital world together.

November 13, 2025

Book a Demo

Why Enterprises Are Re-Evaluating Multi-Cloud Architecture

Back to Articles

On this page

Why are Legacy SIEMs a problem?

For more than a decade, a handful of technical giants have been the invisible gravity that holds the digital world together. Together, they power over half of the world’s cloud workloads with Amazon S3 alone peaking at nearly 1 petabyte per second in bandwidth. With average uptimes measured at 99.999% and data centers spanning every continent, these clouds have made reliability feel almost ordinary.

When you order a meal, book a flight, or send a message, you don’t wonder where the data lives. You expect it to work – instantly, everywhere, all the time. That’s the brilliance, and the paradox, of hyperscale computing: the better it gets, the less we remember it’s there.

So, when two giants falter, the world didn’t just face downtime – it felt disconnected from its digital heartbeat. Snapchat went silent. Coinbase froze. Heathrow check-ins halted. Webflow blinked out.

And Meredith Whittaker, the President of Signal, reminded the internet in a now-viral post, “There are only a handful of entities on Earth capable of offering the kind of global infrastructure a service like Signal requires.”

She’s right, and that’s precisely the problem. If so much of the world runs on so few providers, what happens when the sky blinks?

In this piece, we’ll explore what the recent AWS and Azure outages teach us about dependency, and why multi-cloud resilience may be the only way forward. And how doing it right requires re-thinking how enterprises design for continuity itself.

Why even the most resilient systems go down

For global enterprises, only three cloud providers on the planet – AWS, Azure, and Google Cloud – offer true global reach with the compliance, scale, and performance demanded by millions of concurrent users and devices.

Their dominance wasn’t luck; it was engineered. Over the past decade, these hyperscalers built astonishingly resilient systems with unmatched global reach, distributing workloads across regions, synchronizing backups between data centers, and making downtime feel mythical.

As these three providers grew, they didn’t just sell compute – they sold confidence. The pitch to enterprises was simple: stay within our ecosystem, and you’ll never go down. To prove it, they built seamless multi-region replication, allowing workloads and databases to mirror across geographies in real time. A failover in Oregon could instantly shift to Virginia; a backup in Singapore could keep services running if Tokyo stumbled. Multi-region became both a technological marvel and a marketing assurance – proof that a single-cloud strategy could deliver global continuity without the complexity of managing multiple vendors.

That’s why multi-region architecture became the de facto safety net. Within a single cloud system, creating secondary zones and failover systems was a simple, cost-effective, and largely automated process. For most organizations, it was the rational resilient architecture. For a decade, it worked beautifully.

Until this October.

The AWS and Azure outages didn’t start in a data center or a regional cluster. They began in the global orchestration layers – the digital data traffic control systems that manage routing, authentication, and coordination across every region. When those systems blinked, every dependent region blinked with them.

Essentially, the same architecture that made cloud redundancy easy also created a dependency that no customer of these three service providers can escape. As Meredith Whittaker added in her post, “Cloud infrastructure is a choke point for the entire digital ecosystem.”

Her words capture the uncomfortable truth that the strength of cloud infrastructure – its globe-straddling, unifying scale – has become its vulnerability. Control-plane failures have happened before, but they were rare enough and systems recovered fast enough that single-vendor, multi-region strategies felt sufficient. The events of October changed that calculus. Even the global scaffolding of these global cloud providers can falter – and when it does, no amount of intra-cloud redundancy can substitute for independence.

If multi-region resilience can no longer guarantee uptime, the next evolution isn’t redundancy; it is reinvention. Multi-cloud resilience – not as a buzzword, but as a design discipline that treats portability, data liquidity, and provider-agnostic uptime as first-class principles of modern architecture.

Multi-cloud is the answer – and why it’s hard

For years, multi-cloud has been the white whale of IT strategy – admired from afar, rarely captured. The premise was simple: distribute workloads across providers to minimize risk, prevent downtime, and avoid over-reliance on a single vendor.

The challenge was never conviction – it was complexity. Because true multi-cloud isn’t just about having backups elsewhere – it’s about keeping two living systems in sync.

Every transaction, every log, every user action must decide: Do I replicate this now or later? To which system? In what format? When one cloud slows or fails, automation must not only redirect workloads but also determine what state of data to recover, when to switch back, and how to avoid conflicts when both sides come online again.

The system needs to determine which version of a record is authoritative, how to maintain integrity during mid-flight transactions, and how to ensure compliance when one region’s laws differ from those of another. Testing these scenarios is notoriously difficult. Simulating a global outage can disrupt production; not testing leaves blind spots.

This is why multi-cloud used to be a luxury reserved for a few technology giants with large engineering teams. For everyone else, the math – and the risk – didn’t work.

Cloud’s rise, after all, was powered by convenience. AWS, Azure, and Google Cloud offered a unified ecosystem where scale, replication, and resilience were built in. They let engineering teams move faster by outsourcing undifferentiated heavy lifting – from storage and security to global networking. Within those walls, resilience felt like a solved problem.

Due to this complexity and convenience, single-vendor multi-region architectures have become the gold standard. They were cost-effective, automated, and easy to manage. The architecture made sense – until it didn’t.

The October outages revealed the blind spot. And that is where the conversation shifts.
This isn’t about distrust in cloud vendors – their reliability remains extraordinary. It’s about responsible risk management in a world where that reliability can no longer be assumed as absolute.

Forward-looking leaders are now asking a new question:
Can emerging technologies finally make multi-cloud feasible – not as a hedge, but as a new standard for resilience?

That’s the opportunity. To transform what was once an engineering burden into a business imperative – to use automation, data fabrics, and AI-assisted operations to not just distribute workloads, but to create enterprise-grade confidence.

The Five Principles of true multi-cloud resilience

Modern enterprises don’t just run on data: they run on uninterrupted access to it.
In a world where customers expect every transaction, login, and workflow to be instantaneous, resilience has become the most accurate measure of trust.

That’s why multi-cloud matters. It’s the only architectural model that promises “always-up” systems – platforms capable of staying operational even when a primary cloud provider experiences disruption. By distributing workloads, data, and control across multiple providers, enterprises can insulate their business from global outages and deliver the reliability their customers already expect to be guaranteed. It would put enterprises back in the driver’s seat on their systems, rather than leaving them vulnerable to provider failures.

The question is no longer whether multi-cloud is desirable, but how it can be achieved without increasing complexity to the extent of making it unfeasible. Enterprises that succeed tend to follow five foundational principles – pragmatic guardrails for transforming resilience into a lasting architecture.

Start at the Edge: Independent Traffic Control
Resilience begins with control over routing. In most single-cloud designs, DNS, load balancing, and traffic steering live inside the provider’s control plane – the very layer that failed in October. A neutral, provider-independent edge – using external DNS and traffic managers – creates a first line of defense. When one cloud falters, requests can automatically shift to another entry point in seconds.

Dual-Home Identity and Access
Authentication outages often outlast infrastructure ones. Enterprises should maintain a secondary identity and secrets system – an auxiliary OIDC or SAML provider, or escrowed credentials – that can mint and validate tokens even if a cloud’s native IAM or Entra service goes dark.

Make Data Liquid
Data is the most complex system to move and the easiest to lose. True multi-cloud architecture treats data as a flowing asset, not as a static store. This means continuous replication across providers, standardized schemas, and automated reconciliation to keep operational data within defined RPO/RTO windows. Modern data fabrics and object storage replication make this feasible without doubling costs. AI-powered data pipelines can also provide schema standardization, indexing, and tagging at the point of ingesting, and prioritizing, routing, duplicating, and routing data with granular policy implementation with edge governance.

Build Cloud-agnostic Application Layers
Every dependency on proprietary PaaS services – queues, functions, monitoring agents – ties resilience to a single vendor. Abstracting the application tier with containers, service meshes, and portable frameworks ensures that workloads can be deployed or recovered anywhere, providing flexibility and scalability. Kubernetes, Kafka, and open telemetry stacks are not silver bullets, but they serve as the connective tissue of mobility.

Govern for Autonomy, not Abandonment
Multi-cloud isn’t about rejecting providers; it is about de-risking dependence. That requires unified governance – visibility, cost control, compliance, and observability – that transcends vendor dashboards. Modern automation and AI-assisted orchestration can maintain policy consistency across environments, ensuring resilience without becoming operational debt.

When these five principles converge, resilience stops being reactive and becomes a design property of the enterprise itself. It turns multi-cloud from an engineering aspiration into a business continuity strategy – one that keeps critical services available, customer trust intact, and the brand’s promise uninterrupted.

From pioneers to the possible

Not long ago, multi-cloud resilience was a privilege reserved for the few – projects measured in years, not quarters.

Coca-Cola began its multi-cloud transformation around 2017, building a governance and management system that could span AWS, Azure, and Google Cloud. It took years of integration and cost optimization for the company to achieve unified visibility across its environments.

Goldman Sachs followed, extending its cloud footprint from AWS into Google Cloud by 2019, balancing trading workloads on one platform with data analytics and machine learning on another. Their multi-cloud evolution unfolded gradually through 2023, aligning high-performance finance systems with specialized AI infrastructure.

In Japan, Mizuho Financial Group launched its multi-cloud modernization initiative in 2020, achieving strict financial-sector compliance while reducing server build time by nearly 80 percent by 2022.

Each of these enterprises demonstrated the principle: true continuity and flexibility are possible, but historically only through multi-year engineering programs, deep vendor collaboration, and substantial internal bandwidth.

That equation is evolving. Advances in AI, automation, and unified data fabrics now make the kind of resilience these pioneers sought achievable in a fraction of the time – without rebuilding every system from scratch.

Modern platforms like Databahn represent this shift, enabling enterprises to seamlessly orchestrate, move, and analyze data across clouds. They transform multi-cloud from merely an infrastructure concern into an intelligence layer – one that detects disruptions, adapts automatically, and keeps the enterprise operational even when the clouds above encounter issues.

Owning the future: building resilience on liquid data

Every outage leaves a lesson in its wake. The October disruptions made one thing unmistakably clear: even the best-engineered clouds are not immune to failure.
For enterprises that live and breathe digital uptime, resilience can no longer be delegated — it must be designed.

And at the heart of that design lies data. Not just stored or secured, but liquid – continuously available, intelligently replicated, and ready to flow wherever it’s needed.
Liquid data powers cross-cloud recovery, real-time visibility, and adaptive systems that think and react faster than disruptions.

That’s the future of enterprise architecture: always-on systems built not around a single provider, but around intelligent fabrics that keep operations alive through uncertainty.
It’s how responsible leaders will measure resilience in the next decade – not by the cloud they choose, but by the continuity they guarantee.

At Databahn, we believe that liquid data is the defining resource of the 21st century – both the foundation of AI and the reporting layer that drives the world’s most critical business decisions. We help enterprises control and own their data in the most resilient and fault-tolerant way possible.

Did the recent outages impact you? Are you looking to make your systems multi-cloud, resilient, and future-proof? Get in touch and let’s see if a multi-cloud system is worthwhile for you.

See all articles

Reducing MSSP Onboarding Time by 90% with AI-Driven Configuration

Learn how AI-driven configuration helps MSSPs reduce onboarding time by up to 90%, improving scalability, consistency, and operational efficiency across customer environments.

February 5, 2026

The managed security services market isn’t struggling with demand. Quite the opposite. As attack surfaces sprawl across cloud, SaaS, endpoints, identities, and operational systems, businesses are leaning more heavily than ever on MSSPs to deliver security outcomes they can’t realistically build in-house.

But that demand brings a different kind of pressure – customers aren’t buying coverage anymore. They’re looking to pay for confidence and reassurance: full visibility, consistent control, and the operational maturity to handle complexity, detect attacks, and find gaps to avoid unpleasant surprises. For MSSP leaders, trust has become the real product.

That trust isn’t easy to deliver. MSSPs today are running on deeply manual, repetitive workflows: onboarding new customers source by source, building pipelines and normalizing telemetry tool by tool, and expending precious engineering bandwidth on moving and managing security data that doesn’t meaningfully differentiate the service. Too much of their expertise is consumed in mechanics that are critical, but not meaningful.

The result is a barrier to scale. Not because MSSPs lack customers or talent, but because their operating model forces highly skilled teams to solve the same data problems over and over again. And that constraint shows up early. The first impression of an MSSP for a customer is overshadowed by the onboarding experience, when their services and professionalism are tested in tangible ways beyond pitches and promises. The speed and confidence with which an MSSP can move to complete, production-grade security visibility becomes the most lasting measure of their quality and effectiveness.

Industry analysis from firms such as D3 Security points to an inevitable consolidation in the MSSP market. Not every provider will scale successfully. The MSSPs that do will be those that expand efficiently, turning operational discipline into a competitive advantage. Efficiency is no longer a back-office metric; it’s a market differentiator.

That reality shows up early in the customer lifecycle most visibly, during onboarding. Long before detection accuracy or response workflows are evaluated, a more basic question is answered. How quickly can an MSSP move from a signed contract to reliable, production-grade security telemetry? Increasingly, the answer determines customer confidence, margin structure, and long-term competitiveness.

The Structural Mismatch: Multi-Customer Services and Manual Onboarding

MSSPs operate as professional services organizations, delivering security operations across many customer environments simultaneously. Each environment must remain strictly isolated, with clear boundaries around data access, routing, and policy enforcement. At the same time, MSSP teams require centralized visibility and control to operate efficiently.

In practice, many MSSPs still onboard each new customer as a largely independent effort. Much of the same data engineering and configuration work is repeated across customers, with small but critical variations. Common tasks include:

Manual configuration of data sources and collectors

Custom parsing and normalization of customer telemetry

Customer-specific routing and policy setup

Iterative tuning and validation before data is considered usable

This creates a structural mismatch. The same sources appear again and again, but the way those sources must be governed, enriched, and analyzed differs for each customer. As customer counts grow, repeated investment of engineering time becomes a significant efficiency bottleneck.

Senior engineers are often pulled into onboarding work that combines familiar pipeline mechanics with customer-specific policies and downstream requirements. Over time, this leads to longer deployment cycles, greater reliance on scarce expertise, and increasing operational drag.

This is not a failure of tools or talent. Skilled engineers and capable platforms can solve individual onboarding problems. The issue lies in the onboarding model itself. When knowledge exists primarily in ad-hoc engineering work, scripts, and tribal knowledge, it cannot be reused effectively at scale.

Why Onboarding Has Become a Bottleneck

At small scales, the inefficiency is tolerable. As MSSPs aim to scale, it becomes a growth constraint.

As MSSPs grow, onboarding must balance two competing demands:

Consistency, to ensure operational reliability across multiple customers; and

Customization, to respect each customer’s unique telemetry, data governance, and security posture.

Treating every environment identically introduces risk and compliance gaps. But customizing every pipeline manually introduces inefficiency and drag. This trade-off is what now defines the onboarding challenge for MSSPs.

Consider two customers using similar toolsets. One may require granular visibility into transactional data for fraud detection; the other may prioritize OT telemetry to monitor industrial systems. The mechanics of ingesting and moving data are similar, yet the way that data is treated — its routing, enrichment, retention, and analysis — differs significantly. Traditional onboarding models rebuild these pipelines repeatedly from scratch, multiplying engineering effort without creating reusable value.

The bottleneck is not the customization itself but the manual delivery of that customization. Scaling onboarding efficiently requires separating what must remain bespoke from what can be standardized and reused.

From Custom Setup to Systemized Onboarding

Incremental optimizations help only at the margins. Adding engineers, improving runbooks, or standardizing steps does not change the underlying dynamic. The same contextual work is still repeated for each customer.

The reason is that onboarding combines two fundamentally different kinds of work.

First, there is data movement. This includes setting up agents or collectors, establishing secure connections, and ensuring telemetry flows reliably. Across customers, much of this work is familiar and repeatable.

Second, there is data treatment. This includes policies, routing, enrichment, and detection logic. This is where differentiation and customer value are created.

When these two layers are handled together, MSSPs repeatedly rebuild similar pipelines for each customer. When handled separately, the model becomes scalable. The “data movement” layer becomes a standardized, automated process, while “customization” becomes a policy layer that can be defined, validated, and applied through governed configuration.

This approach allows MSSPs to maintain isolation and compliance while drastically reducing repetitive engineering work. It shifts human expertise upstream—toward defining intent and validating outcomes rather than executing low-level setup tasks.

In other words, systemized onboarding transforms onboarding from an engineering exercise into an operational discipline.

Applying AI to Onboarding Without Losing Control

Once onboarding is reframed in this way, AI can be applied effectively and responsibly.

AI-driven configuration observes incoming telemetry, identifies source characteristics, and recognizes familiar ingestion patterns. Based on this analysis, it generates configuration templates that define how pipelines should be set up for a given source type. These templates cover deployment, parsing, normalization, and baseline governance.

Importantly, this approach does not eliminate human oversight. Engineers review and approve configuration intent before it is executed. Automation handles execution consistently, while human expertise focuses on defining and validating how data should be treated.

Platforms such as Databahn support a modular pipeline model. Telemetry is ingested, parsed, and normalized once. Downstream treatment varies by destination and use case. The same data stream can be routed to a SIEM with security-focused enrichment and to analytics platforms with different schemas or retention policies, without standing up entirely new pipelines.

This modularity preserves customer-specific outcomes while reducing repetitive engineering work.

Reducing Onboarding Time by 90%

When onboarding is systemized and supported by AI-driven configuration, the reduction in time is structural rather than incremental.

AI-generated templates eliminate the need to start from a blank configuration for each customer. Parsing logic, routing rules, enrichment paths, and isolation policies no longer need to be recreated repeatedly. MSSPs begin onboarding with a validated baseline that reflects how similar data sources have already been deployed.

Automated configuration compresses execution time further. Once intent is approved, pipelines can be deployed through controlled actions rather than step-by-step manual processes. Validation and monitoring are integrated into the workflow, reducing handoffs and troubleshooting cycles.

In practice, this approach has resulted in onboarding time reductions of up to 90 percent for common data sources. What once required weeks of coordinated effort can be reduced to minutes or hours, without sacrificing oversight, security, or compliance.

What This Unlocks for MSSPs

Faster onboarding is only one outcome. The broader advantage lies in how AI-driven configuration reshapes MSSP operations:

Reduced time-to-value: Security telemetry flows earlier, strengthening customer confidence and accelerating value realization.

Parallel onboarding: Multiple customers can be onboarded simultaneously without overextending engineering teams.

Knowledge capture and reuse: Institutional expertise becomes encoded in templates rather than isolated in individuals.

Predictable margins: Consistent onboarding effort allows costs to scale more efficiently with revenue.

Simplified expansion: Adding new telemetry types or destinations no longer creates operational variability.

Collectively, these benefits transform onboarding from an operational bottleneck into a competitive differentiator. MSSPs can scale with control, predictability, and confidence — qualities that increasingly define success in a consolidating market.

Onboarding as the Foundation for MSSP Scale

As the MSSP market matures, efficient scale has become as critical as detection quality or response capability. Expanding telemetry, diverse customer environments, and cost pressure require providers to rethink how their operations are structured.

In Databahn’s model, multi-customer support is achieved through a beacon architecture. Each customer operates in an isolated data plane, governed through centralized visibility and control. This model enables scale only when onboarding is predictable and consistent.

Manual, bespoke onboarding introduces friction and drift. Systemized, AI-driven onboarding turns the same multi-customer model into an advantage. New customers can be brought online quickly, policies can be enforced consistently, and isolation can be preserved without slowing operations.

By encoding operational knowledge into templates, applying it through governed automation, and maintaining centralized oversight, MSSPs can scale securely without sacrificing customization. The shift is not merely about speed — it’s about transforming onboarding into a strategic enabler of growth.

Conclusion

The MSSP market is evolving toward consolidation and maturity, where efficiency defines competitiveness as much as capability. The challenge is clear: onboarding new customers must become faster, more consistent, and less dependent on manual engineering effort.

AI-driven configuration provides the structural change required to meet that challenge. By separating repeatable data movement from customer-specific customization, and by automating the configuration of the former through intelligent templates, MSSPs can achieve both speed and precision at scale.

In this model, onboarding is no longer a friction point; it becomes the operational foundation that supports growth, consistency, and resilience in an increasingly demanding security landscape.

Blog: The Modern Observability Challenge for Enterprises | Databahn

Data Observability

5 min read

The Modern Observability Challenge for Enterprises

Why today’s systems struggle and what the future requires

January 28, 2026

For most CIOs and SRE leaders, observability has grown into one of the most strategic layers of the technology stack. Cloud-native architectures depend on it, distributed systems demand it, and modern performance engineering is impossible without it. And yet, even as enterprises invest heavily in their platforms, pipelines, dashboards, and agents, the experience of achieving true observability feels harder than it should be.

Telemetry and observability systems have become harder to track and manage than ever before. Data flows, sources, and volumes shift and scale unpredictably. Different cloud containers and applications straddle different regions and systems, introducing new layers of complexity and chaos that enterprises never built these systems for.

In this environment, the traditional assumptions underpinning observability begin to break down. The tools are more capable than ever, but the architecture that feeds them has not kept pace. The result is a widening gap between what organizations expect observability to deliver and what their systems are actually capable of supporting.

Observability is no longer a tooling problem. It is a challenge to create future-forward infrastructure for observability.

The New Observability Mandate

The expectations for observability systems today are much higher than when those systems were first created. Modern organizations require observability solutions that are fast, adaptable, consistent across different environments, and increasingly enhanced by machine learning and automation. This change is not optional; it is the natural result of how software has developed.

Distributed systems produce distributed telemetry. Every service, node, pod, function, and proxy contributes its own signals: traces, logs, metrics, events, and metadata form overlapping but incomplete views of the truth. Observability platforms strive to provide teams with a unified view, but they often inherit data that is inconsistent or poorly structured. The responsibility to interpret the data shifts downstream, and the platform becomes the place where confusion builds up.

Meanwhile, telemetry volume is increasing rapidly. Most organizations collect data much faster than they can analyze it. Costs rise with data ingestion and storage, not with gaining insights. Usually, only a small part of the collected telemetry is used for investigations or analytics, even though teams feel the need to keep collecting it. What was meant to improve visibility now overwhelms the very clarity it aimed to provide.

Finally, observability must advance from basic instrumentation to something smarter. Modern systems are too complex for human operators to interpret manually. Teams need observability that helps answer not just “what happened,” but “why it happened” and “what matters right now.” That transition requires a deeper understanding of telemetry at the data level, not just more dashboards or alerts.

These pressures lead to a clear conclusion. Observability requires a new architectural foundation that considers data as the primary product, not just a byproduct.

Why Observability Architectures are Cracking

When you step back and examine how observability stacks developed, a clear pattern emerges. Most organizations did not intentionally design observability systems; they built them up over time. Different teams adopted tools for tracing, metrics, logging, and infrastructure monitoring. Gradually, these tools were linked together through pipelines, collectors, sidecars, and exporters. However, the architectural principles guiding these integrations often received less attention than the tools themselves.

This piecemeal evolution leads to fragmentation. Each tool has its own schema, enrichment model, and assumptions about what “normal” looks like. Logs tell one story, metrics tell another, and traces tell a third. Combining these views requires deep expertise and significant operational effort. In practice, the more tools an organization adds, the harder it becomes to maintain a clear picture of the system.

Silos are a natural result of this fragmentation, leading to many downstream issues. Visibility becomes inconsistent across teams, investigations slow down, and it becomes harder to identify, track, and understand correlations across different data types. Data engineers must manually translate and piece together telemetry contexts to gain deeper insights, which creates technical debt and causes friction for the modern enterprise observability team.

Cost becomes the next challenge. Telemetry volume increases predictably in cloud-native environments. Scaling generates more signals. More signals lead to increased data ingestion. Higher data ingestion results in higher costs. Without a structured approach to parsing, normalizing, and filtering data early in its lifecycle, organizations end up paying for unnecessary data processing and can't make effective use of the data they collect.

Complexity adds another layer. Traditional ingest pipelines weren't built for constantly changing schemas, high-cardinality workloads, or flexible infrastructure. Collectors struggle during burst periods. Parsers fail when fields change. Dashboards become unreliable. Teams rush to fix telemetry before they can fix the systems the telemetry is meant to monitor.

Even the architecture itself works against teams. Observability stacks were initially built for stable environments. They assume predictable data formats, slow-moving schemas, and a manageable number of sources. Modern environments break each of these assumptions.

And beneath it all lies a deeper issue: telemetry is often gathered before it is fully understood. Downstream tools receive raw, inconsistent, and noisy data, and are expected to interpret it afterward. This leads to a growing insight gap. Organizations collect more information than ever, but insights do not keep up at the same rate.

The Architectural Root Cause

Observability systems were built around tools rather than a unified data model. The architecture expanded through incremental additions instead of being designed from first principles. The growing number of tools, along with the increased complexity and scale of telemetry, created systemic challenges. Engineers now spend more time tracking, maintaining, and repairing data pipelines than developing systems to enhance observability. The unexpected surge in complexity and volume overwhelmed existing systems, which had been improved gradually. Today, Data Engineers inherit legacy systems with fragmented and complex tools and pipelines, requiring more time to manage and maintain, leaving less time to improve observability and more on fixing it. 

A modern observability system must be designed to overcome these brittle foundations. To achieve adaptive, cost-efficient observability that supports AI-driven analysis, organizations need to treat telemetry as a structured, governed, high-integrity layer. Not as a byproduct that downstream tools must interpret and repair.

The Shift Upstream: Intelligence in the Pipeline

Observability needs to begin earlier in the data lifecycle. Instead of pushing raw telemetry downstream, teams should reshape, enrich, normalize, and optimize data while it is still in motion. This single shift resolves many of the systemic issues that plague observability systems today.

AI-powered parsing and normalization ensure telemetry is consistent before reaching a tool. Automated mapping reduces the operational effort of maintaining thousands of fields across numerous sources. If schemas change, AI detects the update and adjusts accordingly. What used to cause issues becomes something AI can automatically resolve.

The analogy is straightforward: tracking, counting, analyzing, and understanding data in pipelines while it is streaming is easier than doing so when it is stationary. Volumes and patterns can be identified and monitored more effortlessly within the pipeline itself as the data enters the system, providing the data stack with a better opportunity to comprehend them and direct them to the appropriate destination. 

Data engineering automation enhances stability. Instead of manually built transformations that fail silently or decline in quality over time, the pipeline becomes flexible. It can adapt to new event types, formats, and service boundaries. The platform grows with the environment rather than being disrupted by it.

Upstream visibility adds an extra layer of resilience. Observability should reveal not only how the system behaves but also the health of the telemetry that describes it. If collectors fail, sources become noisy, fields drift, or events spike unexpectedly, teams need to know at the source. Troubleshooting starts before downstream tools are impacted.

Intelligent data tiering is only possible when data is understood early. Not every signal warrants the same storage cost or retention period. By assessing data based on relevance rather than just time, organizations can significantly reduce costs while maintaining high-signal visibility.

All of this contributes to a fundamentally different view of observability. It is no longer something that happens in dashboards. It occurs in the pipeline.

By managing telemetry as a governed, intelligent foundation, organizations achieve clearer visibility, enhanced control, and a stronger base for AI-driven operations.

How Databahn Supports this Architectural Future

In the context of these structural issues shaping the future of observability, it is essential to note that AI-powered pipelines can be the right platform for enterprises to build this next-generation foundation – today, and not as part of an aspirational future.

Databahn provides the upstream intelligence described above by offering AI-powered parsing, normalization, and enrichment that prepare telemetry before it reaches downstream systems. The platform automates data engineering workflows, adjusts to schema drift, offers detailed visibility into source telemetry, and supports intelligent data tiering based on value, not volume. The result is an AI-ready telemetry fabric that enhances the entire observability stack, regardless of the tools an organization uses.

Instead of adding yet another system to an already crowded ecosystem, Databahn helps organizations modernize the architecture layer underneath their existing tools. This results in a more cohesive, resilient, and cost-effective observability foundation.

The Path Forward: AI-Ready Telemetry Infrastructure

The future of observability won't be shaped by more dashboards or agents. Instead, it depends on whether organizations can create a stable, adaptable, and intelligent foundation beneath their tools.

That foundation starts with telemetry. It needs structure, consistency, relevance, and context. It demands automation that adapts as systems change. It also requires data that is prepared for AI reasoning.

Observability should move from being tool-focused to data-focused. Only then can teams gain the clarity, predictability, and intelligence needed in modern, distributed environments.

This architectural shift isn't a future goal; it's already happening. Teams that adopt it will have a clear edge in cost, resilience, and speed.

5 min read

The Future of the SOC is Intelligent Data Pipelines

A reflection on the recent Software Analyst Cybersecurity Research (SACR) report, and a note on the future of security telemetry

January 23, 2026

Every industry goes through moments of clarity, moments when someone steps back far enough to see not just the technologies taking shape, but the forces shaping them. The Software Analyst Cybersecurity Research (SACR) team’s latest report on Security Data Pipeline Platforms (SDPP) is one of those moments. It is rare for research to capture both the energy and the tension inside a rapidly evolving space, and to do so with enough depth that vendors, customers, and analysts all feel seen. Their work does precisely that.

Themes from the Report

Several themes stood out to us at Databahn because they reflect what we hear from customers every day. One of those themes is the growing role of AI in security operations. SACR is correct in noting that AI is no longer just an accessory. It is becoming essential to how analysts triage, how detections are created, and how enterprises assess risk. For AI to work effectively, it needs consistent, governed, high-quality data, and the pipeline is the only place where that foundation can be maintained.

Another theme is the importance of visibility and monitoring throughout the pipeline. As telemetry expands across cloud, identity, OT, applications, and infrastructure, the pipeline has become a dynamic system rather than just a simple conduit. SOC teams can no longer afford blind spots in how their data flows, what is breaking upstream, or how schema changes ripple downstream. SACR’s recognition of this shift reflects what we have observed in many large-scale deployments.

Resilience is also a key theme in the report. Modern security architecture is multi-cloud, multi-SIEM, multi-lake, and multi-tool. It is distributed, dynamic, and often unpredictable. Pipelines that cannot handle drift, bursts, outages, or upstream failures simply cannot serve the SOC. Infrastructure must be able to gracefully degrade and reliably recover. This is not just a feature; it is an expectation.

Finally, SACR recognizes something that is becoming harder for vendors to admit: the importance of vendor neutrality. Neutrality is more than just an architectural choice; it’s the foundation that enables enterprises to choose the right SIEM for their needs, the right lake for their scale, the right detection strategy for their teams, and the right AI platforms for their maturity. A control plane that isn’t neutral eventually becomes a bottleneck. SACR’s acknowledgment of this risk demonstrates both insight and courage.

The future of the SOC has room for AI, requires deep visibility, depends on resilience, and can only remain healthy if neutrality is preserved. Another trend that SACR’s report tracked was the addition of adjacent functions, bucketed as ‘SDP Plus’, which covered a variety of features – adding storage options, driving some detections in the pipeline directly, and observability, among others. The report has cited Databahn for their ‘pipeline-centric’ strategy and our neutral positioning.

As the report captures what the market is doing, it invites each of us to think more deeply about why the market is doing it and whether that direction serves the long-term interests of the SOC.

The SDP Plus Drift

Pipelines that started with clear purpose have expanded outward. They added storage. They added lightweight detection. They added analytics. They built dashboards. They released thin AI layers that sat beside, rather than inside, the data. In most cases, these were not responses to customer requests. They were responses to a deeper tension, which is that pipelines, by their nature, are quiet. A well-built pipeline disappears into the background. When a category is young, vendors fear that silence. They fear being misunderstood. And so they begin to decorate the pipeline with features to make it feel more visible, more marketable, more platform-like.

It is easy to understand why this happens. It is also easy to see why it is a problem.

A data pipeline has one essential purpose. It moves and transforms data so that every system around it becomes better. That is the backbone of its value. When a pipeline begins offering storage, it creates a new gravity center inside the enterprise. When it begins offering detection, it creates a new rule engine that the SOC must tune and maintain. When it adds analytics, it introduces a new interpretation layer that can conflict with existing sources of truth. None of these actions are neutral. Each shifts the role of the pipeline from connector to competitor.

This shift matters because it undermines the very trust that pipelines rely on. It is similar to choosing a surgeon. You choose them for their precision, their judgment, their mastery of a single craft. If they try to win you over by offering chocolates after the surgery, you might appreciate the gesture, but you will also question the focus. Not because chocolates are bad, but because that is not why you walked into the operating room.

Pipelines must not become distracted. Their value comes from the depth of their craft, not the breadth of their menu. This is why it is helpful to think about security data pipelines as infrastructure. Infrastructure succeeds when it operates with clarity. Kubernetes did not attempt to become an observability tool. Snowflake did not attempt to become a CRM. Okta did not attempt to become a SIEM. What made them foundational was their refusal to drift. They became exceptional by narrowing their scope, not widening it. Infrastructure is at its strongest when it is uncompromising in its purpose.

Security data pipelines require the same discipline. They are not just tools; they are the foundation. They are not designed to interpret data; they are meant to enhance the systems that do. They are not intended to detect threats; they are meant to ensure those threats can be identified downstream with accuracy. They do not own the data; they are responsible for safeguarding, normalizing, enriching, and delivering that data with integrity, consistency, and trust.

The Value of SDPP Neutrality

Neutrality becomes essential in this situation. A pipeline that starts to shift toward analytics, storage, or detection will eventually face a choice between what's best for the customer and what's best for its own growing platform. This isn't just a theoretical issue; it's a natural outcome of economic forces. Once you sell a storage layer, you're motivated to route more data into it. Once you sell a detection layer, you're motivated to optimize the pipeline to support your detections. Neutrality doesn't vanish with a single decision; it gradually erodes through small compromises.

At Databahn, neutrality isn't optional; it's the core of our architecture. We don’t compete with the SIEM, data lake, detection systems, or analytics platforms. Instead, our role is to support them. Our goal is to provide every downstream system with the cleanest, most consistent, most reliable, and most AI-ready data possible. Our guiding principle has always been straightforward: if we are infrastructure, then we owe our customers our best effort, not our broadest offerings.

This is why we built Cruz as an agentic AI within the pipeline, because AI that understands lineage, context, and schema drift is far more powerful than AI that sits on top of inconsistent data. This is why we built Reef as an insight layer, not as an analytics engine, because the value lies in illuminating the data, not in competing with the tools that interpret it. Every decision has stemmed from a belief that infrastructure should deepen, not widen, its expertise.

We are entering an era in cybersecurity where clarity matters more than ever. AI is accelerating the complexity of the SOC. Enterprises are capturing more telemetry than at any point in history. The risk landscape is shifting constantly. In moments like these, it is tempting to expand in every direction at once. But the future will not be built by those who try to be everything. It will be built by those who know exactly what they are, and who focus their energy on becoming exceptional at that role.

Closing thoughts

The SACR report highlights how far the category has advanced. I hope it also serves as a reminder of what still needs attention. If pipelines are the control plane of the SOC, then they must stay pure. If they are infrastructure, they must be built with discipline. If they are neutral, they must remain so. And if they are as vital to the future of AI-driven security as we believe, they must form the foundation, not just be a feature.

At Databahn, we believe the most effective pipeline stays true to its purpose. It is intelligent, reliable, neutral, and deeply focused. It does not compete with surrounding systems but elevates them. It remains committed to its craft and doubles down on it. By building with this focus, the future SOC will finally have an infrastructure layer worthy of the intelligence it supports.