The Quiet Redesign Around Kafka: What Databricks and Snowflake’s New Ingestion Bets Reveal

Apache Kafka is not going anywhere. That needs to be said plainly before anything else, because the headlines coming out of this year’s Snowflake Summit and Databricks Data & AI Summit could easily be read the wrong way. More than 80 percent of Fortune 100 companies run Kafka today, and by some counts, more than 250,000 organizations worldwide have it somewhere in their stack. It remains the default substrate for event-driven architecture, the system of record for enterprises that need a durable, replayable log feeding dozens of downstream consumers at once. None of that has changed.

What has changed is that the two largest data platform vendors on the planet just told the market something worth paying attention to. Databricks and Snowflake, both of which have spent years as some of the largest downstream consumers of Kafka traffic, have each built a way to ingest streaming data without requiring a Kafka cluster to sit in the middle. This move is much narrower and more commercially interesting as an attempt to eliminate the ingestion pipeline itself for a specific and very common use case where the platform is the only real destination for the data.

This is the trend this series will unpack. This first part in the series lays out what Databricks and Snowflake actually announced and why. Part two will get into the operational reality that made this trend possible and analyze the cost, maintenance, and latency friction that traditional Kafka plus stream processor setups create when the use case does not actually need that much machinery. Part three will focus on solutions. For now, the goal is simply to understand the shift on its own terms.

Databricks: Removing the Hop Between Producer and Lakehouse

Databricks’ answer to this problem is Zerobus Ingest, part of the Lakeflow Connect suite and now generally available on AWS and Azure. Instead of a producer writing to a Kafka topic, a stream processor picking up that data, transforming it, and finally landing it in Delta tables, Zerobus lets the producer push event data straight into the lakehouse via a serverless API, typically landing within seconds.

Databricks has been fairly candid about the reasoning. Kafka is a strong choice when you need a durable, low-latency way to fan data out to multiple sinks. But when the lakehouse is the only destination, keeping a message bus in the middle adds a data copy, an extra operational layer, and a chunk of specialized expertise most teams would rather not carry. Zerobus is built for exactly that narrower case.

Databricks paired this with Lakehouse RT, a serving layer built on what the company is calling its Reyden engine, designed to deliver sub-second to sub-100-millisecond query response times directly against governed Delta and Iceberg tables. Together, Zerobus and Lakehouse RT describe a full loop where data lands in the lakehouse in seconds without a broker in between, and it can be queried back out just as fast.

Snowflake: Kafka Compatible, Kafka Optional

Snowflake took a related but distinct approach with Datastream, announced at Snowflake Summit 2026 and heading toward private preview. While Databricks built a new API that happens to be Kafka-compatible, Snowflake built a fully managed streaming service that speaks the actual Kafka wire protocol. Existing Kafka producers and consumers can point at Datastream with what amounts to a configuration change, no rewrites required, and the topics land as governed Snowflake or Apache Iceberg tables, automatically inheriting role-based access control (RBAC).

Snowflake is not advising customers to rip out Kafka. It is telling customers who are already running a Kafka cluster purely to feed Snowflake that they can now get the openness of the Kafka protocol without the operational burden of running and maintaining that cluster themselves. Architecturally, Datastream is a meaningful departure from how Kafka clusters typically run. Its processors are stateless, writing directly to blob storage rather than holding data in local broker memory, with a separate highly available metadata store handling coordination. This reduces the operational surface area associated with managing partitions, replication, and broker capacity.

Data Immediacy – Pattern Underneath Both Announcements

Neither Databricks nor Snowflake is arguing that Kafka’s role as an event-driven backbone is obsolete. Both target a specific, well-understood inefficiency where organizations run a full Kafka cluster, sometimes alongside a separate stream-processing layer like Flink or Spark Structured Streaming, purely to get data into a single downstream data platform. When the lakehouse or the warehouse is the only sink, that middle layer is a cost center and a maintenance burden with no corresponding architectural benefit.

I have coined and spoken about this term called “Data Immediacy” in several talks and write-ups over the last couple of years. It is the concept of gaining instantaneous access to the most relevant, context-specific data across the enterprise to make key business decisions in real time when needed. The challenge that most IT setups face is one or more of the following – weak data ingestion mechanisms, poor choice of connectors, highly disparate data silos, high-latency stream processing engines, and way too many hops between when the data lands and when the data is processed.

The commercial angle is that every hop that data takes between the producer and the platform is a hop where a competitor could sit, whether that is a dedicated streaming vendor or a rival data platform. By absorbing ingestion into the platform itself, Databricks and Snowflake are not just removing latency and operational overhead for customers. They are also reducing the number of places where governance, billing, and architectural decisions happen outside their own perimeter. Owning more of the data path, from ingestion through storage through serving, is a strategic objective in its own right, and these announcements advance that objective regardless of how they are marketed. From a customer perspective, while this may read like a “one throat to choke” situation, it is also a vendor lock-in scenario. The openness in your architecture just vanished by adopting everything within a single platform.

The Kafka protocol has effectively become the de facto standard for moving events around, the way the S3 API became the standard interface for object storage. That is exactly why both Databricks and Snowflake chose to speak that protocol rather than invent something new. It lowers the switching cost for customers while still pulling the workload onto their own platform.

Why This Matters Beyond the Databricks and Snowflake Ecosystems

The broader data streaming market is still growing quickly, and Confluent, the commercial steward of Kafka, remains central to that growth even as it now sits inside IBM following the acquisition that closed in December 2025. Market surveys consistently find that a large majority of organizations plan to increase investment in real-time data streaming over the next year, and that data streaming adoption correlates with faster time-to-market for new data products and better SLA performance. Real-time architecture is not shrinking as a category. It is fragmenting into more specialized paths depending on what the data is actually for.

That fragmentation is the reason this trend deserves more attention than a simple headline about Kafka losing ground. Enterprises are increasingly running multiple ingestion patterns side by side rather than standardizing on a single one. A multi-consumer, event-driven workload like security event streams feeding fraud detection, personalization, and audit logging simultaneously still belongs on a full Kafka deployment or a platform like Confluent, Volt Active Data, or others built for that pattern. A single-destination workload like telemetry that only needs to land in a lakehouse for analysis now has a legitimate lighter-weight path that did not exist eighteen months ago.

For architects and platform teams, my guidance is to ask questions like “Which of our streaming workloads actually need Kafka’s multi-sink durability model, and which ones are paying Kafka’s operational tax for a benefit they never use?” That is a more relevant question today, and it is the one this series is really about.

The other thing to note is that neither of these announcements addresses the real-time decisioning layer, which is the system that needs to process these real-time streams with the relevant context to produce meaningful business decisions when they are needed. While such announcements are meant to reduce the data ingestion time and costs, they still don’t address the parts that are more meaningful to the business. They are not built to replace the class of system that has to approve a transaction, block a fraudulent trade, or reprice inventory the instant an event occurs, and that gap is precisely where the third post in this series will spend most of its time.

What Comes Next

Part two of this series will dig into that operational tax directly. Running a Kafka cluster, particularly one paired with a stream processing layer to reshape data before it lands, carries real costs in infrastructure spend, specialized staffing, and end-to-end latency, costs that are easy to justify for a genuinely complex, multi-consumer architecture and much harder to justify for a workload that only ever needed to get data from point A to point B as fast as possible. We will look at where that tax shows up most acutely and why it has become a bigger conversation now than it was even two years ago.

Part three will bring the discussion back to real-time decisioning specifically, and to where a model like Volt Active Data’s direct-to-Volt ingestion fits alongside what Databricks and Snowflake have just shipped. The short version is that this is not a two-horse race between lakehouse vendors. It is a broader rethinking of when the ingestion pipeline itself is the right architectural answer, and when it is simply overhead standing between an event and the decision it should be driving.