The Kafka Tax: Where Cost, Complexity, and Latency Add Up When You Do Not Need the Full Cluster

In the first part of this series, I argued that Databricks and Snowflake are not walking away from Apache Kafka. They are targeting a narrower and more commercially interesting problem, which is the ingestion pipeline that exists purely to move data into a single downstream platform. Zerobus and Datastream both exist because a real inefficiency has been sitting in plain sight for years, and the two largest data platform vendors finally decided to go after it.

This second part is about that inefficiency itself. I want to be precise here, because the point is not that Kafka is expensive or complex in some abstract sense. Kafka has earned its place in the architectures it was designed for. The point is that a specific class of workload has been paying for capabilities it never uses, and the bill shows up in three places that are worth examining one at a time. Those three are cost, maintenance, and latency. Each one is defensible when the workload genuinely needs a durable, multi-consumer event backbone. Each one becomes hard to justify the moment the workload only ever needed to get data from point A to point B.

I call the underlying goal Data Immediacy, a term I have used across several talks and write-ups over the last couple of years. It is the idea of gaining instantaneous access to the most relevant, context-specific data across the enterprise so that key business decisions can happen in real time when they are needed. The Kafka tax is what stands between many organizations and that goal, and understanding where it accrues is the first step toward deciding whether you should be paying it at all.

The Cost Line Is Bigger Than the Cluster

When most teams estimate the cost of running Kafka, they look at just the broker infrastructure. While it may not be the largest number, the full cost of a Kafka deployment includes the brokers, the storage those brokers consume for retention, the network transfer between availability zones, the stream processing layer that so often sits alongside it, and the human expertise required to keep it all healthy. When you add these together, the infrastructure line item is often the smallest part of the total.

Consider the common pattern where Kafka feeds a single analytical destination. Data lands in a topic, a stream processor like Flink or Spark Structured Streaming reshapes it, and the result is written into a warehouse or lakehouse. In that setup, you are paying for at least two distinct compute systems, Kafka and the processor, plus the storage each one needs, plus the cost of holding a duplicate copy of the data as it moves through the pipeline. Databricks was explicit about this logic when it introduced Zerobus. When the lakehouse is the only destination, the message bus adds a data copy, an operational layer, and specialized expertise that produce no corresponding benefit. 

Staffing cost deserves particular attention because it rarely appears in a budget forecast and almost always shows up in actual spend. Kafka is not a system you deploy and forget. Running it well requires people who understand partitioning strategy, replication, consumer group behavior, and capacity planning. Practitioner reviews of Kafka consistently name this as the tradeoff, and the phrasing tends to be some version of the same complaint, which is that Kafka demands dedicated expertise for monitoring, scaling, and maintenance, and smaller teams find that overwhelming. Another point is that Kafka replication is necessary to maintain business continuity, and it requires its own separate infrastructure. Factor that cost into the Kafka tax, too.

The Maintenance Burden Compounds Over Time

Cost is just a snapshot. Maintenance is the cost that keeps accumulating, and it compounds in ways that are easy to underestimate at the start of a deployment. A Kafka cluster is a distributed system, and distributed systems demand ongoing attention, no matter how much data is flowing through them. Partitions need to be balanced. Replication has to be monitored. Schema evolution has to be managed so a change on the producer side doesn’t silently break a downstream consumer. One has to observe consumer lag as well because it is often the first sign that something upstream or downstream has gone wrong.

There is a pattern I have seen described well by engineers who spend their days inside these deployments, and it captures the maintenance problem better than any cost model. Organizations that have run Kafka clusters for five or six years are now proactively seeking help with upgrades and standardization, not because anything is broken, but because each cluster has quietly become a one-off, configured and re-tuned so many times that no two look alike and none map cleanly to a standard. Every deployment ends up configured a little differently, running a little differently, and nearly impossible to compare against best practices. That drift is the real maintenance burden. It is not a single large event. It is the slow accumulation of small, individually reasonable decisions that leave you with infrastructure only a handful of people fully understand.

This burden is entirely justified when Kafka is doing the job it was built for. If a single event stream is feeding fraud detection, personalization, audit logging, and a data warehouse all at once, the durable multi-consumer log is genuinely earning the maintenance it requires. The problem is that workload where none of that fan-out exists. When Kafka sits between one producer and one destination, you are carrying the full operational weight of a multi-consumer system to serve a single-consumer need. Maintenance doesn’t scale down just because the use case is simple. It is a fixed tax on a variable benefit.

Consider the market backdrop here too. Confluent, the commercial steward of Kafka, now sits inside IBM following the acquisition that closed in December 2025. For teams running managed Kafka, vendor consolidation of that scale introduces exactly the kind of uncertainty that makes people re-examine what they are running and why. Don’t throw the baby out with the bathwater yet but use this moment to review your architecture and deem what is strategic vs what is redundant.

Latency Is Where the Architecture Reveals Itself

The third element of the Kafka tax is latency, and it is the one that matters most for the real-time decisioning use cases I care about. Every hop in a data pipeline adds time. A producer writes to Kafka. A stream processor reads from Kafka, transforms the data, and writes it onward. The destination ingests it and makes it queryable. Each of those stages is measured in milliseconds under good conditions, and those milliseconds add up. For a batch analytics workload, that sum is irrelevant. For a decision that has to happen while an event is still unfolding, the sum is the whole game.

This is precisely why the new ingestion products are built the way they are. Zerobus advertises a direct path to the lakehouse by removing the broker hop entirely. Snowflake’s Datastream writes stateless processors directly to blob storage rather than holding data in broker memory, collapsing the operational surface area and shortening the path data has to travel. Both designs admit that the traditional multi-hop pipeline introduces latency that a meaningful set of workloads simply cannot afford. 

Here is the distinction I want to draw sharply, because it sets up the third part of this series. Reducing ingestion latency by a few seconds is a real improvement, and it is genuinely valuable for near real-time analytical and operational workloads. It is not the same thing as real-time decisioning. Landing data in a table in five seconds is a different problem from acting on an event in single-digit milliseconds. When a system has to approve a transaction, block a fraudulent trade, or reprice inventory the instant an event occurs, five seconds is not fast; it is a failure. The ingestion improvements from Databricks and Snowflake move data faster. They do not, on their own, make the decision faster.

Where the Tax Is Fair, and Where It Is Not

None of this is an argument against Kafka. I want to be as clear about that in Part 2 as I was in Part 1. Kafka remains the right foundation for genuinely event-driven architecture, and more than 80 percent of Fortune 100 companies run it for good reasons. When you need a durable, replayable log feeding many independent consumers at their own pace, the cost, maintenance, and latency of Kafka are the price of a capability you are actually using. Paying that price is not a tax in that case. It is a necessary exchange.

The tax appears when the exchange stops being fair. It appears when you run a full cluster and a stream processor to serve a workload with one producer and one destination. It appears when you are funding specialized staff to maintain a one-off deployment that supports a single analytical sink. It appears when you are absorbing three or four hops of latency for data that only ever needed to move in a straight line. In each case, you pay for durability, fan-out, and replayability the workload will never touch.

What Comes Next

I have deliberately left the hardest workload for last. The cost, maintenance, and latency friction described here is real, and the direct ingestion products from Databricks and Snowflake address a meaningful slice of it. But what is more critical is the decisioning layer. This system has to take a real-time stream, combine it with the relevant context, and produce a business decision in the window where that decision still matters. Reducing ingestion time and cost is worthwhile. It is not the same as closing the gap between an event and the action it should drive.

Part three turns to exactly that gap. I will make the case for where a direct-to-Volt model fits alongside what the lakehouse vendors have shipped, and why the real question was never whether to keep Kafka, but which of your workloads need an ingestion pipeline at all and which ones need something built to decide, not just to deliver. That is where Data Immediacy stops being a concept and becomes an architecture.