How to migrate from open source Apache Kafka to Confluent without stopping the business

Plenty of organizations in the region arrived at Apache Kafka the right way: a capable team stood it up to solve a specific problem, it worked, and it gradually became critical infrastructure without anyone ever formally deciding that it should be.

The breaking point comes when the cluster that started as one team’s project is holding up processes that cannot go down, and the organization realizes the whole operation depends on the two or three people who know how it works.

That is when the conversation about moving to a managed platform starts. And that is where most plans fall apart, because they are framed as a cutover.

Why the maintenance window does not work

The natural instinct is to schedule a weekend, move everything and come back on Monday. With a database that sometimes works. With an event platform, almost never.

The reason is that Kafka is not a store, it is a stream with state distributed across many consumers. Migrating means moving not just the data but the read position of every consuming application, and doing it so that none of them reprocesses what it already handled or skips what it has not.

With five applications it can go well. With forty, the odds that all forty land exactly where they should inside the same window are low. And the failure mode is not a visible error: it is one consumer left behind that nobody notices until a discrepancy shows up in a reconciliation three days later.

Coexistence instead of cutover

The approach we take is different: both clusters run in parallel during the migration, and domains move from one to the other when each one is ready.

The piece that makes this possible is replication between clusters. Topics are replicated continuously from source to target, including consumer positions. That turns a single irreversible event into a series of small, reversible decisions.

The order that has worked for us:

Replicate first, move nobody. The target cluster receives everything and nobody consumes from it. This is where you validate performance, retention, schemas and real sizing, with production traffic and no risk.

Move the consumers next, not the producers. A consumer reading from the target instead of the source is a reversible configuration change. If something goes wrong, point it back at the source and it keeps running. Start with the least critical consumers.

Move the producers last, domain by domain. This is the step that makes the cutover irreversible for that domain, which is exactly why it comes last. By the time a producer writes to the target, its consumers have been reading from there for a while without incident.

The source is shut down when nobody is using it any more, not when the plan said it was due.

What you find along the way

A streaming migration almost always uncovers things that were hidden. Better to expect them than to be surprised by them.

Topics nobody consumes. It is normal to find that part of what is being published has not been read by anyone in months. A migration is a good moment to stop paying to move it.

Schemas that do not actually exist. Many open source clusters run without a Schema Registry, with the contract living in the heads of the team that wrote it. Moving to a platform with a Schema Registry surfaces the incompatibilities that were sitting there latent. Better they surface during the migration than during an incident.

Inherited retention settings. Seven-day retention set by default three years ago, on topics where the business would need thirty, or the other way around.

Consumers that depend on ordering without knowing it. Applications that work because the partitions happened to reach them in a certain order and nobody ever wrote it down.

What you need before you start

Before replicating the first topic, three things are worth having settled.

A real inventory of producers and consumers, not the architecture diagram but who is actually connected today. They almost always differ.

A criterion for grouping domains to decide what moves together. Moving in alphabetical order by topic guarantees problems; moving by business domain does not.

An explicit definition of what it means for a domain to be migrated, agreed up front rather than negotiated in flight. Without it, every cutover turns into an argument.

How long it takes

It depends almost entirely on the number of producing applications and how coupled the consumption is, not on data volume. A cluster with heavy traffic and few applications migrates quickly. A mid-sized cluster with dozens of applications from different teams takes considerably longer, because the real work is coordination, not engineering.

What can be said with confidence is the opposite of the initial instinct: you do not need to stop the business. If a migration plan requires a downtime window for an event platform, the plan is probably wrong.