Kafka Deep Intuition — Deep Tech Intuition

How to read this: Sections 1–12 take you from zero to designing and operating Kafka competently — stop there and you’re in good shape. Section 13 (“The Protocols Underneath”) is a deliberately separate advanced layer: replication/high-watermark/leader-epoch mechanics, the rebalance protocol, transaction 2PC, KRaft as a consensus system, the on-disk storage and tiered-storage internals, and share groups (KIP-932 — Kafka’s new queue semantics, GA in 4.2). Skip it first time through; return when the failure modes — or the “can Kafka be our job queue?” question — stop being abstract.

1. One-Sentence Essence

Kafka is a distributed, append-only commit log — an ordered, immutable, replayable sequence of records that producers write to the end of and consumers read forward from at their own pace, with nothing deleted when it’s read.

Everything else — topics, partitions, consumer groups, exactly-once semantics, the entire operational model — is a consequence of that one idea. If you internalize “it’s a log, not a queue,” most of Kafka’s surprising behaviors stop being surprising. A queue removes a message when you take it; a log keeps it and just remembers where each reader is. Hold onto that distinction. It is the whole game.

2. The Problem It Solved

Picture LinkedIn around 2010. They had user activity data (clicks, page views, searches), operational metrics, and application logs, and they needed to get all of it from hundreds of producing services to a growing zoo of consuming systems: a data warehouse, monitoring, search indexing, recommendation engines, fraud detection. The naive approach is point-to-point pipes — every source wired to every sink. With N producers and M consumers that’s N×M brittle integrations, each with its own format, its own failure behavior, its own backfill story. It doesn’t scale organizationally, let alone technically.

The existing tools didn’t fit either. Traditional message brokers (ActiveMQ, RabbitMQ, JMS systems) were built for a different shape of problem: relatively low-volume task queues where a message is delivered, acknowledged, and then deleted. They assumed consumers keep up and the backlog stays small. Point them at firehose-volume event data and they fell over — they kept per-message delivery state, did random-access I/O, and offered weak distributed support. Databases were the wrong tool too: you don’t want to hammer a relational database with a million writes a second of ephemeral clickstream data.

The insight that made Kafka different: don’t model this as messaging, model it as a log. Instead of a broker that tracks “has consumer X seen message Y,” make the storage a dumb, append-only file and push the “where am I” bookkeeping onto the consumer (just an integer offset). The broker’s job collapses to “append bytes to the end of a file” and “serve a byte range starting at offset N” — both of which the operating system is extraordinarily good at (we’ll see why in the Architecture section). This decouples producers from consumers completely. A producer doesn’t know or care who reads, when, or how many times. New consumers can show up years later and replay from the beginning. The same event can feed the warehouse, the search index, and the fraud system independently, each at its own speed.

Jay Kreps, Neha Narkhede, and Jun Rao built it at LinkedIn, open-sourced it through Apache, and later founded Confluent around it. The name is a literary in-joke (Kafka, the writer — “a system optimized for writing”). The design has barely changed in spirit since the 2011 paper; what’s changed is scale, reliability machinery, and the recent removal of its last external dependency (ZooKeeper — see Section 6).

3. The Concepts You Need

These are the words. You cannot reason about Kafka without them, and the rest of this document leans on every one. Read this section slowly; it’s the vocabulary that makes the Mental Model land.

Data organization

Record (message): the unit of data. A key-value pair plus a timestamp and optional headers. Both key and value are just bytes to Kafka — it neither knows nor cares about your schema. The key is special: it determines which partition the record lands in (more below).
Topic: a named category of records — “orders,” “page-views,” “payments.” It’s the highest-level abstraction you publish to and subscribe from. A topic is not a single log; it’s a logical name for a collection of partitions.
Partition: the actual append-only log. A topic is split into one or more partitions, and each partition is an independent, ordered, immutable sequence of records. This is the single most important structural concept in Kafka. A partition lives entirely on one broker’s disk (plus its replicas) — it is never split across machines. Ordering is guaranteed within a partition and nowhere else.
Offset: a monotonically increasing integer ID assigned to each record within a partition when it’s written. Offsets are per-partition (partition 0 and partition 1 both have an offset 5, referring to different records) and never change once assigned. A consumer’s entire notion of “where am I” is a set of (topic, partition, offset) positions.
Segment: partitions aren’t one giant file — they’re chopped into segment files on disk. At any moment one segment per partition is “active” (being appended to); when it hits a size or age threshold, Kafka rolls a new one. Segments are the unit of retention and deletion. This is plumbing, but it explains why retention works the way it does.

The participants

Producer: an application that appends records to a topic. It decides (via the record key and a partitioner) which partition each record goes to.
Consumer: an application that reads records from one or more partitions, tracking its own offset.
Consumer group: a set of consumers that cooperate to read a topic, identified by a shared group.id. Kafka divides the topic’s partitions among the group’s members so each partition is consumed by exactly one consumer in the group at a time. This is how you scale out consumption and how you get fan-out: different groups each get a full, independent copy of the stream. Remember this — it drives half the design decisions you’ll make.
Broker: a single Kafka server. It stores partition data on disk and serves produce/fetch requests. A handful of brokers form a cluster.
Partition leader / follower: each partition has one broker that’s the leader (all reads and writes for that partition go through it) and others holding follower replicas that copy the leader’s log. If the leader dies, a follower is promoted.

Reliability machinery

Replication factor: how many copies of each partition exist across brokers. RF=3 means three brokers each hold the partition. You can lose two brokers and not lose data — but, as we’ll see, with min.insync.replicas=2 you lose write availability after losing two. Storage durability and write availability are different things; don’t conflate them.
ISR (In-Sync Replicas): the subset of replicas currently caught up to the leader (within replica.lag.time.max.ms). Only ISR members are eligible to become leader under normal (“clean”) election. The ISR shrinking is one of the first symptoms of trouble.
LEO and high-watermark (the two offsets that matter most for correctness): the Log End Offset (LEO) is the offset of the next record a replica will write — its local tail. The high-watermark (HW) is the offset up to which all ISR members have replicated. The crucial rule: consumers can only read up to the high-watermark, never up to the leader’s LEO. Records the leader has written but that haven’t yet replicated to the full ISR are invisible to consumers — they’re not “committed” yet. This single mechanism is how Kafka guarantees a consumer never sees a record that could later vanish in a failover. The advanced section unpacks why this matters.
Leader epoch: a monotonically increasing number identifying each period of leadership for a partition, stamped into the log. It exists to fix a real data-loss/divergence bug in the old replication protocol (followers used to truncate to the high-watermark on recovery, which could silently lose committed data). Followers now truncate using leader-epoch lineage instead. You rarely touch this directly, but it’s why modern Kafka’s replication is actually safe — see the advanced section.
acks: the producer’s durability setting. acks=0 (fire and forget), acks=1 (leader wrote it locally), acks=all (all in-sync replicas have it — i.e. the write has reached the high-watermark). This is the single most consequential producer knob.
min.insync.replicas: a broker/topic setting that says “a write with acks=all only succeeds if at least this many replicas are in sync.” The standard production combo is RF=3, min.insync.replicas=2, acks=all — survive one broker loss with no data loss and continued writes; survive two with no data loss but writes blocked.
unclean.leader.election.enable: the knob that decides what happens when no in-sync replica is available to take over a partition. false (the modern default): refuse to elect a non-ISR replica — the partition goes offline, preserving consistency at the cost of availability. true: promote an out-of-sync replica — the partition stays available but silently loses whatever committed data that replica was missing. This is Kafka’s rawest CAP-style lever, and it lives in the Judgment Calls for a reason.

The control plane

Controller / KRaft quorum: the component that manages cluster metadata — which broker leads which partition, topic configs, the ISR list. Historically this lived in an external ZooKeeper ensemble; as of Kafka 4.0 it’s handled internally by a quorum of controller nodes running KRaft (Kafka Raft). KRaft is Raft-inspired, not literally Raft — it keeps Raft’s leader-election and quorum-commit guarantees but replaces Raft’s push-based log replication with Kafka’s own pull-based model (followers fetch from the leader), so it could reuse Kafka’s existing replication machinery. Metadata is itself stored as an event log (__cluster_metadata) that brokers replay to rebuild cluster state. See Section 6 and the advanced section.

Transactions and consistency

Transaction coordinator / transactional.id / producer epoch: the machinery behind exactly-once. A transactional producer is identified by a durable transactional.id; the broker-side transaction coordinator runs a two-phase commit across the partitions a transaction touches, and a producer epoch fences out “zombie” producers (an old instance that restarted but might still try to commit). State lives in an internal __transaction_state topic.
Control records / LSO (Last Stable Offset): transactions are made atomic by writing invisible control records (commit/abort markers) into the log. A read_committed consumer reads only up to the Last Stable Offset — the offset before the earliest still-open transaction — so it never sees records from a transaction that hasn’t committed (or that aborted). LSO is to transactions what the high-watermark is to replication: the visibility boundary.

Delivery semantics

At-most-once / at-least-once / exactly-once: the three delivery guarantees. At-least-once (the default and most common) means a record is never lost but may be processed more than once. Exactly-once is achievable within Kafka with specific machinery. We’ll dig into all three in Section 4 and Section 8.
Log compaction: an alternative retention mode where, instead of deleting by age, Kafka keeps the latest record for each key and garbage-collects older values. Turns a topic into a changelog / materialized table.

4. The Distilled Introduction

This is the section that replaces the ten-hour tutorial. We’ll go end to end: install, create a topic, produce, consume, scale with groups, and touch the configs that matter from day one. At each step I’ll tell you what’s actually happening, not just what to type.

Getting it running

Modern Kafka (4.x) runs in KRaft mode with no ZooKeeper. To start a single-broker cluster you generate a cluster ID, format the storage directory, and start the broker:

KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"
bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/kraft/server.properties
bin/kafka-server-start.sh config/kraft/server.properties

In practice almost nobody installs Kafka by hand for production. You run it via a managed service (Confluent Cloud, AWS MSK, Aiven, Redpanda Cloud) or on Kubernetes via the Strimzi operator. But running a local broker once, by hand, is worth doing — it demystifies the thing. For local dev, a single broker in a Docker container is plenty.

The CLI tools live in bin/ and you’ll use a handful constantly: kafka-topics.sh, kafka-console-producer.sh, kafka-console-consumer.sh, and kafka-consumer-groups.sh. Learn these four before you touch a client library — they’re how you’ll debug everything later.

Creating a topic

bin/kafka-topics.sh --create --topic orders \
  --partitions 6 --replication-factor 3 \
  --bootstrap-server localhost:9092

Two numbers you’re choosing here matter enormously. Partitions is your ceiling on consumer parallelism — a topic with 6 partitions can be consumed by at most 6 consumers in a group working in parallel; a 7th sits idle. Replication factor is your durability/availability. Both are easy to set and hard to change well: increasing partitions later breaks key-to-partition mapping (a major gotcha, Section 7), and you can’t reduce partition count at all. We’ll cover how to size these in the Judgment Calls. For now: in production, RF=3 always; partitions, slightly over-provisioned for your projected peak.

--bootstrap-server is just any broker’s address; the client uses it to discover the rest of the cluster. You don’t have to list them all.

Producing records

Conceptually a producer does three things: serialize your data to bytes, decide a partition, and send batches to the right broker (the leader of that partition). Here’s the shape in Java — the dominant client, though there are first-class clients for Python (confluent-kafka), Go, and others:

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("acks", "all");                 // durability: wait for all in-sync replicas
props.put("enable.idempotence", "true");  // dedupe producer retries (more below)

KafkaProducer<String, String> producer = new KafkaProducer<>(props);
producer.send(new ProducerRecord<>("orders", "customer-42", "{...order json...}"));
producer.flush();

The key here is "customer-42". Kafka hashes it (hash(key) % numPartitions) to pick a partition, which means every record for customer-42 lands on the same partition and is therefore strictly ordered relative to other customer-42 records. If you pass a null key, records are spread across partitions (round-robin-ish), giving you maximum throughput but no per-entity ordering. This choice — key or no key — is one of the most important design decisions you’ll make on any topic. We’ll see in the Mental Model why ordering is fundamentally tied to partitioning.

acks=all plus enable.idempotence=true is the safe default for anything that matters. Idempotence makes the producer attach a sequence number so the broker can discard duplicates caused by retries — without it, a network blip that triggers a resend can write the same record twice. Turning idempotence on automatically sets acks=all, infinite retries, and bounded in-flight requests for you.

Producers batch. They don’t send one record per network call — they accumulate records (controlled by linger.ms and batch.size) and ship them in batches, often compressed (compression.type=lz4 or zstd). This batching is most of why Kafka producers are fast; tuning linger.ms up slightly (say 5–20ms) trades a hair of latency for a big throughput and compression win.

Consuming records

A consumer subscribes to a topic and polls in a loop:

props.put("group.id", "order-processor");
props.put("enable.auto.commit", "false");   // we'll commit manually — see below
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(List.of("orders"));

while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
    for (ConsumerRecord<String, String> r : records) {
        process(r.value());                  // your business logic
    }
    consumer.commitSync();                   // commit offsets AFTER processing
}

The poll loop is the heart of a consumer, and it does far more than fetch data. poll() is also the consumer’s heartbeat and liveness signal. If you don’t call it often enough (because process() is slow), the group coordinator decides you’re dead and kicks off a rebalance. This is the source of an enormous number of production incidents (Section 7 and 10).

The group.id is what makes this a group consumer. Start a second instance of this app with the same group.id, and Kafka will split the 6 partitions between the two — each now reads 3. Start six, each reads one. Start seven, one sits idle. That’s horizontal scaling, for free, just by running more instances. Want a totally independent copy of the stream (say, a separate analytics pipeline)? Use a different group.id; that group gets all partitions from offset zero (or wherever you configure auto.offset.reset).

Offsets and committing — the crux of reliability

A consumer’s progress is a set of offsets, and when you commit them determines your delivery guarantee. Kafka stores committed offsets in an internal topic called __consumer_offsets (yes, offsets are themselves just records in a Kafka topic — very on-brand).

Commit after processing (as above) → at-least-once. If you crash between processing and committing, the work gets redone after rebalance. No data lost; possible duplicates.
Commit before processing → at-most-once. If you crash after committing but before processing, that record is gone forever. Rarely what you want.
Auto-commit (enable.auto.commit=true, the default) commits periodically in the background. Convenient and dangerous: it can commit offsets for records you’ve fetched but not yet finished processing, silently giving you at-most-once-ish behavior with data loss on crash. Most experienced teams turn it off and commit manually after processing.

Because at-least-once is the realistic default, your consumers must be idempotent — processing the same record twice must not corrupt anything. This is not optional; it is the single most important application-level discipline in Kafka. Design every consumer assuming it will occasionally see a record twice (Section 8).

Exactly-once, briefly

Kafka can give you exactly-once semantics (EOS), but only for the consume-transform-produce pattern (read from Kafka, process, write back to Kafka) and only within Kafka. It combines three things: the idempotent producer (dedupes retries via a producer ID + sequence number that the broker checks), transactions (a transactional producer atomically writes output records and commits the input offsets — sendOffsetsToTransaction — coordinated by a broker-side transaction coordinator running a two-phase commit), and read-committed consumers (which only read up to the Last Stable Offset and skip records from aborted transactions). The atomicity is implemented by writing invisible commit/abort control records into each partition; if the transaction aborts, neither the output nor the offset commit becomes visible, so it’s as if the batch never happened. The advanced section walks the protocol.

The critical caveat, which trips up almost everyone: Kafka transactions only span Kafka. They do not extend to your database. If your consumer reads from Kafka and writes to Postgres, Kafka transactions can’t make those atomic. For that you need application-level patterns — the transactional outbox, or idempotent writes keyed by something deterministic. We’ll return to this in Section 8.

Monitoring from day one

The metric you watch above all others is consumer lag: the gap between the latest produced offset and the consumer’s committed offset, per partition. Lag is your early-warning system — it’s the difference between “we’re keeping up” and “we’re falling behind reality.” You can read it from the CLI:

bin/kafka-consumer-groups.sh --describe --group order-processor \
  --bootstrap-server localhost:9092

This shows current offset, log-end offset, and lag for each partition. Memorize this command. When something is wrong, this is usually the first thing you run.

That’s the working core. With topics, keyed producers, group consumers, deliberate offset commits, and lag monitoring, you can build real systems. Everything below makes you good at it.

5. The Mental Model

Three ideas. Internalize these and you can predict Kafka’s behavior without reading docs.

Core Idea 1: It’s a log, not a queue. Reading doesn’t consume.

A queue is destructive: take a message, it’s gone. A Kafka partition is an immutable, append-only log; consuming is just advancing a pointer (the offset) over data that stays put until retention deletes it by age or size, regardless of who has or hasn’t read it.

This single idea predicts a startling number of behaviors:

Multiple consumer groups read the same data independently. Fan-out is free and requires no duplicate storage — each group just keeps its own offset. (A queue would need a separate copy per consumer.)
You can replay history. Reset a consumer group’s offset to zero (or to a timestamp) and it reprocesses everything still in retention. This is why Kafka underpins event sourcing and why “just replay it” is a real recovery strategy.
A slow consumer never blocks the producer or other consumers. It just falls behind (lag grows). The data is already on disk; it’ll catch up or it won’t, but it doesn’t back-pressure the writer.
Deleting is by policy, not by consumption. If retention is 7 days and a consumer is down for 8, it permanently misses a day’s data — the broker deleted it on schedule, not caring that someone hadn’t read it.
Order is preserved only within a partition, because a partition is the log and offsets are per-partition. There is no global order across a topic. Ever.

Core Idea 2: The partition is the atom of everything — ordering, parallelism, and storage all at once.

A partition is simultaneously the unit of ordering (records in it are strictly sequenced), the unit of parallelism (one partition → at most one consumer per group), and the unit of placement (it lives on one broker’s disk). These three are welded together, and that welding is the source of Kafka’s most important tradeoff.

What this predicts:

Ordering and parallelism are in direct tension. Want strict total order? One partition — and therefore one consumer, no scaling. Want parallelism? More partitions — and therefore order only within each partition, not across them. You cannot have global ordering and parallelism at the same time. You can only have per-key ordering with parallelism, by keying related records to the same partition.
Your partition count is your parallelism ceiling. Six partitions means at most six working consumers in a group. The seventh is idle. This makes partition count a capacity-planning decision, not an implementation detail.
The partition key is a load-balancing decision in disguise. Because hash(key) % N picks the partition, a skewed key distribution (one giant customer, a timestamp key) creates a hot partition — one overloaded broker and consumer while the rest idle. Good key choice = high cardinality, even distribution. (Section 7, Section 8.)
Changing partition count is destructive to keyed topics. hash(key) % N changes when N changes, so a key that used to map to partition 3 now maps somewhere else — per-key ordering breaks permanently, and for stateful stream apps, state mapping corrupts. This is why people over-provision partitions up front.

Core Idea 3: Durability comes from replication and acknowledgments, not from the disk write.

It’s tempting to think “Kafka persists to disk, so my data is safe once written.” Wrong model. A write goes into the OS page cache (RAM) and is flushed to physical disk later, asynchronously. Durability does not come from the bytes hitting the platter — it comes from the data being replicated to other brokers and the producer waiting for acknowledgment that enough replicas have it.

This predicts:

acks is your real durability dial, not disk flushing. acks=all with min.insync.replicas=2 means “don’t tell me it succeeded until at least two brokers have it.” That’s what survives a broker dying with un-flushed page cache.
A single broker can lose un-flushed data on power loss — and that’s fine, because the replicas have it. Kafka deliberately trades single-node fsync durability for replication-based durability, which is faster and more available.
The ISR is the live durability set. If replicas fall out of sync (slow follower, network partition), the ISR shrinks, and with min.insync.replicas=2 and only one replica left in sync, writes with acks=all start failing — Kafka chooses to reject writes rather than under-replicate. That’s a feature: it’s refusing to lie to you about durability.
Per partition, Kafka behaves like a CP system — by default. During a network partition or insufficient ISR, a partition will refuse writes (sacrificing availability) to avoid acknowledging data it can’t safely keep (preserving consistency). The escape hatch is unclean.leader.election.enable=true, which flips that partition toward AP: it promotes an out-of-sync replica so writes continue, at the cost of silently discarding committed records the new leader never received. So Kafka isn’t globally “CP” or “AP” — it’s CP per partition with a per-topic knob that buys availability by spending consistency. Knowing which side of that knob your critical topics sit on is a real design decision, not a default to inherit.
Tuning acks down trades safety for speed, visibly. acks=1 is faster but loses data if the leader dies before a follower copies the latest writes. acks=0 can’t even tell you the write happened. These aren’t “performance settings” — they’re “how much data am I willing to lose” settings.

6. The Architecture in Plain English

Let’s narrate what physically happens, end to end, when a record flows through Kafka — and where state lives, because state location is always the real insight.

The write path. Your producer batches records and sends a produce request to the broker that is leader for the target partition (the client learned the leader from cluster metadata it fetched at startup and refreshes periodically). The leader appends the batch to the end of the active segment file for that partition, advancing its Log End Offset (LEO). Here’s the performance trick: that append is a sequential disk write, and sequential disk I/O is wildly faster than random I/O — hundreds of MB/s versus hundreds of KB/s, and per a well-known ACM Queue measurement, sequential disk access can in some cases beat random memory access. The data goes into the OS page cache and is flushed to disk lazily by the kernel. Meanwhile, the follower replicas issue fetch requests to the leader (replication in Kafka is pull-based — followers pull, the leader doesn’t push) and append the new records to their own logs. Once every replica in the ISR has fetched up to a given offset, the leader advances the partition’s high-watermark (HW) to that offset. Only now — once the write is at or below the HW, meaning the full ISR has it — does the leader acknowledge an acks=all produce request, and only now does the record become visible to consumers. The gap between the leader’s LEO and the HW is exactly the set of records that are written but not yet committed.

Why the high-watermark exists (and the bug that nearly defeated it). The HW is what lets Kafka promise that a consumer never reads a record that could later disappear. If the leader dies and an ISR follower is promoted, the new leader is guaranteed to have everything up to the old HW — so committed, consumer-visible data survives. The subtle part: for years, recovering followers truncated their log down to the high-watermark before re-fetching, and because the HW propagates a round-trip behind the data, a badly-timed crash-plus-failover could silently drop a record that had actually been committed, or cause two replicas’ logs to diverge. The fix (KIP-101) was the leader epoch: every leadership period gets a monotonic ID stamped into the log, and a recovering follower asks the leader “where did my epoch actually end?” and truncates to that lineage point rather than blindly to the HW. This is the kind of detail that separates “Kafka replicates data” from understanding why Kafka’s replication is actually safe — and it’s worth knowing that the safety is comparatively recent and was model-checked into existence (KIP-279, KIP-320 closed related gaps).

The read path. A consumer sends a fetch request to the leader: “give me records from partition 3 starting at offset 1000.” The broker locates the byte range in the segment files and ships everything up to the high-watermark (never beyond it). The second performance trick lives here: zero-copy. Normally, sending a file over a socket copies the data four times (disk → kernel page cache → application memory → socket buffer → NIC) with multiple user/kernel context switches. Kafka uses the sendfile system call (transferTo in the JVM) to send bytes straight from the page cache to the network socket, never copying into the JVM heap at all. No deserialization, no garbage to collect, no CPU wasted shuffling bytes. This is why one copy of a hot topic in page cache can feed many consumer groups cheaply — the bytes never enter user space. (Note: enabling TLS/SSL breaks zero-copy, because the broker must encrypt the bytes, which requires pulling them into user space. That’s a real, often-overlooked cost of in-transit encryption.)

Where state lives. This is the key insight. The data lives in segment files on broker disks, replicated. The consumer’s position lives in the __consumer_offsets topic — itself a replicated, compacted Kafka topic. The cluster metadata (which broker leads which partition, topic configs, the ISR lists, ACLs) lives in the controller.

That last one changed fundamentally in Kafka 4.0. Historically, metadata lived in ZooKeeper — a separate distributed system you had to deploy, secure, and operate alongside Kafka. The controller broker talked to ZooKeeper; leader elections and config changes flowed through it. This was two systems stitched together, with all the operational overhead and failure-mode complexity that implies. KRaft (Kafka Raft) removed ZooKeeper entirely. Now a small quorum of controller nodes stores all cluster metadata in an internal replicated log (__cluster_metadata); brokers are observers that pull this log and replay it to rebuild cluster state in memory (an application of stream-table duality — the metadata log is the stream, each broker’s in-memory cluster state is the table). The consensus layer is KRaft, which is Raft-inspired but deliberately not vanilla Raft: it keeps Raft’s quorum-based election and commit guarantees (a metadata change is committed once a majority of controllers replicate it) but swaps Raft’s push-based replication for Kafka’s own pull-based fetch model, so it could reuse the existing log and compaction machinery (KIP-595). KRaft was production-ready from Kafka 3.3, became the default in 3.5, and is the only mode as of 4.0 (released March 2025; ZooKeeper support is fully gone). The payoff: one system instead of two, faster metadata propagation and failover (brokers cache metadata rather than querying ZooKeeper on demand), far higher partition ceilings (the old ZooKeeper-era ~200K-partition cluster cap is gone), and a much smaller operational footprint. If you’re learning Kafka today, learn KRaft and treat ZooKeeper as history you’ll only meet in old clusters.

The cluster as a whole. Brokers are mostly peers — each leads some partitions and follows others, so load and leadership spread across the cluster. The controller quorum is the brain that assigns leadership and reacts to broker failures by promoting in-sync followers. Producers and consumers are thin clients that discover topology, talk directly to partition leaders, and otherwise stay out of the way. There is no central message-routing bottleneck — that’s the architectural reason Kafka scales horizontally just by adding brokers and partitions.

7. The Things That Bite You

Each of these connects back to the mental model. They’re the bugs you’ll hit in your first year.

1. You expected global ordering. You only get per-partition ordering. New users assume a topic preserves the order things were sent. It doesn’t — only each partition does (Core Idea 2). If order-created and order-shipped for the same order land on different partitions (because you used null keys or different keys), a consumer can see “shipped” before “created.” Fix: key by the entity whose ordering you care about (the order ID), so all its events share a partition. Accept that you get per-key order, never global order.

2. Slow processing triggers rebalancing storms. Your process() is slow, so you don’t call poll() within max.poll.interval.ms (default 5 minutes). The coordinator concludes the consumer is dead, revokes its partitions, and rebalances. Now another consumer picks up the work, is also slow, gets kicked, and you spiral into a rebalancing loop where the group spends all its time reassigning partitions and almost none processing. Fix: reduce max.poll.records so each poll does less work, move slow I/O off the poll thread, raise max.poll.interval.ms if the work is genuinely long, and use the cooperative-sticky assignor (below). This is probably the #1 Kafka production incident category.

3. Eager rebalancing stops the whole world. With the old default (“eager”) assignor, every rebalance makes every consumer drop all its partitions and stop processing until reassignment completes — a “stop-the-world” pause that gets worse as the group grows. Fix: use partition.assignment.strategy=CooperativeStickyAssignor (the modern default since 2.4+). It only moves the partitions that actually need to move, and consumers keep processing the rest during the rebalance. For large groups this is the difference between a stable system and a perpetual rebalancing nightmare.

4. Adding partitions silently breaks keyed ordering and stateful apps. You hit a throughput wall and add partitions to a live keyed topic. Now hash(key) % N maps existing keys to different partitions than before — per-key ordering breaks at the boundary, and for Kafka Streams apps, the state store (partitioned by the same hash) corrupts its mapping (Core Idea 2). Fix: over-provision partitions at creation time; treat partition-count changes on keyed topics as a migration, not a config tweak.

5. A hot partition makes a “scaled” system behave like a single thread. You have 10 partitions and 10 consumers, but one tenant generates 90% of traffic and your key is tenant_id. Nine consumers idle while one drowns; adding brokers doesn’t help because the hot partition is one partition on one broker (Core Idea 2). Fix: pick a higher-cardinality key (order ID, not tenant ID), or give whale tenants their own topics, or add a salt to the key — but only if you don’t need strict per-tenant ordering. Watch per-partition byte rate and lag variance, not just averages; the average looks calm while the tail is on fire.

6. A “poison pill” record wedges a partition forever. One malformed record fails deserialization or processing every time. With at-least-once, the consumer never advances past it — it retries forever, lag on that partition climbs without bound, and everything behind it is stuck. Fix: bounded retries with backoff, then route the irrecoverable record to a dead-letter topic (DLQ) and move on. And make sure your DLQ has replay tooling, or it just becomes a data graveyard.

7. Auto-commit lost your data and you didn’t notice. With enable.auto.commit=true, offsets commit on a timer for records you’ve fetched, possibly before you finished processing them. Crash at the wrong moment and those records are marked done but never actually processed — silent data loss (connects to the offset discussion in Section 4). Fix: turn auto-commit off; commit manually after processing succeeds.

8. Caching the committed offset across a rebalance reports phantom state. A subtler one (Cloudflare hit this): you cache “last committed offset” in memory, but after a rebalance you’re assigned different partitions. Your cached value is now meaningless, your health check thinks you’re stuck, the liveness probe restarts the pod, which triggers another rebalance, which restarts another pod — a self-inflicted cascade. Fix: only track offsets for the partitions you currently own, and reset that state on partition-assignment changes.

9. The consumer that depends on a slow database amplifies failure. Your consumer calls a database per record. The DB slows down; consumers block on it, miss their poll interval, get kicked, rebalance, and the backlog explodes — a slow dependency turns into a Kafka availability incident. Fix: bound and time-box external calls, use backpressure (pause/resume partitions when saturated), and never assume the consumer can outrun a degraded downstream.

10. Retention deleted data you assumed was permanent. Kafka is not a database. Default retention is finite (often 7 days). If you treat a topic as a system of record and a consumer is down past the retention window, that data is gone (Core Idea 1). Fix: set retention deliberately per topic; for long-term retention use compaction (keep latest-per-key forever) or tiered storage, or sink to a real store of record. Don’t confuse “durable for the retention window” with “durable forever.”

8. The Judgment Calls

This is where experience shows. Each of these is a real fork, not “it depends.”

1. Partition count: how many? Too few and you cap consumer parallelism and risk hot partitions; too many and you pay in broker memory, open file handles, longer leader elections, slower failover, larger metadata, and longer rebalances. The signal: estimate peak throughput and divide by realistic per-partition throughput (rule of thumb ~1–3 MB/s/partition, or measure your own), ensure partition count ≥ your target max consumer count, add ~1.5–2× headroom, then round up to a number with many divisors (avoid primes, so partitions divide evenly among varying consumer counts). What experienced engineers do: size for one-to-two years of peak and slightly over-provision, because adding partitions later is destructive on keyed topics (Section 7). But don’t gross-over-provision into thousands of idle partitions — that’s its own operational tax.

2. To key or not to key? Keyed = per-key ordering but vulnerable to skew/hot partitions. Unkeyed = even distribution and max throughput but no ordering. The signal: do you need ordering, and at what granularity? If events for an entity must be processed in sequence (order lifecycle, account balance), key by that entity. If records are independent (logs, metrics, clicks), go keyless for clean distribution. Watch out for: keys that look balanced but aren’t — tenant_id + user_id still hot-spots if one tenant dominates the hash; timestamp keys give perfect ordering and zero parallelism (everything in a window hits one partition).

3. Delivery semantics: at-least-once vs exactly-once? EOS is real but costs latency and complexity (transactions, transactional.id management, read-committed consumers) and only covers Kafka-to-Kafka. The signal: can you make your processing idempotent cheaply? If yes — and you usually can, by keying writes on a deterministic ID — choose at-least-once + idempotent consumers. It’s simpler, faster, and battle-tested. Reserve EOS for genuine consume-transform-produce stream pipelines staying entirely within Kafka where dedup is genuinely hard. For anything touching an external database, EOS won’t save you anyway — you need the outbox pattern or idempotent writes regardless. The mature default is at-least-once with idempotent processing; reach for EOS deliberately, not reflexively.

4. acks and the durability triangle. acks=all + RF=3 + min.insync.replicas=2 is the standard “I care about this data” baseline: survive one broker loss, no data loss, still accept writes. The signal: what’s the cost of losing a record? For payments, audit logs, anything financial — acks=all, no exceptions. For high-volume metrics or clickstream where the occasional loss is invisible and throughput is king, acks=1 (or even acks=0) is a legitimate, deliberate choice. The mistake is leaving it on a default without deciding. Trap: acks=all with RF=3 but min.insync.replicas=1 quietly defeats itself — “all in-sync replicas” can mean “just the leader” when followers lag out, so a single failover loses data. The 2 is load-bearing.

4b. Unclean leader election: availability or consistency? When a partition loses every in-sync replica, you’ve reached the rawest tradeoff Kafka exposes. unclean.leader.election.enable=false (modern default) keeps the partition offline until an ISR member returns — consistent, but unavailable, and a stuck partition can cascade into producer backpressure. true promotes a stale out-of-sync replica — available, but it silently discards committed records the new leader never received, and can even cause read_committed consumers to see offsets move backward. The signal: for financial ledgers, audit trails, anything where a lost committed record is a correctness incident — keep it false and treat partition unavailability as the lesser evil (and fix your real problem, which is why you lost the whole ISR). For telemetry or recoverable derived data where being down is worse than a gap — true can be right. The point a senior reviewer checks: did you choose, per topic, or inherit the default blind?

5. Where do you commit offsets? Commit after processing (at-least-once, possible dupes) is almost always right. Commit before (at-most-once, possible loss) only for data where staleness beats duplication and loss is acceptable. The signal: is your processing idempotent? If yes, commit-after is free of downside. Experienced default: manual commit after processing, auto-commit off.

6. One big topic with many partitions, or many topics? Per-tenant or per-type topics give isolation and independent tuning but explode operational complexity (hundreds of topics, harder cross-cutting debugging). One topic with good partitioning is simpler but couples tenants. The signal: do tenants need different retention/RF/SLAs, or is one whale tenant distorting a shared topic? Big tenants → dedicated topics; the long tail → a shared multi-tenant topic. This is exactly the pattern large shops (Uber-scale) converge on.

7. Rebalance assignor: which strategy? CooperativeStickyAssignor (incremental, keeps processing during rebalance) vs the legacy eager assignors (stop-the-world). The signal: basically always cooperative-sticky on modern clients — there’s rarely a reason not to. Combine with static group membership (group.instance.id) so a consumer restart (deploy, pod reschedule) doesn’t trigger a full rebalance at all. For large or frequently-deploying groups this is transformative.

8. Retention strategy: delete, compact, or tier? Time/size deletion (the default) for transient event streams; log compaction for changelog/table-like topics where you want the latest value per key kept indefinitely (config topics, CDC, materialized state); tiered storage when you want long retention without paying for hot broker disk. The signal: is this topic a stream of events (delete by age) or a table of current state (compact)? Getting this wrong means either unbounded disk growth or losing data you needed.

9. Is Kafka even the right tool here? (The biggest call.) Kafka shines at high-throughput, multi-consumer, replayable event streams and decoupling many producers from many consumers. It is overkill and operationally heavy for: simple task queues where you just need work distribution and per-message ack/retry (RabbitMQ, SQS, or a database-backed queue are simpler); request/reply RPC (that’s not what a log is for); low-volume workloads where the operational cost dwarfs the benefit; and anything needing per-message TTLs, priority queues, or selective consumption (classic Kafka has none of these — it’s a log, you read it in order). The signal: if you find yourself fighting Kafka to make it behave like a queue (deleting individual messages, prioritizing, selective ack), you’re probably reaching for the wrong tool. The important caveat as of Kafka 4.2: share groups (KIP-932) now add genuine queue semantics — concurrent consumers per partition, per-message acknowledge/release/reject, consumers beyond partition count — so the “use a separate queue” reflex is no longer automatic. If you already run Kafka and need a job queue where ordering doesn’t matter, a share group may beat standing up a second system. A dedicated queue is still simpler if queuing is your only need. See §13.7 for the full picture and the tradeoffs (you give up ordering, and exactly-once isn’t there yet).

10. Self-managed vs managed? Running Kafka well demands real, scarce operational expertise — capacity planning, partition rebalancing, JVM/GC tuning, disk and page-cache management, upgrades, monitoring. The signal: unless you have (or want to build) a dedicated platform team with Kafka depth, use a managed offering (MSK, Confluent Cloud, Aiven, Redpanda). The “free” open-source software has a very non-free operational bill (Section 11). Self-manage only when scale, cost, or control genuinely justify owning that burden.

11. Replica placement: are your three replicas actually independent? RF=3 only buys fault tolerance if the three replicas can’t fail together. Without rack awareness (broker.rack + a rack-aware replica assignment), the controller may place all three replicas of a partition in the same availability zone or on the same rack/power domain — so one AZ outage takes the whole partition down despite RF=3, and you discover the gap during the incident. The signal: always set broker.rack to your AZ/failure domain in any multi-AZ deployment; verify with kafka-topics.sh --describe that replicas of each partition span domains. The catch: spreading replicas across AZs means follower-fetch traffic crosses AZ boundaries, which on cloud providers is a real (and often surprising) cross-AZ data-transfer bill — a cost the newer follower-fetch-from-closest-replica and diskless/S3-backed designs exist specifically to attack. Durability, latency, and cloud egress cost are a three-way tension here, not a free lunch.

9. The Commands/APIs That Actually Matter

The 20% you’ll use 80% of the time, grouped by task, with the why.

Topics

# Create — partitions and RF are the decisions that matter
kafka-topics.sh --create --topic orders --partitions 6 --replication-factor 3 \
  --bootstrap-server BROKER

# Describe — your first stop for "how is this topic laid out, who leads what, what's the ISR?"
kafka-topics.sh --describe --topic orders --bootstrap-server BROKER

# Add partitions (CAREFUL on keyed topics — breaks hash mapping)
kafka-topics.sh --alter --topic orders --partitions 12 --bootstrap-server BROKER

# Change config (retention, compaction) without recreating
kafka-configs.sh --alter --topic orders \
  --add-config retention.ms=604800000 --bootstrap-server BROKER

--describe is the one you’ll run constantly: it shows leaders, replicas, and ISR per partition. An ISR shorter than the replica list is a red flag — replicas are lagging or down.

Consumer groups (your primary debugging tool)

# THE command. Shows current offset, log-end offset, and LAG per partition.
kafka-consumer-groups.sh --describe --group order-processor --bootstrap-server BROKER

# Reset offsets — replay from start, jump to end, or seek to a timestamp.
# --dry-run first, ALWAYS. Then --execute.
kafka-consumer-groups.sh --reset-offsets --group order-processor --topic orders \
  --to-earliest --dry-run --bootstrap-server BROKER

Lag per partition is your health signal. Lag concentrated on a few partitions points at a hot key or a stuck consumer; lag growing everywhere points at under-provisioned consumers or a slow downstream. Offset reset is how you replay (recovery, reprocessing after a bug fix) or skip (past a poison pill).

Producing and consuming from the CLI (for testing/debugging)

# Produce with keys so you can see partitioning behavior
kafka-console-producer.sh --topic orders --property "parse.key=true" \
  --property "key.separator=:" --bootstrap-server BROKER

# Consume from the beginning, showing keys and partitions
kafka-console-consumer.sh --topic orders --from-beginning \
  --property print.key=true --property print.partition=true --bootstrap-server BROKER

These are how you sanity-check “is data actually flowing and landing where I expect” before blaming your application code.

Producer config that matters

acks=all, enable.idempotence=true (durability + dedup), linger.ms and batch.size (throughput via batching), compression.type=lz4/zstd (cheaper network and disk). For EOS, transactional.id.

Consumer config that matters

group.id (the group), enable.auto.commit=false (commit deliberately), max.poll.records and max.poll.interval.ms (the rebalance-storm controls), partition.assignment.strategy=CooperativeStickyAssignor, group.instance.id (static membership), auto.offset.reset (earliest vs latest on first run), isolation.level=read_committed (for EOS).

10. How It Breaks

For each failure mode: symptom, root cause (tied to the model), diagnosis, fix.

Consumer lag climbing. Symptom: the gap between produced and consumed offsets grows; downstream data goes stale (the “silent killer” — dashboards green, system falling behind reality). Root cause: consumers can’t keep up — too few of them, slow processing, a slow downstream dependency, or lag concentrated on a hot partition (Core Idea 2). Diagnose: kafka-consumer-groups.sh --describe; is lag uniform (under-provisioned) or concentrated (hot key/stuck partition)? Fix: add consumers (up to partition count), speed up processing, fix the hot key, or add partitions if you’ve hit the parallelism ceiling.

Rebalancing loop. Symptom: CommitFailedError (“group has already rebalanced”), consumers repeatedly joining/leaving, the group stuck in “rebalancing,” throughput collapses. Root cause: consumers missing max.poll.interval.ms (slow processing) or flapping liveness, made worse by eager assignment (Section 7). Diagnose: check group state in --describe; look for consumers cycling; correlate with processing time and pod restarts. Fix: cooperative-sticky assignor, static membership, lower max.poll.records, raise the interval, move slow work off the poll thread.

Under-replicated partitions / ISR shrinking. Symptom: UnderReplicatedPartitions > 0, ISR smaller than replica count, possibly acks=all writes failing. Root cause: a follower can’t keep up — broker overload, disk pressure, network issues, or GC pauses (Core Idea 3: durability depends on the ISR). Diagnose: kafka-topics.sh --describe for ISR vs replicas; check broker disk, CPU, network, and JVM GC logs. Fix: relieve the struggling broker (rebalance leadership/partitions, add capacity), fix GC/disk; if ISR drops below min.insync.replicas, writes correctly fail rather than risk loss.

Producer throughput collapse / timeouts. Symptom: slow sends, TimeoutException, buffer-full errors. Root cause: leader unavailable, broker overload, or under-tuned batching. Diagnose: check which partition/broker, look at broker load and ISR, review linger.ms/batch.size/compression. Fix: tune batching, spread load (partition/key strategy), scale brokers, verify acks isn’t stricter than needed for the data.

Poison pill / stuck partition. Symptom: lag on one partition grows without bound while others are fine; the same record reappears in logs. Root cause: a record that fails processing every time; at-least-once never advances the offset (Section 7). Diagnose: inspect the record at the stuck offset (console-consumer from that offset). Fix: bounded retry → dead-letter topic → advance; fix the bug; replay the DLQ later.

Data “loss.” Symptom: expected records missing. Root cause: usually not Kafka losing data — it’s acks too low and a leader failover, or auto-commit losing in-flight records on crash, or retention deleting data before a down consumer read it (Core Ideas 1 and 3). Diagnose: check acks/min.insync.replicas, auto-commit setting, and retention vs consumer downtime. Fix: acks=all+RF=3+min.insync=2, manual commit after processing, deliberate retention.

The general debugging workflow. When something’s wrong and you’re not sure what: (1) kafka-consumer-groups.sh --describe — is there lag, and is it uniform or concentrated? (2) kafka-topics.sh --describe — are ISRs healthy, leaders balanced? (3) check broker health — disk, CPU, network, JVM GC pauses. (4) check consumer logs for rebalances, commit failures, deserialization errors. (5) kafka-console-consumer.sh --from-beginning on a test group to confirm data is actually present and well-formed. (6) check producer-side errors and acks. Lag location and ISR health answer most questions before you go deeper.

11. The Downsides / Disadvantages

The honest accounting. These don’t go away with experience or a newer version — they’re the price of admission. A reader who sees Kafka criticized fairly should trust everything else here more.

1. The operational burden is large and permanent. Running Kafka well means ongoing capacity planning, partition and leadership rebalancing, JVM and GC tuning, disk and page-cache management, careful upgrades, and deep monitoring. Where it comes from: Kafka’s performance comes from squeezing the OS, the JVM, and the disk layout — and that machinery is exposed to you to tune. What it costs: effectively a dedicated platform skill set, often a whole team at scale. Dealbreaker when: you’re a small team without that expertise and a managed offering isn’t on the table. What people think mitigates it but doesn’t: “we’ll just run the Docker image” — that gets you a demo, not a production cluster. KRaft genuinely reduced the burden (no more ZooKeeper to operate) but did not eliminate it.

2. “Free” open-source Kafka has a steep, invisible operational bill. Where it comes from: the same as #1 — the adoption decision sees a $0 license and misses the human cost. What it costs: engineer-years of operational attention, or a meaningful managed-service spend (which is often the cheaper option once you price the team honestly). Dealbreaker when: leadership budgeted the license cost and not the people cost.

3. The partition is a rigid unit, and that rigidity bites at scale. Partition count caps parallelism, can’t be reduced, and can’t be increased on keyed topics without breaking ordering and state (Core Idea 2). Where it comes from: welding ordering, parallelism, and placement into one concept — elegant, but inflexible. What it costs: you must forecast capacity up front and over-provision; getting it wrong means a painful migration. Hot partitions from skewed keys can’t be load-balanced away — you design for the hottest partition and waste the rest (Section 7). Dealbreaker when: your workload is inherently skewed and per-key ordering is also required — those two demands fight, and Kafka can’t reconcile them for you.

4. It is a log, not a queue — and that historically excluded whole categories of feature by construction. No per-message TTL, no priority queues, no selective/out-of-order consumption, and — until recently — no per-message acknowledgment or redelivery. You read a partition in order, full stop. Where it comes from: the core essence (Section 1). What it costs: if your problem needs queue semantics, you’ve traditionally had to bolt on awkward workarounds (DLQs, retry topics, client-side filtering) or pick a different tool. The 4.2 update: share groups (KIP-932) now add real queue semantics — concurrent consumers per partition and per-message ack/release/reject (§13.7) — so this downside is genuinely shrinking. But it’s not gone: share groups give up ordering and (for now) exactly-once, priority queues and per-message TTLs still don’t exist, and the feature is new. Dealbreaker when: you need priority/TTL queue semantics, or you need a simple standalone queue and aren’t already running Kafka — RabbitMQ, SQS, or a database queue will still save you operational grief.

5. Exactly-once is narrower than the marketing implies. EOS only covers Kafka-to-Kafka. The moment your pipeline touches an external database, Kafka transactions can’t make the Kafka write and the DB write atomic. Where it comes from: transactions are implemented inside Kafka’s own log/offset machinery; they have no reach into your database. What it costs: you still must build idempotency or outbox patterns at the application level, so the EOS feature often doesn’t buy you what you hoped. What people think mitigates it but doesn’t: “we turned on exactly-once” — for a DB-writing consumer, that phrase is close to meaningless without app-level idempotency.

6. It is not a database, no matter how tempting. Finite retention by default, no rich queries, no random access by anything but offset, no secondary indexes. Treating a topic as your system of record without compaction/tiering/external sinks leads to silently deleted data (Section 7, Core Idea 1). Where it comes from: the log is optimized for sequential append and sequential read, not for being queried like a store. What it costs: an extra system of record and the pipelines to keep it in sync. Dealbreaker when: you wanted a queryable durable store — that’s a database, not Kafka.

7. Cognitive load is high and durable. The mental model (partitions, offsets, consumer groups, rebalancing, ISR, acks, retention, compaction, transactions) is a lot to hold, and every developer who touches a consumer has to understand at-least-once and idempotency or they’ll write subtly broken code. Where it comes from: Kafka pushes correctness decisions (when to commit, how to dedup, how to key) onto you by design — that’s the flip side of its flexibility. What it costs: a real ongoing chunk of your team’s collective attention, and a steeper-than-usual onboarding for every new engineer.

8. Failure modes can be quiet rather than loud. The worst incidents (consumer lag, auto-commit data loss, hot partitions) don’t crash anything — they degrade silently while dashboards stay green, and you find out when the data is hours stale or already gone (Section 10). Where it comes from: the decoupling that’s Kafka’s great strength (slow consumers don’t block anyone) also means falling behind is invisible unless you’re explicitly watching lag. What it costs: you must invest in lag and ISR monitoring up front; without it, you’re flying blind toward a quiet failure.

12. The Taste Test

What separates someone who’s read the docs from someone who’s run Kafka in anger. Glance at these and you can rank the author.

Topic and partition design.

Good: partition count chosen from a throughput estimate with headroom, rounded to a divisor-friendly number; keys chosen for high cardinality and even distribution; RF=3 with min.insync.replicas=2. Bad: the default partition count (often 1) left in place; tenant_id as a key on a tenant-skewed workload; RF=1 in production; partitions cranked to thousands “to be safe.”

Producer config.

Good: acks=all and enable.idempotence=true on data that matters, with a conscious downgrade to acks=1 only where loss is explicitly acceptable; batching and compression tuned. Bad: defaults everywhere with no decision recorded; acks=0 on payments; idempotence off while expecting no duplicates.

Consumer code.

Good: auto-commit off, manual commit after processing; idempotent processing keyed on a deterministic ID; bounded retries with a dead-letter topic; cooperative-sticky assignor and static membership; slow I/O off the poll thread; only tracks offsets for currently-owned partitions. Bad: auto-commit on, processing assumed exactly-once; no DLQ so one bad record wedges a partition; heavy synchronous DB calls inside the poll loop; offset state cached across rebalances.

Operational posture.

Good: consumer lag and under-replicated-partition alerts exist before launch; retention set deliberately per topic; partition-count changes treated as migrations. Bad: monitoring added after the first incident; one global retention; “just add partitions” applied to a live keyed topic.

The tell. Ask someone “what happens if your consumer processes a record twice?” If they say “it can’t, we have exactly-once,” they haven’t run Kafka. If they say “it’s fine, our processing is idempotent because we key the write on the event ID” — they have.

13. The Protocols Underneath (Advanced)

Everything above is enough to use Kafka well and design with it. This section is for the reader who wants the distributed-systems mechanics — the parts a senior engineer reaches for when reasoning about correctness under failure. Skip it on a first read; come back when “the ISR shrank and we lost data, how?” stops being abstract.

13.1 The replication protocol, precisely

A partition’s replicas are not symmetric. The leader holds the authoritative log; followers are observers that pull. The cycle, per follower:

The follower sends a Fetch request carrying its own fetch offset (i.e. “I have everything up to offset N, give me N onward”).
The leader uses that offset as an implicit acknowledgment — “this follower has replicated through N” — and updates its view of each follower’s position.
The leader advances the partition’s high-watermark (HW) to the minimum LEO across all current ISR members. The HW only moves when the slowest in-sync follower catches up. That’s the whole point: the HW is a promise that every ISR replica has the data, so any of them can become leader without losing it.
Consumers (and acks=all acknowledgments) are bounded by the HW, never the leader’s LEO.

A follower is in the ISR as long as it keeps fetching within replica.lag.time.max.ms. Fall behind (GC pause, slow disk, network) and the leader ejects it from the ISR; the HW can then advance without waiting for it. Crucially, a replica that was ejected and rejoins must catch up to within the current leader’s epoch before it’s allowed back into the ISR — letting it in early was the cause of a real committed-data-loss bug (KAFKA-7128).

13.2 Why the high-watermark alone wasn’t safe — the leader-epoch fix

The classic data-loss scenario, worth being able to reconstruct on a whiteboard:

Brokers A (follower) and B (leader). A fetches record m2 from B, so A has m2 in its log — but the second round-trip that would advance the HW past m2 hasn’t happened yet, so neither broker’s HW covers m2.
A restarts. Under the old protocol it truncates its log down to its last known HW — dropping m2.
Before B can re-replicate, B fails and A is elected leader. m2 is now gone everywhere, even though it had been acknowledged as committed. Worse variants leave A and B with divergent logs at the same offsets.

The root cause: the HW lags the data by a round-trip, so “truncate to HW on recovery” can discard committed records during a fast failover. The fix (KIP-101) is the leader epoch — a monotonically increasing leadership-period ID stamped into the log. On recovery, a follower no longer truncates to the HW; it sends an OffsetsForLeaderEpoch request asking the leader “for epoch E, what offset did it actually end at?” and truncates to that lineage point. Because epochs are authoritative and ordered, divergence becomes impossible and committed data survives. KIP-279 and KIP-320 closed remaining edge cases (fast back-to-back elections; letting consumers detect truncation after unclean elections). The takeaway for intuition: Kafka’s replication safety is real, but it is comparatively recent, was hard-won, and was model-checked (TLA+) into existence. Treat “data below the HW is durable” as true because of the leader epoch, not by magic.

13.3 The consumer group rebalance protocol

“Rebalancing” in Section 7 is a black box; here’s the machinery. Each group has a broker-side group coordinator (the broker that leads the relevant __consumer_offsets partition). Membership is a two-phase dance:

JoinGroup. Every consumer sends JoinGroup to the coordinator with the topics it wants and its supported assignment strategies. The coordinator picks one member as the group leader (just a designated consumer, not a broker) and returns the full member list to it.
SyncGroup. The group leader consumer — not the broker — computes the partition-to-consumer assignment locally (using the chosen assignor) and sends the result back via SyncGroup. The coordinator distributes each member its slice.

Two design consequences worth holding: assignment logic runs client-side (so you can plug in custom assignors without touching brokers), and the coordinator drives liveness via heartbeats (heartbeat.interval.ms / session.timeout.ms) plus the poll-liveness check (max.poll.interval.ms). Miss heartbeats → you’re dead and removed. Heartbeat fine but don’t poll() in time (slow processing) → you’re removed anyway. That dual mechanism is why slow processing triggers rebalances even when the consumer process is perfectly healthy (Section 7, gotcha #2).

Eager vs cooperative. Eager (old): every rebalance revokes all partitions from all members and reassigns from scratch — a stop-the-world pause. Cooperative-sticky (KIP-429, default since 2.4): the assignor computes the new assignment, revokes only the partitions that must move, and does it in incremental rounds so members keep processing everything they retain. Static membership (group.instance.id, KIP-345) goes further: a member that restarts within session.timeout.ms reclaims its previous assignment without triggering a rebalance at all — invaluable for rolling deploys.

13.4 The transaction protocol (exactly-once, under the hood)

EOS is two layers. The idempotent producer handles single-partition dedup: the broker assigns each producer a Producer ID (PID) and the producer tags every record batch with a monotonic sequence number per partition; the broker rejects duplicates and out-of-order batches, killing retry-induced duplicates. The transaction layer handles atomicity across partitions and offset commits, and it’s a genuine two-phase commit:

The producer registers its transactional.id with the transaction coordinator (a broker role; state in __transaction_state) and is issued a PID and a producer epoch. A restarted producer with the same transactional.id gets a higher epoch, which fences the zombie: the old instance’s writes are rejected, so a crashed-then-restarted producer can’t commit a stale transaction.
As the producer writes to partitions, it informs the coordinator which partitions are enrolled.
On commit, the coordinator runs 2PC: it writes a prepare marker to __transaction_state, then writes commit control records into every involved partition (including the __consumer_offsets partition for the atomically-committed input offsets via sendOffsetsToTransaction), then marks the transaction complete. On abort it writes abort markers instead.
A read_committed consumer uses the Last Stable Offset (LSO) — the offset before the earliest still-open transaction — as its visibility ceiling, and uses the control records to skip aborted batches. Records from open transactions are buffered/invisible until resolved.

So the LSO is to transactions exactly what the HW is to replication: a visibility boundary that hides data which isn’t yet safe to expose. And the hard boundary remains: all of this is inside Kafka. The 2PC spans Kafka partitions and Kafka offsets; it has no reach into your database, which is why DB-touching consumers still need outbox/idempotency regardless (Section 8, call #3).

13.5 KRaft as a consensus system

KRaft deserves a distributed-systems-grade description because “it’s Raft” undersells the design choice. A small set of controller nodes are Raft voters; brokers are observers that replicate the metadata log without voting. A metadata change is committed once a majority of voters have it — standard Raft quorum semantics, with Raft-style leader election and terms (epochs). The deliberate divergence from textbook Raft: replication is pull-based, not push-based. In classic Raft the leader tracks each follower and pushes entries; in KRaft, voters and observers fetch from the leader exactly like Kafka’s data-plane replication, which let the team reuse the existing log, fetch, and compaction machinery rather than build a parallel stack (KIP-595). Brokers materialize the metadata log into in-memory cluster state by replay (stream-table duality again), and large/old logs are compacted into snapshots (KIP-630) that a lagging broker fetches via FetchSnapshot instead of replaying from zero. Membership changes are serialized one voter at a time (KIP-853) to avoid disjoint-majority split-brain. The practical upshot already noted in Section 6 — faster failover, locally-cached metadata, no 200K-partition ZooKeeper ceiling — all falls out of “metadata is now just another Kafka-style replicated log with a quorum on top.”

13.6 Storage internals: segments, indexes, and tiered storage

We’ve treated a partition as “an append-only log,” but on disk it’s more structured, and understanding the structure explains both Kafka’s speed and its cost model.

Segments and indexes. A partition’s log is split into segment files (default ~1 GB, or rolled by time). Exactly one segment per partition is active (being appended); the rest are immutable and closed. Alongside each .log segment sit index files: a .index mapping offset → byte position (so a fetch for offset N doesn’t scan the file — it binary-searches the sparse index to the nearest position and reads forward), a .timeindex mapping timestamp → offset (this is what powers offset-reset-to-timestamp and time-based retention), and for transactional topics a .txnindex of aborted-transaction ranges and a producer-snapshot file. Retention and the segment boundary are intertwined: Kafka can only delete whole closed segments, never individual records, which is why retention is coarse-grained and why a single huge active segment can delay reclamation. Log compaction (the changelog mode from Section 8) runs a background cleaner that rewrites segments keeping only the latest value per key, plus tombstones (null-value records) that mark deletions and are themselves garbage-collected after a delay — that’s how a compacted topic becomes a durable materialized table rather than an ever-growing stream.

Tiered storage (KIP-405). The structural problem Kafka hit at scale: storage and compute are welded together. If you want 90 days of retention, you provision 90 days of expensive local broker disk on every replica — and you scale brokers for storage you’ll rarely read instead of for the throughput you actually need. Tiered storage breaks that coupling. With remote.storage.enable=true on a topic, closed segments are offloaded to cheap object storage (S3, GCS, Azure Blob) once they age past local.retention.ms/local.retention.bytes, while the headline retention.ms/retention.bytes now governs how long data lives in the remote tier. Hot, recent data stays on local disk (served by the page cache and zero-copy as always); cold data lives in object storage and is fetched on demand when a consumer reads far enough back.

The mechanism is, predictably, another log-and-metadata-topic pattern. A broker-side RemoteLogManager runs per-partition tasks that, for each rolled segment, copy the .log plus its offset/time/transaction indexes, the producer snapshot, and the leader-epoch cache to the remote store via a pluggable RemoteStorageManager (each cloud has its own plugin). The leader-epoch cache travels with the segment specifically so that the replication-safety machinery from §13.2 still works for data that no longer lives on any broker. Metadata about every remote segment (its offset range, location, epoch lineage) is tracked by a RemoteLogMetadataManager; the default implementation writes it to an internal __remote_log_metadata topic, and every broker consumes all of its partitions to build an in-memory view of what’s where — again, stream-table duality. A read for an offset that’s been tiered transparently fetches the segment (or, in good plugins, prefetched chunks of it) from object storage; the consumer API is unchanged. Two constraints worth knowing because they bite: tiered storage does not support compacted topics and does not support JBOD (multiple log directories), and a topic can’t switch from delete to compact once tiered. The frontier is moving here too — KIP-1176 extends tiering to the active segment (uploading sub-second slices to fast object storage as an intermediate hop between leader and follower), which is the same idea the diskless/S3-native Kafka rebuilds (and the cross-AZ-cost motivation from Judgment Call 11) are chasing: make object storage the primary tier, not just the cold one.

This is the change most relevant if you’ve ever wanted Kafka to be a queue — and it directly attacks two limits this document has hammered: the partition-as-parallelism-ceiling (Core Idea 2) and the absence of per-message acknowledgment (Section 11, downside #4). Status matters because it moved fast: early access in Kafka 4.0, preview in 4.1, and production-ready (GA) in Kafka 4.2 (released February 2026). On Confluent Cloud it’s GA as a managed offering. If you read older material calling it “experimental, do not use in production,” that’s now out of date — though the internal record format changed between 4.0 and 4.1, so 4.0 sandboxes can’t be upgraded.

What a share group is. A share group is a new kind of group that sits alongside (not replacing) classic consumer groups. The defining difference: multiple consumers in a share group can read the same partition concurrently. A partition consumed this way is a share partition, and a broker-side share-partition leader hands records out to group members under a time-limited lock (default 30s), tracking the state of each in-flight record. This severs the partition-to-consumer welding — you can have more consumers than partitions, and scale consumers up and down elastically without repartitioning the topic. For a job/task queue where each record is an independent unit of work and ordering doesn’t matter, this is exactly the model you always wanted and Kafka never had.

Per-record acknowledgment. Instead of a single advancing offset, each delivered record can be individually acknowledged (processed — done), released (put back for redelivery to someone else), rejected (unprocessable — don’t redeliver), or renewed (the RENEW type, added in 4.2, extends the lock for a long-running task). Delivery attempts are counted, so a record that keeps failing can be detected and (with KIP-1191, targeting a later release) routed to a dead-letter topic automatically — no more hand-rolled retry-topic plumbing. New wire APIs (ShareFetch, ShareAcknowledge) back this; state lives in an internal share-group state topic via a share coordinator. The acknowledgment state is what makes this a real queue rather than a shared cursor: a released record genuinely goes to another consumer, and a rejected record genuinely stops being delivered.

What you give up, and the honest framing. Share groups trade away ordering — concurrent consumers on one partition means no per-partition sequence guarantee — and, for now, exactly-once (at-least-once only; EOS for share groups is roadmap, not shipped). So the decision rule sharpens nicely: if records are an ordered stream of related events (the order lifecycle, a change feed, anything stateful), use a classic consumer group and key for ordering. If records are independent units of work (jobs, tasks, notifications, fan-out processing where order is irrelevant), a share group now gives you queue semantics, elastic scaling beyond partition count, and per-message ack/reject/retry — natively, on the same cluster, with Kafka’s durability and replay underneath. The strategic significance: this is Kafka deliberately growing into the one major use case Judgment Call 9 told you to use RabbitMQ/SQS for. That advice isn’t wrong yet — share groups are new, the assignor and tooling are still maturing, and a dedicated queue is still simpler operationally if queuing is all you need — but the gap is closing, and “we already run Kafka, do we still need a separate queue?” is now a legitimate question rather than a settled no.

14. Where to Go Deeper

“Kafka: The Definitive Guide” (2nd ed., Narkhede/Shapira/Palino/Petty, O’Reilly). The canonical book. Read cover to cover once; it’s the single best grounding in the real operational model.
The original LinkedIn paper, “Kafka: a Distributed Messaging System for Log Processing” (Kreps et al., 2011). Short, readable, and shows you the design intent in its purest form. Read it after Section 6 here — it’ll click.
Jay Kreps, “The Log: What every software engineer should know about real-time data’s unifying abstraction.” The essay that explains why the log as a foundational idea, beyond Kafka itself. The piece that makes Core Idea 1 feel inevitable.
The official Apache Kafka docs — the “Design” and “Implementation” sections specifically. Most docs are reference; these two sections are genuinely worth reading for the page-cache, zero-copy, and replication reasoning behind Section 6.
Confluent’s “Exactly-Once Semantics Are Possible” blog post and the KIP-447 writeups. The clearest explanation of how idempotence + transactions + read-committed actually compose, and the honest boundaries of EOS.
KIP-405 (tiered storage) and KIP-932 (queues / share groups), plus the Aiven “Tiered Storage in Depth” two-part series. The primary sources behind §13.6 and §13.7; read the KIP motivations even if you skip the wire-protocol detail — they explain why, which is where the intuition lives.
The KRaft documentation and KIP-500 (plus KIP-595 for the consensus protocol). For understanding the post-ZooKeeper architecture you’ll actually run today, and why KRaft is pull-based rather than textbook Raft.
The replication-safety KIPs — KIP-101 (leader epoch), KIP-279, KIP-320. Read these once if you ever need to reason about data loss during failover; they’re the primary sources behind Section 13.2 and show how the guarantees were actually established (and model-checked).
Hands-on project: stand up a local KRaft broker, build a producer that keys by entity, run multiple consumers in a group, then deliberately break things — kill a consumer mid-processing and watch the rebalance, send a poison pill and watch a partition wedge, reset offsets and replay, and (if you’re feeling brave) force an unclean leader election and confirm the data gap. You’ll learn more from one afternoon of breaking it than from a week of reading.

15. The Final Verdict

Here’s the honest take, off the clock. Kafka is one of the genuinely great pieces of infrastructure software — and it is also routinely chosen for problems it’s wrong for, by teams who underestimate what it costs to run. Both of those are true, and holding both is the whole point.

What it gets profoundly right: the log abstraction. The decision to make storage a dumb append-only file and push “where am I” onto the consumer is the kind of simplification that pays dividends for a decade. It’s why fan-out is free, why replay is a real recovery strategy, why a slow consumer can’t take down your producers, and why the same data can feed five systems at five speeds without anyone coordinating. The performance engineering on top — sequential I/O, page cache, zero-copy — is a masterclass in using the operating system instead of fighting it. And KRaft finally collapsing two stitched-together systems into one is the maturation the project needed.

What it costs you: the log is rigid where you’ll wish it were flexible. The partition welds ordering, parallelism, and placement together so tightly that capacity planning becomes a one-way door, skewed keys become hot partitions you can’t balance away, and “just add partitions” becomes a migration. The operational burden is real and permanent — Kafka rewards a dedicated platform team and quietly punishes teams that don’t have one. And its scariest failures are silent: lag and data loss that creep while the dashboards stay green.

Who should reach for it: teams with high-throughput event streams, multiple independent consumers, a need to replay history, and either the operational depth to run it or the budget for a managed service. Event-driven architectures, log/metric pipelines, CDC, stream processing, event sourcing — this is Kafka’s home turf and nothing else does it as well at scale. Who should not: anyone who needs a task queue, request/reply, per-message priorities or TTLs, or a queryable store — and any small team that needs a simple message bus and would drown in Kafka’s operational tax. If you’re reaching for Kafka to do a queue’s job, you’ve already made a mistake; reach for RabbitMQ, SQS, or your database.

What you should now believe: believe that it’s a log, not a queue, and let that one fact resolve most of your confusion. Believe that at-least-once plus idempotent consumers is the mature default, and that anyone selling you exactly-once across a database is selling you something Kafka can’t deliver. Don’t believe that writing to disk means your data is safe — replication and acks make it safe. And when someone says “we’ll just put it on Kafka,” hear the unspoken operational bill, and ask whether the problem actually wants a log or whether it wants a queue wearing a log’s clothes.

The hard-won line: Kafka doesn’t lose your data — your acks setting, your auto-commit, and your retention policy lose your data, and Kafka does exactly what you configured it to do while you weren’t watching the lag.

The ideas are mine. The writing is AI assisted