Kafka is a distributed event streaming platform. It provides a structure for publishing and subscribing to large volumes of events in real time. It serves real-time data pipelines, event-driven architectures, log aggregation, and more.

This post covers Kafka’s core concepts and explains why KRaft mode — which removes the ZooKeeper dependency — was introduced.

Topics and Partitions

Topics

In Kafka, messages are published to a Topic. A topic is a logical category of messages. Topics are created per event type: order-events, user-signups, and so on.

A topic is an append-only log that stores messages. Once written, messages are immutable. They are deleted when the retention period expires.

Partitions

A single topic is divided into multiple Partitions. Partitions are the core unit that provides both parallelism and ordering guarantees.

flowchart LR
    subgraph Topic["Topic: order-events"]
        P0["Partition 0
msg0, msg3, msg6..."]
        P1["Partition 1
msg1, msg4, msg7..."]
        P2["Partition 2
msg2, msg5, msg8..."]
    end

Messages within a partition maintain order. Across partitions, no ordering is guaranteed. Messages with the same key are assigned to the same partition, ensuring event ordering for a specific entity (e.g., a particular order).

Increasing partition count increases throughput, since multiple consumers can process each partition in parallel.

Offset

Each message within a partition has a unique Offset number, starting from 0 and incrementing sequentially. Offsets serve as the reference point for tracking “how far a consumer has read.”

Producer

A Producer publishes messages to a topic.

When a producer sends a message, it must decide which partition to target. Three approaches exist.

Key-based partitioning. When a message has a key, a hash of the key determines the partition. The same key always maps to the same partition. This is used when event ordering for a specific user or order is required.

Round robin. Without a key, messages are distributed across partitions in sequence. Suitable when ordering is unnecessary and even load distribution is desired.

Custom partitioner. Custom partitioning logic can be implemented. Used when specific business rules dictate partition selection.

Acks

The producer can configure the level of acknowledgment required from brokers.

  • acks=0: No acknowledgment. Fastest, but messages can be lost.
  • acks=1: Leader broker acknowledges after writing. Messages can still be lost if the leader fails before replication.
  • acks=all: All ISR (In-Sync Replicas) acknowledge. Safest, but increases latency.

Consumer

A Consumer reads messages from a topic. Unlike the producer’s “push,” consumers “pull” messages themselves, processing at their own pace.

Consumers commit the offset of messages they have read. Committed offsets are stored in an internal Kafka topic (__consumer_offsets). When a consumer restarts, it resumes from the last committed offset.

Consumer Groups

Multiple consumers can be grouped into a Consumer Group. Within the same group, each partition is assigned to exactly one consumer.

flowchart LR
    subgraph Topic["Topic (3 Partitions)"]
        P0["P0"]
        P1["P1"]
        P2["P2"]
    end
    subgraph Group["Consumer Group A"]
        C1["Consumer 1"]
        C2["Consumer 2"]
        C3["Consumer 3"]
    end
    P0 --> C1
    P1 --> C2
    P2 --> C3

If the number of consumers exceeds the number of partitions, the excess consumers remain idle. To increase throughput, increase the partition count first.

When consumers join or leave a group, Rebalancing occurs — the process of reassigning partitions. During rebalancing, message processing for that group pauses temporarily.

Multiple Consumer Groups

Different consumer groups read the same topic independently, each managing its own offsets.

flowchart LR
    subgraph Topic["Topic (3 Partitions)"]
        P0["P0"]
        P1["P1"]
        P2["P2"]
    end
    subgraph GA["Group A (Order Processing)"]
        A1["Consumer A1"]
        A2["Consumer A2"]
    end
    subgraph GB["Group B (Analytics)"]
        B1["Consumer B1"]
    end
    P0 --> A1
    P1 --> A2
    P2 --> A1
    P0 --> B1
    P1 --> B1
    P2 --> B1

Multiple consumer groups subscribing to a single topic is the pub/sub pattern. A common example: an order processing system and an analytics system independently consuming the same events.

Brokers and Clusters

Broker

A Broker is a Kafka server instance. It receives messages, persists them to disk, and delivers them to consumers. Multiple brokers form a Cluster.

Each partition is assigned to one broker as the Leader. Producers and consumers communicate with the leader broker.

Replication

Partitions are replicated across multiple brokers. If the leader fails, one of the followers is promoted to the new leader.

flowchart TB
    subgraph Cluster["Kafka Cluster"]
        subgraph B1["Broker 1"]
            P0L["P0 (Leader)"]
            P1F["P1 (Follower)"]
        end
        subgraph B2["Broker 2"]
            P0F["P0 (Follower)"]
            P1L["P1 (Leader)"]
        end
        subgraph B3["Broker 3"]
            P0F2["P0 (Follower)"]
            P1F2["P1 (Follower)"]
        end
    end
    P0L -.->|replication| P0F
    P0L -.->|replication| P0F2
    P1L -.->|replication| P1F
    P1L -.->|replication| P1F2

ISR (In-Sync Replicas) is the set of replicas synchronized with the leader. If a follower falls behind, it is removed from the ISR. With acks=all, writes are acknowledged only after all ISR replicas have recorded the message.

min.insync.replicas sets the minimum ISR count. With a replication factor of 3 and min ISR of 2, writes succeed even if one broker fails. If two brokers fail, writes are rejected to protect data consistency.

ZooKeeper and Its Limitations

Before Kafka 3.3, ZooKeeper managed cluster metadata: broker lists, topic/partition configurations, controller election, and ACL information.

The ZooKeeper-based architecture had several problems.

Operational overhead of a separate system. A ZooKeeper cluster (typically 3-5 nodes) must be operated alongside the Kafka cluster. Monitoring, upgrades, and incident response targets double.

Metadata propagation bottleneck. Brokers fetch metadata from ZooKeeper, so as partition counts grow, metadata synchronization takes longer. This slows controller failover recovery in large clusters.

Dual consensus problem. ZooKeeper runs its own consensus algorithm (ZAB), while Kafka separately operates ISR-based replication. The two systems can temporarily fall out of sync.

KRaft Mode

KRaft (Kafka Raft) removes ZooKeeper and lets Kafka manage metadata internally. Production use became available in Kafka 3.3, and ZooKeeper mode was removed starting from 4.0.

In KRaft, some brokers take on the Controller role. Controller nodes use the Raft consensus algorithm to agree on a metadata log. Metadata is stored in an internal Kafka topic, eliminating the need for a separate system.

Key changes from ZooKeeper mode:

  • No ZooKeeper cluster. The operational target reduces to Kafka alone.
  • Metadata is managed as an event log. Brokers subscribe to the metadata log and maintain their own state. Propagation is faster than polling from ZooKeeper.
  • Controller failover speeds up. The Raft protocol elects a new leader who takes over the metadata log.

Summary

Kafka’s core consists of topics, partitions, and consumer groups. Partitions provide parallelism and ordering guarantees. Consumer groups enable horizontal scaling. Broker replication ensures fault tolerance.

KRaft mode removed ZooKeeper as an external dependency from this structure. Kafka now handles metadata consensus and management on its own.