Messaging

Dumb broker, Smart consumer Model
Kafka does not attempt to track which messages were read by each consumer and only retains unread messages.
Rather, Kafka retains all messages for a set amount of time, and consumers are responsible for tracking their location in each log.
Kafka uses Zookeeper to do leadership election of Kafka Broker and Topic Partition pairs.
Topics are the logical categorization of messages in Kafka model.
Partition: A topic partition is a unit of parallelism in Kafka, i.e. two consumers cannot consume messages from the same partition at the same time.
A consumer can consume from multiple partitions at the same time.
You can think of a Topic as a feed name.
A topic has a Log which is the topic’s storage on disk. A Topic Log is broken up into partitions and segments.
The Kafka Producer API is used to produce streams of data records.
The Kafka Consumer API is used to consume a stream of records from Kafka.
A Broker is a Kafka server that runs in a Kafka Cluster. Kafka Brokers form a cluster.
Best effort delivery across process, TCP, UDP, just once
No ordering

Zookeeper as a general purpose distributed process coordination system
Zookeeper is a system for distributed cluster management. It is a distributed key-value store.
It is highly optimized for reads but writes are slower.
It consists of an odd number of znodes known as an ensemble.
zookeeper as the backbone for maintaining cluster state and leader election. Handles the consensus part.
Zookeeper solves these problems using its magical tree structure file system called nodes, somewhat similar to the Unix file system.
These znodes are analogous to folders and files in a Unix file system with some additional magical abilities.
Zookeeper provides primitive operations to manipulate these znodes, through which we will solve our distributed system problems.
Split Brain Problem

Provide feedback