Kafka

Introduction

Apache Kafka is a distributed event streaming platform for high-performance data pipelines and streaming applications.

Core Concepts

Topics

Categories or feeds to which records are published.

Partitions

Topics are split into partitions for parallel processing.

Producers

Applications that publish messages to Kafka topics.

Consumers

Applications that subscribe to topics and process messages.

Consumer Groups

Multiple consumers working together to consume a topic.

Architecture

Producer → Broker (Topic/Partition) → Consumer
              ↓
           ZooKeeper (coordination)

Basic Commands

# Create a topic
kafka-topics.sh --create --topic my-topic --partitions 3 --replication-factor 2

# List topics
kafka-topics.sh --list --bootstrap-server localhost:9092

# Produce messages
kafka-console-producer.sh --topic my-topic --bootstrap-server localhost:9092

# Consume messages
kafka-console-consumer.sh --topic my-topic --from-beginning --bootstrap-server localhost:9092

Use Cases

• Real-time analytics
• Log aggregation
• Event sourcing
• Stream processing