Day 20: Kafka Fundamentals

Phase 4 · Aug 10, 2026

Agenda (2–3 hours)

• Read (45 min): Kafka "The Log" by Jay Kreps (blog post); Kafka documentation on partitions and consumer groups
• Study (45 min): Trace a message from producer to consumer; understand partition leadership, replication, and consumer group rebalancing
• Practice (45 min): Run Kafka locally (Docker compose); produce 10,000 messages; consume with two consumers in the same group; observe partition assignment
• Challenge (30 min): Why does Kafka guarantee ordering only within a partition, not across partitions? Design a partitioning strategy for an order events topic

Kafka Architecture

Producers → [Topic (N partitions)] → Consumers

Topic "orders" with 3 partitions:
  Partition 0: [msg1, msg4, msg7, ...]    Leader: broker-1
  Partition 1: [msg2, msg5, msg8, ...]    Leader: broker-2
  Partition 2: [msg3, msg6, msg9, ...]    Leader: broker-3

Each partition is an ordered, immutable, append-only log. Consumers read from an offset; Kafka retains messages for a configurable period (not ephemeral like SQS).

Partitioning and Ordering

Messages to the same partition key go to the same partition:

partition = hash(key) % num_partitions

Order is guaranteed only within a partition.

Partitioning strategy for orders:

• Key by order_id → all events for an order go to the same partition → ordering guaranteed per order
• Key by customer_id → all orders for a customer are ordered → useful for user-level event sourcing
• No key (round-robin) → maximum throughput, no ordering

Consumer Groups

Multiple consumers can read the same topic independently:

• Each consumer group has its own offset pointer per partition
• Within a group: each partition assigned to exactly one consumer
• Different groups: each gets all messages independently

Topic: orders (3 partitions)
Consumer Group "payments": 3 consumers → partition 0→c1, 1→c2, 2→c3
Consumer Group "analytics": 1 consumer → all 3 partitions → 1 consumer

Scaling: add consumers to a group → partitions are rebalanced. Max consumers = num partitions.

Delivery Guarantees

At-most-once: commit offset before processing. Risk: crash after commit, before processing = lost message.
At-least-once: commit offset after processing. Risk: crash after processing, before commit = duplicate.
Effectively-exactly-once: write to idempotent downstream (idempotency key or Kafka transactions).

Most production systems use at-least-once + idempotent consumer.

Kafka vs SQS

Key Takeaways

• Kafka stores messages in ordered partitioned logs; retention enables replay
• Ordering is guaranteed per partition; use a consistent partition key for event ordering
• Consumer groups: one consumer per partition; add consumers to scale (up to num_partitions)
• Use at-least-once + idempotent consumers; Kafka transactions for exactly-once

Tomorrow: Phase 4 Challenge — CQRS event store with projection in Rust.