Message Queue (Kafka)

A component that holds messages sent by producers until consumers process them, enabling asynchronous, decoupled communication between services.

• Decoupling: producer and consumer evolve and scale independently and needn't be online simultaneously.
• Load leveling: the queue buffers spikes so consumers process at a steady rate instead of being overwhelmed.

A producer publishes/sends messages to the broker. A consumer subscribes to and reads/processes those messages.

A named category/stream to which records are published, e.g. payments. Consumers subscribe to topics. Each topic is divided into partitions for scalability.

Order processing: a checkout service publishes an "order placed" event; inventory, email, and analytics services each consume it independently, so checkout responds instantly without waiting for all downstream work.

Apache Kafka, RabbitMQ, Amazon SQS/SNS, Google Pub/Sub, Apache Pulsar, and NATS.

A queue delivers each message to exactly one consumer (work distribution). Pub/sub broadcasts each message to all subscribers. Kafka supports both via consumer groups.

A partition is an ordered, immutable, append-only log. Splitting a topic into partitions allows parallel writes and reads across brokers and consumers, which is how Kafka scales throughput. Each partition is consumed by at most one member of a consumer group at a time.

Ordering is guaranteed only within a single partition. Messages sharing a key are hashed to the same partition, preserving their relative order. Across partitions there is no global order. If you need per-entity ordering (e.g. per user), key by that entity.

An offset is the sequential position of a record within a partition. Each consumer group stores the last committed offset per partition (in an internal Kafka topic). On restart it resumes from the committed offset. Committing after processing gives at-least-once; committing before gives at-most-once.

A set of consumers that cooperatively read a topic. Kafka assigns each partition to exactly one consumer in the group, so adding consumers (up to the partition count) increases parallelism. Different groups each get all messages independently, enabling multiple pipelines over the same topic.

Kafka: a distributed log; retains messages, allows replay, optimized for high-throughput streaming and many consumers; consumers pull and track offsets. RabbitMQ: a traditional broker with flexible routing (exchanges, bindings), per-message acks, and push delivery; great for complex routing and request/RPC patterns, but messages are typically removed once consumed. Kafka favors streaming/analytics; RabbitMQ favors task queues and routing.

A separate queue/topic where messages that repeatedly fail processing are sent after exceeding a retry limit. It prevents a bad ("poison") message from blocking the pipeline and lets operators inspect, fix, and replay failures without data loss.

Add consumers to the group up to the number of partitions — each new consumer takes over some partitions (a rebalance). Beyond the partition count, extra consumers sit idle, so throughput is ultimately bounded by partition count. Plan partition counts up front (they're hard to reduce) to allow future scaling.

Lag is the difference between the latest offset produced and the offset a consumer group has committed — how far behind it is. Growing lag means consumers can't keep up with producers, signaling a need to scale consumers, optimize processing, or add partitions. It's the primary health metric for streaming pipelines.

When you need an immediate synchronous response (a user waiting on a result), for simple low-volume calls where the added latency and operational complexity aren't justified, or when strict global ordering across all messages is required. Async messaging adds eventual consistency, debugging complexity, and infrastructure to operate.

Kafka keeps messages on disk for a configured time or size limit regardless of whether they've been consumed, so consumers can re-read or replay history and new consumer groups can start from the beginning. This "retain the log" model is a key difference from queues that delete on consume.

At-most-once: commit offset before processing — no duplicates but messages can be lost on crash. At-least-once: process then commit — no loss but duplicates possible on retry (requires idempotent consumers). Exactly-once: no loss and no duplicates; Kafka achieves it for stream processing via idempotent producers and transactions that atomically write output and commit offsets. True end-to-end exactly-once also requires the consumer's side effects to participate (idempotency or transactional sinks).

Each partition is replicated to multiple brokers; one is the leader, others are followers forming the in-sync replica (ISR) set. acks=all (with min.insync.replicas) means the producer's write is acknowledged only after all in-sync replicas have it, so a single broker failure can't lose the message. acks=1 waits only for the leader (faster, riskier); acks=0 fire-and-forget. Durability trades against latency.

Because at-least-once delivery can redeliver messages (after a crash between processing and offset commit, or on rebalance), consumers must tolerate duplicates. Techniques: use a unique message/event ID and record processed IDs (dedup table / cache) to skip repeats; make the operation naturally idempotent (e.g. upsert by key, set-to-value rather than increment); or use a transactional outbox so downstream effects are deduplicated. Idempotency turns "duplicate delivery" from a bug into a no-op.

It solves the dual-write problem: you can't atomically write to a database and publish to Kafka. Instead, in the same DB transaction you insert the domain change and an "outbox" row describing the event. A separate relay (or CDC tool like Debezium reading the DB log) publishes outbox rows to Kafka and marks them sent. This guarantees the event is published if and only if the transaction committed, giving reliable at-least-once event publishing without distributed transactions.

When members join/leave or partitions change, Kafka reassigns partitions across the group. During a "stop-the-world" rebalance, consumers pause processing, which spikes lag; in-flight work may be reprocessed by the new owner (duplicates). Frequent rebalances (from slow processing tripping max.poll.interval, flapping instances, or scaling) cause instability. Mitigations: cooperative/incremental rebalancing, static group membership, tuning poll intervals, and keeping processing time per poll bounded.

Retry with exponential backoff a bounded number of times (ideally on a separate retry topic to avoid blocking the partition), then route the message to a dead-letter topic with metadata (error, stack, original offset) for later inspection and replay. Never block a partition indefinitely on one message, since Kafka's ordering means everything behind it stalls. Alert on DLQ growth and build tooling to reprocess after a fix.

Base it on target throughput (partition count sets max consumer parallelism and roughly caps throughput), key cardinality (enough partitions to distribute keys without hotspots), and ordering needs (per-key order requires keying). More partitions improve parallelism but increase broker overhead (open files, leader elections, end-to-end latency, rebalance time) and memory. Over-provision modestly since increasing partitions later changes key→partition mapping (breaking ordering) and decreasing is not supported.

Log compaction retains only the latest message per key (instead of deleting by time), so the topic becomes a durable, replayable snapshot of the current value for every key. It's ideal for changelogs, database CDC streams, and materialized state (e.g. a topic that always holds each user's latest profile). Consumers can rebuild full state by reading a compacted topic from the start. Tombstones (null values) delete keys.

Checkout writes the order and an outbox event in one DB transaction; CDC publishes to an orders topic keyed by order ID (per-order ordering). Downstream services (inventory, billing, email) consume with at-least-once semantics and idempotent handlers (dedup on order/event ID, upserts). Use acks=all + replication for durability, retries with backoff and a DLQ for failures, and monitor consumer lag. For multi-step workflows use the saga pattern with compensating actions rather than distributed transactions. This yields no loss (outbox + durable log) and no visible duplicates (idempotency).

Kafka's durable log inherently buffers: producers keep appending and consumers fall behind (lag grows) but nothing is lost within retention. To handle it you scale consumers (up to partition count), optimize per-message processing, increase partitions for more parallelism, or shed/route load. If retention could expire before consumption, you risk losing unprocessed data, so alert on lag approaching retention. Unlike bounded in-memory queues, Kafka pushes back via disk/retention rather than blocking producers, so capacity planning of retention and consumer throughput is essential.

Strict ordering forces messages of a given key onto one partition consumed by one thread, capping that key's throughput and creating hotspots for skewed keys. Relaxing to per-key (rather than global) ordering restores parallelism. Fully unordered processing maximizes throughput but requires the logic to be order-independent. In-order retries also serialize a partition. The design lever is choosing the smallest ordering scope your correctness requires (global → per-key → none) to unlock parallelism.