Queues, Logs & Backpressure

The difference that matters

Both are async pipes. They differ in one thing: what makes a message disappear.

A queue — RabbitMQ, SQS, ActiveMQ — tracks per-message state in the broker. A consumer acknowledges, the broker deletes. Fan-out means duplicating the queue.

A log — Kafka, Kinesis, Pulsar — is an append-only sequence. Consumers store an offset saying how far they have read; the broker deletes on a retention policy (Kafka's default: 7 days), never on consumption. Reading changes nothing.

That buys what you cannot retrofit. Replay: rewind the offset and reprocess after a bug. Independent consumers: search, analytics and audit read the same records at their own pace, and a fourth arrives next quarter without touching the producer — starting at offset 0 and building its state from history.

The Kafka model, concretely

A topic splits into partitions, each an ordered, immutable, append-only sequence — the unit of parallelism, ordering and storage.

Ordering holds only within a partition — there is no global ordering across a topic at any price. The producer hashes a key to pick one, so every event for order-4471 stays ordered. Choose that key like a database partition key: same decision, same hot-key failure.

Each record has an offset, monotonic within its partition, and a group's committed offsets are its only durable state. Within a group, a partition goes to exactly one consumer — that is how ordering survives parallelism, and it means consumers beyond the partition count sit idle. Groups are independent of each other.

Rebalancing happens when membership changes: a consumer joins, crashes, or misses max.poll.interval.ms (default 5 minutes). The classic eager protocol is stop-the-world — everyone revokes every partition and the group re-forms. Consumption stops for seconds and lag spikes.

The nasty part is the loop: processing slows past the poll interval, a consumer is evicted, its partitions move to the survivors, they slow too, and the group rebalances itself into a stall. Fixes — lower max.poll.records; use the cooperative sticky assignor so only moved partitions pause; set group.instance.id so rolling restarts skip rebalancing.

Choosing the partition count

Partition count is your maximum consumer parallelism. Twelve partitions means at most twelve useful consumers, forever.

You can add partitions; you can never remove them. And adding changes hash(key) % N, so a key that lived in partition 3 starts landing in partition 7 — history and future in different partitions, per-key ordering broken across the change.

So over-provision, within reason. Each partition costs file handles, broker memory and a fetch per consumer per poll, and controller failover scales with total partitions. Size it as target throughput / per-consumer throughput, times 2 to 3 for growth: usually 12 to 50, not 500.

Delivery semantics

Commit the offset before processing: at-most-once, a crash loses the record. Commit after: at-least-once, a crash reprocesses it. Everything real uses at-least-once.

So duplicates are not an edge case, they are the contract, and your consumer must be idempotent — idempotency covers how. Kafka's exactly-once is real but narrow: transactions where the output is also Kafka. Write to Postgres or call a payment API and you are back to an idempotency key.

Consumer lag is the metric

Per partition, lag = log end offset − last committed offset.

• Flat and small — consumers are keeping up.
• Sawtooth — batching, GC pauses, rebalances. Self-correcting.
• Growing linearly — consumer throughput is below producer throughput. It never recovers on its own; waiting only enlarges the backlog.

Alert on maximum lag across partitions, not the sum: one hot partition hours behind vanishes in an aggregate. And convert lag into time — 4 million records at 20k/sec is 200 seconds of backlog, and once that approaches retention you are losing data.

Backpressure

Producers outpacing consumers is not rare; it is Monday. Four responses.

Buffer. The default and the trap. Bounded by definition, so it only defers the question of what happens at the bound.

Drop. Explicit load shedding: sample the metrics stream, stay responsive. Correct for telemetry, unacceptable for orders — resilience covers shedding on purpose.

Block the producer. Kafka's buffer.memory fills and send() blocks. Real backpressure, propagating into the HTTP thread that called it: the async pipeline turns synchronous and the web tier falls over. Sometimes right, but only if you chose it.

Scale consumers. The actual fix — capped at the partition count.

Dead-letter queues

A poison message fails every time — malformed payload, a missing referenced row, an uncoordinated schema change. On a log it is worse than on a queue: there is no per-message ack, so the consumer retries at the same offset and the whole partition stops.

The pattern: retry a bounded number of times (3 to 5, with backoff), then publish to a dead-letter topic and advance the offset. Carry provenance — topic, partition, offset, exception, attempt count — because an untraceable DLQ entry is unreplayable.

Then monitor it. A DLQ nobody watches is a data-loss mechanism with extra steps, worse than plain loss because it looks like a safety net on the diagram. Alert on depth above zero.

When not to use a queue

If the caller needs the answer now, make the call. A queue adds enqueue time, poll interval and processing to the critical path, plus a failure mode the caller cannot see: accepted, 202, never processed.

The tell is a correlation ID, a reply queue and a timeout bolted onto a broker. That is RPC with more moving parts and worse latency — use a synchronous API with a circuit breaker. Reach for a queue when the caller only needs accepted, when the work can be late, or when you need durable buffering across a downstream outage — the last being where a log wins, since the same records also feed stream processing.

What to take away

• Queues delete on consume; logs delete on a clock. Retention is what enables replay.
• Kafka ordering exists only within a partition; the key hash decides which one.
• Partition count caps parallelism, only goes up, and going up breaks per-key ordering.
• Delivery is at-least-once, so consumers must be idempotent.
• Consumer lag is the metric: flat is healthy, linear growth never self-corrects.
• A queue buffers bursts and disguises deficits; an unwatched DLQ loses data politely.