Idempotency & the Exactly-Once Myth

The three delivery semantics

At-most-once. Send and don't retry. Never duplicates, sometimes loses. Acceptable for metrics samples, log lines, telemetry — anything where one missing point is invisible.

At-least-once. Retry until acknowledged. Never loses, sometimes duplicates. The default for anything that matters.

Exactly-once. Each message is processed precisely once.

And the fact that everything else follows from:

The practical consequence: stop trying to make delivery exact, and make processing insensitive to duplicates.

Natural idempotency

An operation is idempotent if applying it repeatedly has the same effect as applying it once.

Some operations are naturally idempotent, and you can often choose the idempotent form:

Not idempotent	Idempotent equivalent
`balance = balance - 10`	`balance = 90` (absolute), or guarded by a transfer ID
`INSERT INTO orders …`	`INSERT … ON CONFLICT (order_id) DO NOTHING`
`counter++`	Record the event, count distinct events
`POST /orders`	`PUT /orders/{client-generated-id}`
"Send the email"	"Ensure an email for event X has been sent"
`list.append(x)`	`set.add(x)`

The pattern: state the desired end state rather than the delta, or attach an identity to the operation so repeats are recognisable. HTTP encodes this distinction directly — PUT and DELETE are defined as idempotent, POST is not, which is why "create with a client-supplied ID via PUT" is such a common API design.

Idempotency keys

When the operation can't be naturally idempotent, make it so explicitly. This is the Stripe model and it's the standard answer:

The client generates a unique key per logical operation (a UUID) and sends it as a header. Crucially, the key is generated once, before the first attempt, and reused on every retry — a key generated per HTTP call defeats the whole mechanism.
The server, in a transaction, attempts to insert the key into an idempotency_keys table with a unique constraint.
If the insert succeeds, this is the first attempt: do the work, store the response against the key, return it.
If the insert fails on the unique constraint, this is a retry: return the stored response without re-executing.

Details that matter in practice:

Store the response, not just the key. The retrying client needs the original answer — the order ID, the charge ID — not a bare "already done".
Handle the in-flight case. If a retry arrives while the first attempt is still running, return 409 Conflict and let the client retry; don't run it twice, and don't block forever.
Make key insertion and the work atomic. Same database transaction. If they're separate, a crash between them puts you back where you started.
Expire keys. 24 hours is typical. Retries after that are somebody else's problem, and the table would otherwise grow without bound.
Scope the key to the caller so one tenant can't collide with or probe another's keys.

The transactional outbox

Related and equally important: how do you atomically update the database and publish an event?

Dual writes fail — as choosing a datastore covered, a crash between the two leaves them inconsistent forever.

The outbox pattern:

In the same transaction as the business write, insert a row into an outbox table.
A separate process reads unpublished outbox rows and publishes them, marking them sent.

Atomicity is free, because it's one transaction in one database. Publishing is at-least-once, because the publisher can crash after sending and before marking — so consumers must be idempotent, which is the theme of this lesson.

The read side of the same idea is change data capture: read the database's replication log directly and derive events from it. Same guarantee, no application-level outbox table.

Effectively-once in stream processing

Kafka's "exactly-once semantics" is worth understanding precisely, since it's frequently misdescribed.

It works for the consume → process → produce pattern, by making the output writes and the consumer offset commit part of one atomic transaction. If processing fails, neither the output nor the offset advance, so the retry produces the same result — and downstream consumers reading with read_committed never see the aborted attempt.

What it does not do is make an external side effect exactly-once. If your processor charges a credit card or sends an email, Kafka transactions cannot roll that back. Exactly-once applies within the transactional boundary; everything outside it still needs an idempotency key.

Deduplication, when you can't be idempotent

Sometimes you need to detect duplicates rather than tolerate them:

A dedup store keyed by message ID with a TTL — Redis with SET NX and an expiry is the common implementation. Bounded memory, bounded window.
A Bloom filter for very high volume where a small false-positive rate (dropping a genuine message) is acceptable. Usually it isn't.
A monotonic sequence per producer — track the highest sequence seen and reject anything at or below it. This is what Kafka's idempotent producer does internally.

Every one of these has a window. A duplicate arriving after the window expires will be processed again. Size the window from your maximum realistic retry delay, and be explicit that it's a window rather than a guarantee.

What to take away

Exactly-once delivery is impossible; exactly-once effect is achievable with at-least-once delivery plus deduplication.
Prefer naturally idempotent operations: absolute values over deltas, PUT with a client-supplied ID over POST.
Idempotency keys: generated once by the client, unique constraint, response stored, atomic with the work, expired on a schedule.
The outbox pattern makes "write and publish" atomic; CDC achieves the same from the log.
Kafka's exactly-once covers its transaction boundary, not your external side effects.
Dedup windows are windows. Make the side effect carry the key wherever you can.

Check yourself

Why is exactly-once delivery impossible in a distributed system?
This is the two generals problem. The final acknowledgement is itself unconfirmable, so the sender faces an unavoidable choice between resending (possible duplicate) and not resending (possible loss). Exactly-once effect is still achievable by making processing tolerate duplicates.
A client generates a fresh idempotency key on every HTTP attempt, including retries. What happens?
The key identifies the logical operation, not the network attempt. If it changes on retry, every attempt inserts a distinct key and executes the work again. The key must be generated once, before the first attempt, and reused across all retries of that same operation.
You need to update the database and publish an event atomically. What is the correct approach?
Dual writes cannot be made atomic by ordering or retries — a crash between them leaves permanent inconsistency. Writing the event to an outbox table inside the business transaction makes it atomic by construction, and a separate publisher then delivers at-least-once to idempotent consumers.
Your consumer checks whether an order was already processed, then charges a card, then records the result. It crashes after charging but before recording. What happens on retry?
The idempotency check only protects operations recorded before the crash, and this one was not. The check and effect must be atomic, or the idempotency key must be passed through to the payment provider so it deduplicates on its side — which is why payment APIs all accept one.