Batch and stream are one spectrum
A batch job reads a bounded input and terminates; a stream processor reads an unbounded input and never does. Shrink the batch interval towards one record and you slide continuously from one to the other.
Batch size is not the real difference. A bounded input tells you when the computation is done:
SELECT sum(amount) ... WHERE day = '2026-07-27' finishes because the table ends. A stream has no
end, so Monday's total has no moment at which it becomes final. You have to invent that moment;
windows, watermarks and late-data policy are how.
Event time versus processing time
Event time is the timestamp inside the event, set when the thing happened. Processing time is the wall clock when your operator saw it. They diverge constantly: broker retries (seconds), consumer lag or a rebalance (minutes), a mobile client offline in a tunnel (hours), and backfills, where a month of history arrives in twenty minutes.
Processing time therefore makes results a function of your infrastructure's mood — a 90-second lag spike at 23:59 moves money across a day boundary and nothing records it — and makes reprocessing disagree with the original run, since every processing timestamp differs.
Client clocks lie, though (clocks): a phone set four hours ahead dates events in the future and blows past every watermark. Keep both timestamps, client and broker-append, and clamp client values far ahead of ingest.
Windows
| Window | Shape | State cost | Models |
|---|---|---|---|
| Tumbling | Fixed, non-overlapping; each event in one | One window per key | Hourly billing |
| Hopping / sliding | Length L, advancing every S | Multiplied by L/S, output too | Moving averages |
| Session | Dynamic, closed by an inactivity gap | Unbounded, merges out of order | User behaviour |
The sliding multiplier is where people get hurt: a 1-hour window hopping every minute puts each event in 60 windows — 60x state and 60 output rows per key per hour.
Sessions are the only windows derived from the data rather than the clock, and the only ones that merge: an event landing between two open sessions joins them, so their state cannot be finalised locally.
Watermarks
A watermark is a claim: I expect no more events with event time earlier than T. When it passes
a window's end, the window closes and emits — the only mechanism that turns an unbounded stream
into finite answers. It is a heuristic, not a fact: nothing stops an event stamped T − 5min
arriving afterwards, and the engine honours your bet either way.
It is usually max event time seen − a fixed lateness per input partition, combined as the
minimum across partitions — which is why one idle partition freezes the job's watermark and
every window stops emitting, a direct consequence of partitioning.
| Delay | Latency | Completeness | State |
|---|---|---|---|
| 5 s | Real time | Loses retries and lagging consumers | Minimal |
| 5 min | Dashboard-grade | Covers rebalances and short lag | 60x |
| 6 h | No use for alerting | Covers offline mobile clients | Enormous |
Measure ingest time − event time at p99 and p99.9 rather than guessing: telemetry might be 8 s at
p99 and 6 hours at p99.99, so no watermark is cheap and complete at once.
When late data arrives anyway
Drop it. Legitimate — but emit a counter per window and alert on the rate; this is the silent correctness loss observability exists to surface.
Side output. Route past-watermark events to a separate topic for nightly reconciliation.
Let the window update. Hold state past the watermark for a grace period and re-emit when a late event lands: Flink's allowed lateness, Beam's panes.
The third option's cost lands downstream. Consumers now receive updates, not appends: the sink must upsert by window identity, and any downstream aggregation needs retraction semantics — a negative for the old value beside the new — or its sums drift. Teams pick drop-and-count not for correctness but because correction costs three teams a schema change.
State and what it costs
Windowed aggregation is stateful: a partial aggregate per open window per key. Two million users, 24 hourly windows open across a day, 200 bytes each is about 10 GB per operator; larger windows mean proportionally more state.
State must survive a crash, so engines checkpoint: snapshot operator state to durable storage with the offsets it corresponds to, then restore, rewind and replay on recovery. That works only because the source is a replayable log, not a queue that deletes on read (queues and logs).
Checkpointing is also the main source of pauses: barriers stall the faster inputs and a multi-gigabyte RocksDB state means a multi-gigabyte upload, giving a p99 spike exactly one checkpoint interval apart. Levers: incremental or unaligned checkpoints, a longer interval, or less state.
Exactly-once stops at your process boundary
Exactly-once means output and consumed offsets commit atomically, so a replay after failure cannot double-apply — a Kafka transaction spanning produce and offset commit, or Flink's two-phase commit at the checkpoint. Effectively once, inside the pipeline.
It covers nothing outside it. A Stripe charge, an email, an INSERT into external Postgres: the
replay does them again. Safe side effects need idempotency keys (idempotency).
Backfill, and the Lambda/Kappa argument
The test that matters: replay 30 days of log through the current code. Do you get the same numbers as the original run?
On event time, yes: events carry their own timestamps, so windows fall on the same boundaries. On processing time, no — that bucketing was a function of one Tuesday's consumer lag, which no longer exists. A pipeline that cannot be reproduced cannot be audited or safely changed, which is a stronger argument for event time than day-boundary accuracy.
Lambda ran two pipelines — streaming for fresh approximate results, batch recomputing from raw — at the price of one computation implemented twice and drifting. Kappa keeps one: start a second instance from offset zero, let it catch up, switch reads. With 30 to 90 days of retention that covers nearly every backfill, so the separate batch layer is usually unnecessary complexity.
What to take away
- A stream processor is a batch job whose batch size approaches one; the hard part is knowing when it is done.
- Event time is when it happened, processing time when you saw it — on processing time your numbers track yesterday's lag.
- Windows differ mostly in state cost; hopping a 1-hour window every minute is 60x.
- A watermark is a heuristic assertion, and its delay is a latency budget spent on completeness.
- Late data is dropped-and-counted, side-outputted, or retracted — retraction makes updates the downstream's problem.
- Exactly-once is an atomic offset-plus-output commit; it stops dead at every external side effect.
Check yourself
-
A live dashboard and a nightly recomputation of the same day, from the same log, disagree on daily revenue. What is the most likely cause?
Processing time makes window boundaries depend on when a consumer saw the event, so a lag spike near midnight shifts revenue across days. A stale replica tempts, but replaying the log on event time gives the same answer every run.
-
Which statement about watermarks is correct?
Nothing enforces the assertion — an offline client can deliver hours later, so a late-data path is needed anyway. The guarantee option tempts because engines act on the watermark as if it were true, and believing it is how data goes missing.
-
You enable allowed lateness so windows re-emit corrected results when late events arrive. What must change downstream?
A re-emitting window produces a second record for the same key, so an append-only sink or downstream sum double-counts unless it upserts by window or retracts the old value. Nothing replaces automatically: the processor cannot control how a sink reads it.
-
A Flink job with exactly-once enabled calls a payment API inside an operator. After a failure and restart, can a customer be charged twice?
Exactly-once commits outputs and offsets atomically, so replay does not double-apply inside the pipeline; the external call already happened and no checkpoint rolls it back. Protection needs an idempotency key at the API, not a stronger guarantee.