Batch & Stream Processing

Batch processes a bounded, finite dataset — yesterday's logs, the whole user table — in one big job. High throughput, high latency: answers arrive in minutes to hours. Stream processes an unbounded, never-ending flow of events, producing results within seconds of each event arriving.

The split is really input shape. Batch has an end; stream never does. Most real systems run both: a stream layer for fresh, approximate answers and a batch layer for complete, correct ones.

MapReduce is the mental model even when the engine is modern. Map transforms each input record into key-value pairs in parallel. Shuffle groups all pairs with the same key onto the same node. Reduce aggregates each group. Word count: map emits (word, 1), shuffle gathers each word, reduce sums.

The shuffle is where the cost lives — it moves data across the network to co-locate keys, and it's the bottleneck of every batch job. Modern engines like Spark keep the same map/shuffle/reduce model but run in memory and chain stages, cutting the disk round trips MapReduce paid between steps. ETL vs ELT is a related choice: transform before loading into the warehouse (ETL) or load raw and transform inside it (ELT, now the default given cheap warehouse compute).

Stream processing (Flink, Kafka Streams) runs events through a topology of operators — a graph of transforms, filters, and aggregations that data flows through. Operators come in two kinds, and the difference drives everything hard about streaming.

You can't sum an infinite stream — you sum a window of it. Three shapes cover most needs, and the deep problem is which clock defines them.

The subtlety is event time vs processing time. Event time is when the event actually happened; processing time is when your system saw it. They differ because events arrive late and out of order — a phone was offline, a network stalled. Bucket by processing time and a delayed event lands in the wrong window, corrupting the count.

Streaming engines advertise "exactly-once," and it needs unpacking. Delivery over a network is at-least-once; what's achievable is exactly-once effect, built from three pieces: checkpointing (periodically snapshot operator state and input offsets together), replayable sources (rewind the log to the last checkpoint after a crash), and idempotent or transactional sinks (writing the same result twice has no extra effect).

Lambda runs two layers in parallel: a batch layer that recomputes complete, correct results slowly, and a speed layer that gives fast approximate results now. Queries merge the two. The cost is maintaining the same logic twice, in two codebases that drift.

Kappa drops the batch layer. Keep everything as a replayable log and use the stream engine for both: for a "recompute from scratch," just replay the log from the beginning. One codebase, one system. Given durable logs and mature stream engines, kappa is winning.

"Trending topics" and "unique visitors" sound like GROUP BY. At billions of events, exact answers need too much memory — a counter per distinct item, a set of every visitor ID. The move is probabilistic sketches: fixed-size structures that trade a small, bounded error for enormous memory savings.

These merge cleanly, which is what makes them work in a distributed pipeline: each partition builds its own sketch, and you combine them into a global answer without re-scanning the data. Count per window (tumbling or sliding), keep heavy hitters per window, merge across partitions.