Real-world distributed systems generate continuous, unbounded streams of data—user clicks, IoT sensor readings, financial transactions, and server telemetry. In an ideal world, events would arrive at the processing engine instantaneously in the exact order they occurred.
In reality, mobile devices go into subway tunnels, network packets drop, and servers experience clock drift. Events arrive out-of-order, delayed by seconds, minutes, or days. Batch processing engines (Hadoop, MapReduce) introduce high latency by dividing data into arbitrary hourly chunks.
This module deconstructs modern stream processing foundations:
- The Time Triad: Event Time (when the event occurred) vs. Ingestion Time (when it entered Kafka) vs. Processing Time (when the worker CPU executes it).
- Watermark Mathematics: How watermarks act as temporal progress metrics, telling the engine when all events up to timestamp have arrived with high probability.
- Windowing Topologies: Fixed/Tumbling windows, overlapping Sliding windows, and dynamic gap-based Session windows.
- Stateful Snapshotting: How Apache Flink implements Chandy-Lamport asynchronous barrier snapshotting to recover state without replaying full message logs.