Stream Processing and Analytics (Elective-III)

Unit I

Fundamentals of Stream Processing: What is Stream Processing, Examples, Scaling Up Data Processing, Distributed Stream Processing, Introducing Apache Spark. Stream-Processing Model: Sources and Sinks, Immutable Streams, Transformations and Aggregations, Window Aggregations, Stateless and Stateful Processing, The Effect of Time. Practical: Installing and configuring Apache Spark, Scala IDE, JDK.

Unit II

Streaming Architectures: Components of a Data Platform, Architectural Models, Batch-Processing Component in a Streaming Application, Referential Streaming Architectures, Streaming vs Batch Algorithms. Apache Spark as a Stream-Processing Engine: Spark's Memory Usage, Latency, Throughput-Oriented Processing, Fast Data Analysis. Practical: word count over TCP socket stream, sampling tweets with Spark Streaming.

Unit III

Spark's Distributed Processing Model: Cluster Managers, Resilience and Fault Tolerance, Data Delivery Semantics, Microbatching and One-Element-at-a-Time, Dynamic Batch Interval, Structured Streaming Processing Model. Spark's Resilience Model: Resilient Distributed Datasets, Spark Components, Fault-Tolerance Guarantees. Practical: creating RDDs, reading files into RDDs, CSV to RDD with Scala, fault-tolerant integer stream.

Unit IV

Structured Streaming: Programming Model, Structured Streaming in Action, Sources and Sinks, Event-Time-Based Stream Processing. Practical: end-to-end streaming application, structured streaming job, IoT-inspired streaming program, schema handling, custom sinks.

Unit V

Spark Streaming: Programming and Execution Model, Sources and Sinks, Time-Based Stream Processing, Working with Spark SQL, Checkpointing, Monitoring, Performance Tuning. Practical: Spark Streaming Context, DStreams, output operations.