Big Data (Elective-III)

Unit I

Introduction to Big Data: Types of Digital Data, History of Big Data Innovation, Introduction to Big Data Platform, Drivers for Big Data, Big Data Architecture and Characteristics, 5 Vs of Big Data, Big Data Technology Components, Importance and Applications, Big Data Features — Security, Compliance, Auditing and Protection, Privacy and Ethics, Big Data Analytics, Challenges of Conventional Systems, Intelligent Data Analysis, Nature of Data, Analytic Processes and Tools, Analysis vs Reporting, Modern Data Analytic Tools.

Unit II

Hadoop & MapReduce: History of Hadoop, Apache Hadoop, Hadoop Distributed File System, Components of Hadoop, Data Format, Analyzing Data with Hadoop, Scaling Out, Hadoop Streaming, Hadoop Pipes, Hadoop Ecosystem. MapReduce: Framework and Basics, How MapReduce Works, Developing a MapReduce Application, Unit Tests with MRUnit, Test Data and Local Tests, Anatomy of a MapReduce Job Run, Failures, Job Scheduling, Shuffle and Sort, Task Execution, MapReduce Types, Input and Output Formats, MapReduce Features, Real-World MapReduce.

Unit III

HDFS & Hadoop Environment: Design of HDFS, HDFS Concepts, Benefits and Challenges, File Sizes, Block Sizes and Block Abstraction, Data Replication, How HDFS Stores, Reads and Writes Files, Java Interfaces to HDFS, Command Line Interface, Hadoop File System Interfaces, Data Flow, Data Ingest with Flume and Sqoop, Hadoop Archives, Hadoop I/O — Compression, Serialization, Avro and File-Based Data Structures. Hadoop Environment: Cluster Setup and Installation, Hadoop Configuration, Security, Administering Hadoop, HDFS Monitoring & Maintenance, Hadoop Benchmarks, Hadoop in the Cloud.

Unit IV

Hadoop Ecosystem, YARN, NoSQL, Spark & Scala: Hadoop Ecosystem Components, Schedulers (Fair and Capacity), Hadoop 2.0 New Features — NameNode High Availability, HDFS Federation, MRv2, YARN, Running MRv1 in YARN. NoSQL Databases: Introduction. MongoDB: Data Types, Creating/Updating/Deleting Documents, Querying, Indexing, Capped Collections. Spark: Installing Spark, Spark Applications, Jobs, Stages and Tasks, Resilient Distributed Datasets, Anatomy of a Spark Job Run, Spark on YARN. Scala: Classes and Objects, Basic Types and Operators, Built-in Control Structures, Functions and Closures, Inheritance.