Big Data Hadoop og Spark udvikler - eLearning

- Introduction to Big Data and Hadoop
- Introduction to Big Data
- Big Data Analytics
- What is Big Data?
- Four vs of Big Data
- Case Study Royal Bank of Scotland
- Challenges of Traditional System
- Distributed Systems
- Introduction to Hadoop
- Components of Hadoop Ecosystem Part One
- Components of Hadoop Ecosystem Part Two
- Components of Hadoop Ecosystem Part Three
- Commercial Hadoop Distributions
- Demo: Walkthrough of Simplilearn Cloudlab
- Key Takeaways
- Knowledge Check

- Hadoop-arkitektur Distribueret lagring (HDFS) og YARN
- Hvad er HDFS
- Behovet for HDFS
- Almindeligt filsystem vs. HDFS
- Karakteristika for HDFS
- HDFS-arkitektur og komponenter
- Implementeringer af høj tilgængelighedsklynge
- HDFS-komponent Filsystemnavneområde
- Opdeling af datablok
- Topologi for datareplikering
- HDFS kommandolinje
- Demo: Almindelige HDFS-kommandoer
- Praksisprojekt: HDFS kommandolinje
- Yarn introduktion
- Anvendelseseksempel for Yarn
- Yarn og dets arkitektur
- Ressourcestyring
- Hvordan ressourcestyringen fungerer
- Application Master
- Hvordan Yarn kører en applikation
- Værktøjer for Yarn-udviklere
- Demo: Gennemgang af klynge del et
- Demo: Gennemgang af klynge del to
- Vigtige pointer videnstjek
- Praksisprojekt: Hadoop-arkitektur, distribueret lagring (HDFS) og Yarn

- Data Ingestion Into Big Data Systems and Etl
- Data Ingestion Overview Part One
- Data Ingestion Overview Part Two
- Apache Sqoop
- Sqoop and Its Uses
- Sqoop Processing
- Sqoop Import Process
- Sqoop Connectors
- Demo: Importing and Exporting Data from MySQL to HDFS
- Practice Project: Apache Sqoop
- Apache Flume
- Flume Model
- Scalability in Flume
- Components in Flume’s Architecture
- Configuring Flume Components
- Demo: Ingest Twitter Data
- Apache Kafka Aggregating User Activity Using Kafka
- Kafka Data Model
- Partitions
- Apache Kafka Architecture
- Demo: Setup Kafka Cluster
- Producer Side API Example
- Consumer Side API
- Consumer Side API Example
- Kafka Connect
- Demo: Creating Sample Kafka Data Pipeline Using Producer and Consumer
- Key Takeaways
- Knowledge Check
- Practice Project: Data Ingestion Into Big Data Systems and ETL

- Distributed Processing Mapreduce Framework and Pig
- Distributed Processing in Mapreduce
- Word Count Example
- Map Execution Phases
- Map Execution Distributed Two Node Environment
- Mapreduce Jobs
- Hadoop Mapreduce Job Work Interaction
- Setting Up the Environment for Mapreduce Development
- Set of Classes
- Creating a New Project
- Advanced Mapreduce
- Data Types in Hadoop
- Output formats in Mapreduce
- Using Distributed Cache
- Joins in MapReduce
- Replicated Join
- Introduction to Pig
- Components of Pig
- Pig Data Model
- Pig Interactive Modes
- Pig Operations
- Various Relations Performed by Developers
- Demo: Analyzing Web Log Data Using Mapreduce
- Demo: Analyzing Sales Data and Solving Kpis Using Pig Practice Project: Apache Pig

- Demo: Wordcount
- Key Takeaways
- Knowledge Check
- Practice Project: Distributed Processing - Mapreduce Framework and Pig

- Apache Hive
- Hive SQL over Hadoop MapReduce
- Hive Architecture
- Interfaces to Run Hive Queries
- Running Beeline from Command Line
- Hive Metastore
- Hive DDL and DML
- Creating New Table
- Data Types Validation of Data
- File Format Types
- Data Serialization
- Hive Table and Avro Schema
- Hive Optimization Partitioning Bucketing and Sampling
- Non-Partitioned Table
- Data Insertion
- Dynamic Partitioning in Hive
- Bucketing
- What Do Buckets Do?
- Hive Analytics UDF and UDAF
- Other Functions of Hive
- Demo: Real-time Analysis and Data Filtration
- Demo: Real-World Problem
- Demo: Data Representation and Import Using Hive
- Key Takeaways
- Knowledge Check
- Practice Project: Apache Hive

- NoSQL Databases HBase
- NoSQL Introduction
- Demo: Yarn Tuning
- Hbase Overview
- Hbase Architecture
- Data Model
- Connecting to HBase
- Practice Project: HBase Shell
- Key Takeaways
- Knowledge Check
- Practice Project: NoSQL Databases - HBase

- Grundlæggende om funktionel programmering og Scala
- Introduktion til Scala
- Demo: Installation af Scala
- Funktionel programmering
- Programmering med Scala
- Demo: Grundlæggende literaler og aritmetisk programmering
- Demo: Logiske operatorer
- Typeinferens, klasser, objekter og funktioner i Scala
- Demo: Typeinferens, funktioner, anonym funktion og klasse
- Samlinger
- Typer af samlinger
- Demo: Fem typer af samlinger
- Demo: Operationer på liste Scala REPL
- Demo: Funktioner i Scala REPL
- Vigtige pointer
- Videnstjek
- Praktikprojekt: Apache Hive

- Apache Spark Next-Generation Big Data Framework
- History of Spark
- Limitations of Mapreduce in Hadoop
- Introduction to Apache Spark
- Components of Spark
- Application of In-memory Processing
- Hadoop Ecosystem vs Spark
- Advantages of Spark
- Spark Architecture
- Spark Cluster in Real World
- Demo: Running a Scala Programs in Spark Shell
- Demo: Setting Up Execution Environment in IDE
- Demo: Spark Web UI
- Key Takeaways
- Knowledge Check
- Practice Project: Apache Spark Next-Generation Big Data Framework

- Introduction to Spark RDD
- RDD in Spark
- Creating Spark RDD
- Pair RDD
- RDD Operations
- Demo: Spark Transformation Detailed Exploration Using Scala Examples
- Demo: Spark Action Detailed Exploration Using Scala
- Caching and Persistence
- Storage Levels
- Lineage and DAG
- Need for DAG
- Debugging in Spark
- Partitioning in Spark
- Scheduling in Spark
- Shuffling in Spark
- Sort Shuffle Aggregating Data With Paired RDD
- Demo: Spark Application With Data Written Back to HDFS and Spark UI
- Demo: Changing Spark Application Parameters
- Demo: Handling Different File Formats
- Demo: Spark RDD With Real-world Application
- Demo: Optimizing Spark Jobs
- Key Takeaways
- Knowledge Check
- Practice Project: Spark Core Processing RDD

- Spark SQL Processing DataFrames
- Spark SQL Introduction
- Spark SQL Architecture
- Dataframes
- Demo: Handling Various Data Formats
- Demo: Implement Various Dataframe Operations
- Demo: UDF and UDAF
- Interoperating With RDDs
- Demo: Process Dataframe Using SQL Query
- RDD vs Dataframe vs Dataset
- Practice Project: Processing Dataframes
- Key Takeaways
- Knowledge Check
- Practice Project: Spark SQL - Processing Dataframes

- Spark Mlib Modellering af Big Data med Spark
- Dataforskerens og dataanalytikerens rolle i Big Data
- Analytik i Spark
- Maskinlæring
- Overvåget læring
- Demo: Klassifikation af lineær SVM
- Demo: Lineær regression med virkelige case-studier
- Uovervåget læring
- Demo: Uovervåget klyngeanalyse K-means
- Forstærket læring
- Semiovervåget læring
- Oversigt over Mlib
- Mlib pipelines
- Vigtige pointer
- Videnstjek
- Praksisprojekt: Spark Mlib - Modellering af Big Data med Spark

- Oversigt over streaming
- Real-time behandling af store datamængder
- Arkitekturer til databehandling
- Demo: Real-time databehandling med Spark Streaming
- Demo: Skrivning af Spark Streaming-applikation
- Introduktion til DStreams
- Transformationer på DStreams
- Designmønstre til brug af Foreachrdd
- Statstilstandsfunktioner
- Vinduesoperationer
- Sammenkædningsoperationer Stream-dataset-sammenkædning
- Demo: Vinduesoperationer i real-time databehandling af streamingkilder
- Demo: Behandling af Twitter streamingdata
- Struktureret Spark Streaming-
- Anvendelsescase: Banktransaktioner
- Model for struktureret streamingarkitektur og dens komponenter
- Output-sinks
- API'er til struktureret streaming
- Konstruktion af kolonner i struktureret streaming
- Vinduesbaserede operationer på event-tid
- Anvendelsescases
- Demo: Streamingpipeline
- Praksisprojekt: Spark Streaming
- Vigtige pointer
- Videnstjek
- Praksisprojekt: Streambehandlingsrammer og Spark Streaming

- Spark GraphX
- Introduction to Graph
- GraphX in Spark
- GraphX Operators
- Join Operators
- GraphX Parallel System
- Algorithms in Spark
- Pregel API
- Use Case of GraphX
- Demo: GraphX Vertex Predicate
- Demo: Page Rank Algorithm
- Key Takeaways
- Knowledge Check
- Practice Project: Spark GraphX Project Assistance