Data Engineering using Kafka and Spark Structured Streaming

A comprehensive Data Engineering course on building streaming pipelines using Kafka and Spark Structured Streaming

Created by: Durga Viswanatha Raju Gadiraju
CEO at ITVersity and CTO at Analytiqs, Inc

Created by: Madhuri Gadiraju

Created by: Pratik Kumar

Created by: Sathvika Dandu

Created by: Phani Bhushan Bozzam
Aspiring Project Manager & Creative UI/UX Designer

Created by: Siva Kalyan Geddada

Created by: Raghu Raman Bhattathirippad
Mentor, SME and passionate content creator - All about Data

What Will I Learn?

• Setting up self support lab with Hadoop (HDFS and YARN), Hive, Spark, and Kafka
• Overview of Kafka to build streaming pipelines
• Data Ingestion to Kafka topics using Kafka Connect using File Source
• Data Ingestion to HDFS using Kafka Connect using HDFS 3 Connector Plugin
• Overview of Spark Structured Streaming to process data as part of Streaming Pipelines
• Incremental Data Processing using Spark Structured Streaming using File Source and File Target
• Integration of Kafka and Spark Structured Streaming - Reading Data from Kafka Topics

Requirements

• Laptop with decent configuration
• Decent internet speed to watch the lessons
• Self Support lab (instructions will be provided as part of the course) or ITVersity labs
• Knowledge about Functional Programming (preferably Python or Scala)
• Knowledge or experience using Spark

Target audience

• Experienced ETL Developers who want to learn Kafka and Spark to build streaming pipelines
• Experienced PL/SQL Developers who want to learn Kafka and Spark to build streaming pipelines
• Beginner or Experienced Data Engineers who want to learn Kafka and Spark to build streaming pipelines