Data Engineering using Databricks on AWS and Azure

Build Data Engineering Pipelines using Databricks core features such as Spark, Delta Lake, cloudFiles, etc.

Created by: Durga Viswanatha Raju Gadiraju
CEO at ITVersity and CTO at Analytiqs, Inc

Created by: Pratik Kumar

Created by: Sathvika Dandu

Created by: Madhuri Gadiraju

Created by: Sai Varma

Created by: Phani Bhushan Bozzam
Aspiring Project Manager & Creative UI/UX Designer

Created by: Anushka Chakraborty

What Will I Learn?

• Data Engineering leveraging Databricks features
• Databricks CLI to manage files, Data Engineering jobs and clusters for Data Engineering Pipelines
• Deploying Data Engineering applications developed using PySpark on job clusters
• Deploying Data Engineering applications developed using PySpark using Notebooks on job clusters
• Perform CRUD Operations leveraging Delta Lake using Spark SQL for Data Engineering Applications or Pipelines
• Perform CRUD Operations leveraging Delta Lake using Pyspark for Data Engineering Applications or Pipelines
• Setting up development environment to develop Data Engineering applications using Databricks
• Building Data Engineering Pipelines using Spark Structured Streaming on Databricks Clusters
• Incremental File Processing using Spark Structured Streaming leveraging Databricks Auto Loader cloudFiles
• Overview of Auto Loader cloudFiles File Discovery Modes - Directory Listing and File Notifications
• Differences between Auto Loader cloudFiles File Discovery Modes - Directory Listing and File Notifications
• Differences between traditional Spark Structured Streaming and leveraging Databricks Auto Loader cloudFiles for incremental file processing.

Requirements

• Programming experience using Python
• Data Engineering experience using Spark
• Ability to write and interpret SQL Queries
• This course is ideal for experienced data engineers to add Databricks as one of the key skill as part of the profile

Target audience

• Beginner or Intermediate Data Engineers who want to learn Databricks for Data Engineering
• Intermediate Application Engineers who want to explore Data Engineering using Databricks
• Data and Analytics Engineers who want to learn Data Engineering using Databricks
• Testers who want to learn Databricks to test Data Engineering applications built using Databricks