Data Engineering and Data Integration Tools

Master SQL, Python, and Apache Spark (PySpark) with Hands-On Projects using Databricks on Google Cloud

Created by: Durga Viswanatha Raju Gadiraju
CEO at ITVersity and CTO at Analytiqs, Inc

Created by: Madhuri Gadiraju

Created by: Pratik Kumar

Created by: Phani Bhushan Bozzam
Aspiring Project Manager & Creative UI/UX Designer

Created by: Anushka Chakraborty

What Will I Learn?

• Setup Environment to learn SQL and Python essentials for Data Engineering
• Database Essentials for Data Engineering using Postgres such as creating tables, indexes, running SQL Queries, using important pre-defined functions, etc.
• Data Engineering Programming Essentials using Python such as basic programming constructs, collections, Pandas, Database Programming, etc.
• Data Engineering using Spark Dataframe APIs (PySpark) using Databricks. Learn all important Spark Data Frame APIs such as select, filter, groupBy, orderBy, etc.
• Data Engineering using Spark SQL (PySpark and Spark SQL). Learn how to write high quality Spark SQL queries using SELECT, WHERE, GROUP BY, ORDER BY, ETC.
• Relevance of Spark Metastore and integration of Dataframes and Spark SQL
• Ability to build Data Engineering Pipelines using Spark leveraging Python as Programming Language
• Use of different file formats such as Parquet, JSON, CSV etc in building Data Engineering Pipelines
• Setup Hadoop and Spark Cluster on GCP using Dataproc
• Understanding Complete Spark Application Development Life Cycle to build Spark Applications using Pyspark. Review the applications using Spark UI.

Requirements

• Laptop with decent configuration (Minimum 4 GB RAM and Dual Core)
• Sign up for GCP with the available credit or AWS Access
• Setup self support lab on cloud platforms (you might have to pay the applicable cloud fee unless you have credit)
• CS or IT degree or prior IT experience is highly desired

Target audience

• Computer Science or IT Students or other graduates with passion to get into IT
• Data Warehouse Developers who want to transition to Data Engineering roles
• ETL Developers who want to transition to Data Engineering roles
• Database or PL/SQL Developers who want to transition to Data Engineering roles
• BI Developers who want to transition to Data Engineering roles
• QA Engineers to learn about Data Engineering
• Application Developers to gain Data Engineering Skills