Azure Data Factory For Data Engineers – Project on Covid19

Real world project for Data Engineers using Azure Data Factory, SQL, Data Lake, Databricks, HDInsight, CI/CD [DP203]

Created by: Ramesh Retnasamy
Senior Data Engineer/ Machine Learning Engineer

What Will I Learn?

• You will learn how to build a real-world data pipeline in Azure Data Factory (ADF).
• You will acquire good Data Engineering skills in Azure using Azure Data Factory (ADF), Azure Data Lake Storage Gen2, Azure SQL Database and Azure Monitor
• You will learn how to ingest data from sources such as HTTP and Azure Blob Storage into Azure Data Lake Gen2 using Azure Data Factory (ADF)
• You will learn how to transform data using Data Flows in Azure Data Factory (ADF) and load into Azure Data Lake Storage Gen2
• You will learn how to transform data using Databricks Notebook Activity in Azure Data Factory (ADF) and load into Azure Data Lake Storage Gen2
• You will learn how to transform data using Azure HDInsight Activity in Azure Data Factory (ADF) and load into Azure Data Lake Storage Gen2
• You will learn how to load transformed data from Azure Data Lake Storage Gen2 to Azure SQL Database using Azure Data Factory (ADF)
• You will learn extensively about Triggers in Azure Data Factory (ADF) and how to use them to schedule the data pipelines.
• You will learn how to monitor pipelines using Azure Data Factory (ADF), Azure Monitor and Log Analytics with a real-world project.
• You will learn how to build production ready pipelines and good practices and naming standards
• You will learn the topics required on Azure Data Factory to pass the Azure Data Engineer Associate Certification Exam DP203
• You will learn about how to create CI/CD pipelines in Azure Devops to release ADF pipelines to higher environments (Testing/ Production)

Requirements

• Basic understanding about cloud computing will be useful, but not necessary.
• Experience in Azure is not required, I will take you through everything necessary to learn this course and build the project
• An Azure Account is required, If you don't have one we will create a free account in the course

Target audience

• University students looking for a career in Data Engineering
• IT developers working on other disciplines trying to move to Data Engineering
• Data Engineers/ Data Warehouse Developers currently working on on-premises technologies, or other cloud platforms such as AWS or GCP who want to learn Azure Technologies
• Data Architects looking to gain an understanding about Azure Data Engineering stack
• Data Scientists who want extend their knowledge into data engineering