Azure Data Factory +Synapse Analytics End to End ETL project

Complete end to end Project on OTT platform using Azure Data Factory and Azure Synapse Analytics from Scratch with Labs

Created by: Shanmukh Sattiraju
Azure Data Engineer

What Will I Learn?

• Create incremental pipeline based on last modified date
• Calling Azure Synapse notebook from Azure Data Factory
• Configure CICD from Scratch in Azure Data Factory
• Build end to end project using Azure Data Engineering Services
• Create a Self Hosted Integration Runtime and ingest data from On-premise Environment
• How to transform data with Azure Synapse Analytics
• Create and configure an Azure Data Lake
• Project Oriented Learning of Azure Data Engineering services
• Create an Azure Key vault service
• Using Secrets of Key vault in Linked services
• Create and configure an Azure SQL Database
• Writing PySpark transformation logic in Synapse Notebook
• Use of Triggers in Azure Data Factory
• Create an incremental data load pipeline to ingest file in daily basis
• Build an Automated pipeline in Azure Data Factory
• Setup CICD environment from Scratch
• Create incremental pipeline based on file name and pick today's file
• Perform Data Analysis Using PySpark Code using Synapse Notebook
• Create and configure an Azure Data Factory
• Create and configure an Azure Synapse Analytics
• Orchestrate all pipelines in Azure Data Factory
• Loading Data into Azure SQL Database
• Ingest Data from On-premise to Azure Datalake in Azure Cloud
• Send an Automatic Alert email when a pipeline is failed
• Transform only Today's file in Azure Synapse Analytics
• Report transformed data in Power BI
• Complete hands-on project using Azure Data Engineering Services
• Provides insights on services that needed to clear DP-203
• Understand how GetMetaData, ForEach, If Condition Activities works

Requirements

• Basics of Azure Data Factory
• An Azure account with Subscription to perform Labs
• No Synapse Analytics Experience needed, You will learn everything you needed.

Target audience

• Azure Data Engineers curious to do Hands-on practical Labs
• Data Engineers who want real time project experience using Azure Data Engineering Services
• Students aspiring to learn real time project on Azure Data Engineering
• Folks preparing for DP-203 to get practical understanding