Master Apache Spark using Spark SQL and PySpark 3

Master Apache Spark using Spark SQL as well as PySpark with Python3 with complementary lab access

Created by: Durga Viswanatha Raju Gadiraju
CEO at ITVersity and CTO at Analytiqs, Inc

Created by: Madhuri Gadiraju

Created by: Sathvika Dandu

Created by: Pratik Kumar

Created by: Sai Varma

Created by: Phani Bhushan Bozzam
Aspiring Project Manager & Creative UI/UX Designer

Created by: Siva Kalyan Geddada

What Will I Learn?

• Setup the Single Node Hadoop and Spark using Docker locally or on AWS Cloud9
• Review ITVersity Labs (exclusively for ITVersity Lab Customers)
• All the HDFS Commands that are relevant to validate files and folders in HDFS.
• Quick recap of Python which is relevant to learn Spark
• Ability to use Spark SQL to solve the problems using SQL style syntax.
• Pyspark Dataframe APIs to solve the problems using Dataframe style APIs.
• Relevance of Spark Metastore to convert Dataframs into Temporary Views so that one can process data in Dataframes using Spark SQL.
• Apache Spark Application Development Life Cycle
• Apache Spark Application Execution Life Cycle and Spark UI
• Setup SSH Proxy to access Spark Application logs
• Deployment Modes of Spark Applications (Cluster and Client)
• Passing Application Properties Files and External Dependencies while running Spark Applications

Requirements

• Basic programming skills using any programming language
• Self support lab (Instructions provided) or ITVersity lab at additional cost for appropriate environment.
• Minimum memory required based on the environment you are using with 64 bit operating system
• 4 GB RAM with access to proper clusters or 16 GB RAM to setup environment using Docker

Target audience

• Any IT aspirant/professional willing to learn Data Engineering using Apache Spark
• Python Developers who want to learn Spark to add the key skill to be a Data Engineer
• Scala based Data Engineers who would like to learn Spark using Python as Programming Language