Hadoop Developer In Real World

Data Engineering and Hadoop tutorial with MapReduce, HDFS, Spark, Flink, Hive, HBase, MongoDB, Cassandra, Kafka + more!

Created by: Sundog Education by Frank Kane
Join over 900K students learning ML, AI, AWS, and Data Eng.

Created by: Frank Kane
Ex-Amazon Sr. Engineer and Sr. Manager, CEO Sundog Education

Created by: Sundog Education Team
Sundog Education Team

What Will I Learn?

• Design distributed systems that manage "big data" using Hadoop and related data engineering technologies.
• Use HDFS and MapReduce for storing and analyzing data at scale.
• Use Pig and Spark to create scripts to process data on a Hadoop cluster in more complex ways.
• Analyze relational data using Hive and MySQL
• Analyze non-relational data using HBase, Cassandra, and MongoDB
• Query data interactively with Drill, Phoenix, and Presto
• Choose an appropriate data storage technology for your application
• Understand how Hadoop clusters are managed by YARN, Tez, Mesos, Zookeeper, Zeppelin, Hue, and Oozie.
• Publish data to your Hadoop cluster using Kafka, Sqoop, and Flume
• Consume streaming data using Spark Streaming, Flink, and Storm

Requirements

• You will need access to a x86-based PC running 64-bit Windows, MacOS, or Linux with an Internet connection and at least 8GB of *free* (not total) RAM, if you want to participate in the hands-on activities and exercises. If your PC does not meet these requirements or you only have an M1-based Mac available, you can still follow along in the course without doing hands-on activities.
• Some activities will require some prior programming experience, preferably in Python or Scala.
• A basic familiarity with the Linux command line will be very helpful.

Target audience

• Software engineers and programmers who want to understand the larger Hadoop ecosystem, and use it to store, analyze, and vend "big data" at scale.
• Project, program, or product managers who want to understand the lingo and high-level architecture of Hadoop.
• Data analysts and database administrators who are curious about Hadoop and how it relates to their work.
• System architects who need to understand the components available in the Hadoop ecosystem, and how they fit together.