This lesson is part of the course
Introduction to the Fundamentals of Data Engineering
short_training6 chapters · 59 lessons
Discover the fundamentals of data engineering to understand data management!
1. Welcome message1 lesson
The training platformsubscribers
2. First steps in data engineering12 lessons
Data Engineering Fundamentalssubscribers The missions and responsibilities of a data engineeryou are here Data engineer and data scientist: understanding the differencessubscribers The main challenges of data engineeringsubscribers The technologies and tools of the data engineersubscribers An overview of the different types of databasessubscribers Data processing management and executionsubscribers Automation and task schedulingsubscribers Discovering cloud service providerssubscribers The benefits of cloud computingsubscribers Overview of the main cloud providerssubscribers The different services offered in the cloudsubscribers
3. Essential tools for data engineering16 lessons
The different database management systemssubscribers Comparing SQL and NoSQL databasessubscribers Designing and organizing database schemassubscribers Combining tables using relational joinssubscribers Data modeling with the star schemasubscribers Understanding the principles of parallel computingsubscribers The advantages of parallel data processingsubscribers Breaking down a process into several independent taskssubscribers Manipulating data with DataFramessubscribers The main distributed computing frameworkssubscribers Discovering Spark, Hadoop and Hive technologiessubscribers Grouping and aggregating data with PySparksubscribers Running and managing PySpark scriptssubscribers Workflow scheduling and orchestration toolssubscribers Automating processes with Airflow, Prefect and cronsubscribers Creating and organizing DAGs with Apache Airflowsubscribers
4. Extract, Transform and Load (ETL)16 lessons
The different data extraction methodssubscribers Identify and explore data sourcessubscribers Extract information from an APIsubscribers Import data from a databasesubscribers Data transformation stepssubscribers Extract and break down information about rental pricessubscribers Prepare and organize transformation operationssubscribers Enrich data with joins on reviewssubscribers Data loading methodssubscribers Understand the differences between OLAP and OLTPsubscribers Export data to filessubscribers Insert and store data in PostgreSQLsubscribers Assemble the different steps of an ETL processsubscribers Build a Directed Acyclic Graph (DAG)subscribers Set up an Apache Airflow environmentsubscribers Analyze and understand the structure of a DAGsubscribers
5. Hands-on study of a data engineering project13 lessons
Get to know the course documentationsubscribers Discover the structure of the databasesubscribers Explore the data using SQL queriessubscribers Calculate the average rating given to each coursesubscribers Use the reviews to make recommendationssubscribers Detect and exclude erroneous datasubscribers Apply the transformations needed to generate recommendationssubscribers Automate the scheduling of daily processing jobssubscribers Determine the destination table for the datasubscribers Build the DAG to orchestrate the processsubscribers Activate and run the DAGsubscribers Query the data to get the generated recommendationssubscribers Data Engineering Project Reviewsubscribers
6. Conclusion1 lesson
Course conclusionsubscribers