HomeCoursesIntroduction to the fundamentals of data engineering

Task automation and scheduling

Chapter 2 · First steps in data engineering · lesson 9 of 59
Reserved for subscribers — here is the beginning
The rest of this lesson is reserved for subscribers Create your free account to try the sample exercises, then upgrade to full access whenever you like.

Create your free account

No credit card required, in thirty seconds.
Continue with Google
or
Sign up with an e-mail
Already registered? Log in
By continuing, you agree to the terms of use and the privacy policy. Your data is hosted in the European Union.

Full access DATAGOZ

19 €/ month — or 190 € / year
  • All published courses and career paths
  • Exercises and quizzes with automatic grading
  • Practice, exam and certificate of completion
  • No commitment, cancel online
See the plans
This lesson is part of the course

Introduction to the Fundamentals of Data Engineering

short_training6 chapters · 59 lessons

Discover the fundamentals of data engineering to understand data management!

  • The training platformsubscribers
  • Data Engineering Fundamentalssubscribers
  • The missions and responsibilities of a data engineersubscribers
  • Data engineer and data scientist: understanding the differencessubscribers
  • The main challenges of data engineeringsubscribers
  • The technologies and tools of the data engineersubscribers
  • An overview of the different types of databasessubscribers
  • Data processing management and executionsubscribers
  • Task automation and schedulingyou are here
  • Discovering cloud service providerssubscribers
  • The benefits of cloud computingsubscribers
  • Overview of the main cloud providerssubscribers
  • The different services offered in the cloudsubscribers
  • The different database management systemssubscribers
  • Comparing SQL and NoSQL databasessubscribers
  • Designing and organizing database schemassubscribers
  • Combining tables using relational joinssubscribers
  • Data modeling with the star schemasubscribers
  • Understanding the principles of parallel computingsubscribers
  • The advantages of parallel data processingsubscribers
  • Breaking down a process into several independent taskssubscribers
  • Manipulating data with DataFramessubscribers
  • The main distributed computing frameworkssubscribers
  • Discovering Spark, Hadoop and Hive technologiessubscribers
  • Grouping and aggregating data with PySparksubscribers
  • Running and managing PySpark scriptssubscribers
  • Workflow scheduling and orchestration toolssubscribers
  • Automating processes with Airflow, Prefect and cronsubscribers
  • Creating and organizing DAGs with Apache Airflowsubscribers
  • The different data extraction methodssubscribers
  • Identify and explore data sourcessubscribers
  • Extract information from an APIsubscribers
  • Import data from a databasesubscribers
  • Data transformation stepssubscribers
  • Extract and break down information about rental pricessubscribers
  • Prepare and organize transformation operationssubscribers
  • Enrich data with joins on reviewssubscribers
  • Data loading methodssubscribers
  • Understand the differences between OLAP and OLTPsubscribers
  • Export data to filessubscribers
  • Insert and store data in PostgreSQLsubscribers
  • Assemble the different steps of an ETL processsubscribers
  • Build a Directed Acyclic Graph (DAG)subscribers
  • Set up an Apache Airflow environmentsubscribers
  • Analyze and understand the structure of a DAGsubscribers
  • Get to know the course documentationsubscribers
  • Discover the structure of the databasesubscribers
  • Explore the data using SQL queriessubscribers
  • Calculate the average rating given to each coursesubscribers
  • Use the reviews to make recommendationssubscribers
  • Detect and exclude erroneous datasubscribers
  • Apply the transformations needed to generate recommendationssubscribers
  • Automate the scheduling of daily processing jobssubscribers
  • Determine the destination table for the datasubscribers
  • Build the DAG to orchestrate the processsubscribers
  • Activate and run the DAGsubscribers
  • Query the data to get the generated recommendationssubscribers
  • Data Engineering Project Reviewsubscribers
  • Course conclusionsubscribers
Next lesson

Discovering cloud service providers

View the programme Create an account to continue