Country or region and language

    Choose your country or region and your language.

    Suggested for you

      All countries and regions

        Choose your character

        It shows in the top bar and on your profile. Change it any time.

        How to become a data engineer

        Learn SQL in depth, enough Python to move and check data, a cloud warehouse, tested transformations in dbt and an orchestrator that runs it all on schedule. Every exercise is checked, and the pathway ends in pipeline projects, a timed assessment across the tools and the interview.

        Tools
        SQL, PostgreSQL for engineers, Python, Snowflake SQL, dbt, Dagster, Apache Airflow, Apache Kafka
        Units
        41 units, 208 lessons
        Checked exercises
        1,091
        Cross-tool projects
        3
        Final assessment
        12 tasks, 50 minutes

        About 53 hours of lessons, from the estimate on each lesson.

        Your progress

        Nothing done yet. Progress saves in this browser as you finish lessons, and in your account when you sign in.

        Lessons
        0 of 208 lessons
        Projects
        0 of 3 finished
        Assessment
        Not taken yet

        Start with Query a table

        The pathway, unit by unit

        Take the steps in this order. Each step is a set of units from one course, not the whole course: the units this role uses. Open a unit to start its first lesson.

        1. PostgreSQL for engineers

          Design tables with constraints, choose indexes for a workload and test transaction boundaries in PostgreSQL. Read execution plans and practice safe schema changes before a service depends on the database.

          1. Workload indexesIntermediate · 5 lessons · 26 exercises
          2. Query plansAdvanced · 5 lessons · 26 exercises
          3. Safe schema changesAdvanced · 5 lessons · 28 exercises
        2. Python for data work

          Python moves data between systems and checks it on the way. You learn the language from values and collections to functions, then read and write files and formats, and write code that fails clearly instead of quietly.

          1. BasicsBeginner · 5 lessons · 26 exercisesFree
          2. CollectionsBeginner · 6 lessons · 33 exercises
          3. Control flowBeginner · 5 lessons · 30 exercises
          4. FunctionsBeginner · 4 lessons · 24 exercises
          5. Working with dataIntermediate · 7 lessons · 40 exercises
          6. Reliable codeAdvanced · 4 lessons · 23 exercises
        3. A cloud warehouse: Snowflake

          At many companies, analytics data ends up in a cloud warehouse. You learn your way around Snowflake, query it the Snowflake way, flatten JSON and arrays to the right grain, change data safely and load files incrementally.

          1. Getting around SnowflakeBeginner · 5 lessons · 27 exercisesFree
          2. Query basics, the Snowflake wayBeginner · 6 lessons · 36 exercises
          3. JSON and array grainIntermediate · 7 lessons · 38 exercises
          4. Safe data changesAdvanced · 5 lessons · 28 exercises
          5. Files and incremental pipelinesExpert · 7 lessons · 35 exercises
        4. Orchestration in Dagster

          A pipeline has to run on its own, in order, and tell you when it fails. You define assets and their dependencies, schedule jobs, split work into partitions for backfills, and add checks that catch stale or wrong data.

          1. Assets and the asset graphBeginner · 5 lessons · 27 exercisesFree
          2. Jobs and schedulesIntermediate · 4 lessons · 22 exercises
          3. Partitions and backfillsIntermediate · 6 lessons · 30 exercises
          4. Asset checks and freshnessIntermediate · 6 lessons · 28 exercises
        5. Event streaming in Kafka

          Inspect topic partitions, keyed events and committed consumer offsets. Explain how a failure or rebalance changes the work left to process.

          1. Logs, topics and replicasBeginner · 4 lessons · 20 exercisesFree
          2. Keys, partitions and offsetsBeginner · 4 lessons · 20 exercises
          3. Consumer positions and groupsBeginner · 4 lessons · 20 exercises

        Projects for this role

        Each project starts from a fictional company's brief and moves between tools the way the job does: what you build in one stage is where the next one starts. Every brief is free to read.

        Wayfriend
        DatabricksdbtSQLDagster
        Project

        From invites to meetups

        Wayfriend wants to know whether friends who plan to meet actually spend time together. Clean the raw events, model meetups, measure success and plan the next day of work.

        5 stages · 3 h 30 minView brief
        Wayfriend
        DatabricksSQL
        Case study

        The reminder question

        Denver tested reminder notifications before planned meetups. Decide whether Wayfriend should send them in every city.

        3 stages · 45 minView brief
        Copperline Freight
        SnowflakedbtTableau
        Project

        Move the on-time report into the warehouse

        Copperline's on-time report is a spreadsheet someone rebuilds every Monday. Rebuild it on the warehouse: one current row per shipment in Snowflake, a tested dbt model, and a Tableau view operations can trust.

        4 stages · 2 h 40 minView brief
        Wayfriend
        DatabricksdbtSQLDagster
        Journey

        Your first 90 days as a data engineer

        You've joined Wayfriend's two-person data team. Build the meetup pipeline, answer the growth team's first question, then keep it running.

        4 chapters · 8 hSee the journey

        The job-readiness assessment

        When you finish the units, test yourself across the tools of the job at once: 12 practical tasks in 50 minutes, each in the tool it is about and checked automatically. The clock keeps running if you close the page.

        ToolTasksMinutes
        SQL311
        Python313
        Snowflake SQL311
        dbt313

        How it is scored

        Each tool
        Ready when you get two thirds of its tasks right or more, Close at one third, otherwise Not yet. A coding task earns the share of its tests that pass.
        Ready for interviews
        75% or more overall, and every tool at Ready.
        Nearly ready
        50% or more overall, and no tool at Not yet.
        Keep practicing
        Anything else. The report lists every task you missed with the lesson that teaches it.

        Open the assessment

        The assessment is free. The result is practice feedback in the Interview Hub's report format, not a hiring decision. Retakes draw different tasks where the question pools allow.

        The interview

        The Interview Hub runs the data engineer interview round by round, at a fictional employer: you answer out loud or in writing, the interviewer follows up, and the report quotes your answers.

        Practice the data engineer interview

        What each plan opens

        Free
        The first unit of every course, with no account: 7 of the 41 units on this pathway. Every project brief is free to read, and the job-readiness assessment is free.
        One Course
        One course in full. On this pathway that is one of these: SQL (10 units), PostgreSQL for engineers (3 units), Python (6 units), Snowflake SQL (5 units), dbt (7 units), Dagster (4 units), Apache Airflow (3 units), Apache Kafka (3 units).
        All Access
        The whole pathway: every unit, every project stage and spoken interview practice.

        See plans and prices

        What a data engineer does

        A data engineer builds and runs the pipelines that bring data from the systems a company uses into a warehouse, cleaned, modeled and on time. Analysts, dashboards and models depend on that data being right. The work is part building, part keeping things running: a source changes a column, a load runs twice, a backfill is needed for last month. Good data engineers make those events boring.

        From SQL to the orchestrator

        SQL comes first and goes deepest, because every later tool runs on it. Python follows for the work SQL can’t do. The warehouse is where both meet. dbt adds structure and tests on top of the warehouse, and the orchestrator runs everything in the right order at the right time.

        Showing what you can do

        The Wayfriend projects are a data team’s first months. You clean raw events, model meetups and measure success in “From invites to meetups”, then answer a product question about reminders from the data you built. Copperline’s on-time report moves from a spreadsheet into Snowflake, a tested dbt model and a view operations can trust. Each gives you a pipeline you can explain from source to dashboard. The assessment then shows which tool to practice before you apply.

        For database reports, practice SQL query optimization: read the plan and preserve the result as you rewrite the query. The SQL cheat sheet covers the query syntax.

        Common questions

        Do I need to be a software engineer first?

        No. Many data engineers start as analysts who wrote more and more SQL, others as software engineers. You need solid SQL, working Python and an understanding of how data moves and breaks. This pathway practices those, step by step.

        Why Snowflake and not BigQuery or Databricks?

        The ideas carry across warehouses: grain, safe changes, incremental loads and cost. If your target jobs name another platform, the BigQuery course and the Databricks course teach the same work in those tools.

        How long does it take?

        The lessons on this pathway add up to about 53 hours. SQL is the longest step on purpose: the interviews test it hardest. We don't promise how long getting hired takes.

        What does it cost?

        The first unit of every course is free, with no account. One Course opens one course in full; All Access opens the whole pathway. The pricing page shows every plan and its price.

        Your Work Gym account

        Save your progress and pick up on another device. Your browser's progress comes with you.

        or

        You can keep practicing without an account.

        Help and feedback

        Ask a question, tell us what is not working, or share an idea. We reply by email.

        What is it about?

        We use it only to reply to you.