Data Engineering#
A comprehensive path through data engineering — from foundational concepts and data modelling to building production pipelines, orchestrating workflows, and operating cloud-native data platforms at scale.
Sections#
| # | Section | Topics |
|---|---|---|
| 1 | Data Engineering Fundamentals | Role of a data engineer, data lifecycle, batch vs streaming, data quality dimensions |
| 2 | Data Modelling for Analytics | Star schema, snowflake schema, slowly changing dimensions, Kimball vs Inmon |
| 3 | SQL for Data Engineering | Window functions, CTEs, MERGE/UPSERT, pivoting, advanced aggregations, performance tuning |
| 4 | ETL & ELT Pipelines | ETL vs ELT, pipeline design patterns, idempotency, data validation, scheduling |
| 5 | Apache Spark | RDDs, DataFrames, Spark SQL, PySpark, partitioning, caching, Spark on EMR/Databricks |
| 6 | Workflow Orchestration | Apache Airflow, Prefect, Dagster, DAGs, operators, sensors, scheduling, backfill |
| 7 | Streaming & Real-Time Data | Apache Kafka, Kafka Streams, Flink, Kinesis, event-time vs processing-time |
| 8 | Data Lakes & Lakehouses | Bronze/silver/gold, Delta Lake, Apache Iceberg, Hudi, schema evolution, time travel |
| 9 | Data Warehouses | BigQuery, Redshift, Snowflake, ClickHouse, columnar storage, MPP, cost models |
| 10 | dbt (Data Build Tool) | Models, sources, seeds, tests, Jinja templating, incremental models, CI/CD for dbt |
| 11 | Data Quality & Governance | Great Expectations, Soda, data contracts, schema registries, lineage, catalogues |
| 12 | Cloud Data Platforms | AWS data stack, GCP BigQuery/Dataflow, Azure Synapse, choosing a platform |
Prerequisites#
- Solid SQL knowledge (joins, aggregations, subqueries)
- Basic Python programming
- Familiarity with command-line tools and version control
- Understanding of databases (relational concepts, basic NoSQL awareness)
What You’ll Be Able To Do#
- Design and build production-grade data pipelines (batch and streaming)
- Model data for analytical workloads using dimensional modelling techniques
- Orchestrate complex workflows with Airflow, Prefect, or Dagster
- Process large-scale data with Apache Spark and streaming frameworks
- Operate data lakes, lakehouses, and cloud data warehouses
- Implement data quality checks, governance policies, and data contracts
- Choose and integrate cloud data platform services for end-to-end architectures