Data Engineering on Software Engineering Learning Paths

Data Engineering Fundamentals

Mon, 01 Jan 0001 00:00:00 +0000

What Is Data Engineering?#

Data engineering is the discipline of designing, building, and maintaining the systems that collect, store, transform, and serve data. While data scientists build models and analysts extract insights, data engineers build the infrastructure and pipelines that make those activities possible.

The Data Engineer’s Responsibilities#

Responsibility	Description
Ingestion	Collecting data from source systems (APIs, databases, files, streams)
Storage	Choosing and managing data stores (lakes, warehouses, object storage)
Transformation	Cleaning, enriching, and reshaping data for consumption
Orchestration	Scheduling and monitoring pipeline execution
Data quality	Validating, testing, and monitoring data correctness
Infrastructure	Provisioning compute, networking, and storage for data workloads
Security	Managing access control, encryption, and compliance

Data Engineering vs Adjacent Roles#

Role	Focus	Typical Tools
Data Engineer	Pipelines, infrastructure, data platforms	Spark, Airflow, Kafka, SQL, dbt
Data Scientist	Models, experiments, statistical analysis	Python, R, Jupyter, scikit-learn
Data Analyst	Reporting, dashboards, ad-hoc queries	SQL, Tableau, Looker, Excel
Analytics Engineer	Transform + model data for analysts	dbt, SQL, version control
ML Engineer	Productionise ML models	MLflow, Kubeflow, TensorFlow Serving
Platform Engineer	Infrastructure for data teams	Terraform, Kubernetes, CI/CD

The Data Lifecycle#

Data moves through a series of stages from creation to consumption. Understanding this lifecycle is essential for building reliable systems.

Data Modelling for Analytics

Mon, 01 Jan 0001 00:00:00 +0000

Why Analytical Data Modelling?#

Transactional databases (OLTP) are designed for fast reads and writes of individual records. Analytical databases (OLAP) are designed for aggregating large volumes of data across many dimensions. The schemas that serve OLTP well — highly normalised, with many joins — are inefficient for analytical queries. Analytical data modelling restructures data to optimise for read-heavy, aggregation-heavy workloads.

OLTP vs OLAP#

Aspect	OLTP	OLAP
Purpose	Transactional operations	Analytical queries
Query pattern	Point lookups, small updates	Full-table scans, aggregations
Schema	Normalised (3NF+)	Denormalised (star/snowflake)
Users	Applications, microservices	Analysts, BI tools, dashboards
Optimised for	Write throughput, consistency	Read throughput, query speed
Example	Insert an order	Total revenue by region per quarter

Dimensional Modelling#

Dimensional modelling, pioneered by Ralph Kimball, organises data into fact tables (measurements) and dimension tables (context). It is the most widely used approach for analytical data warehouses.

SQL for Data Engineering

Mon, 01 Jan 0001 00:00:00 +0000

SQL as the Data Engineer’s Core Language#

SQL is the lingua franca of data engineering. Every warehouse, most processing engines (Spark SQL, Presto, Trino), and transformation tools (dbt) speak SQL. Mastering advanced SQL patterns is non-negotiable.

Common Table Expressions (CTEs)#

CTEs improve readability by breaking complex queries into named, logical steps. They are defined with the WITH keyword and referenced like tables.

WITH daily_revenue AS (
 SELECT
 date_key,
 SUM(total_amount) AS revenue,
 COUNT(DISTINCT customer_key) AS unique_customers
 FROM fact_sales
 GROUP BY date_key
),
revenue_with_avg AS (
 SELECT
 dr.*,
 AVG(revenue) OVER (ORDER BY date_key ROWS BETWEEN 6 PRECEDING AND CURRENT ROW)
 AS rolling_7d_avg
 FROM daily_revenue dr
)
SELECT
 d.full_date,
 r.revenue,
 r.unique_customers,
 ROUND(r.rolling_7d_avg, 2) AS rolling_7d_avg
FROM revenue_with_avg r
JOIN dim_date d ON r.date_key = d.date_key
WHERE d.calendar_year = 2025
ORDER BY d.full_date;

Recursive CTEs#

Useful for traversing hierarchical data (org charts, category trees).

ETL & ELT Pipelines

Mon, 01 Jan 0001 00:00:00 +0000

ETL vs ELT#

The two dominant pipeline paradigms differ in where transformation happens relative to loading.

ETL (Extract, Transform, Load)#

Data is extracted from sources, transformed in a processing engine (Spark, Python, custom code), and then loaded into the target system.

Source ──▶ Extract ──▶ Transform (Spark/Python) ──▶ Load ──▶ Warehouse

When to use: transformations require compute-heavy logic, data cleansing before landing, legacy systems that can’t handle raw data, or when the warehouse lacks transformation power.

Apache Spark

Mon, 01 Jan 0001 00:00:00 +0000

What Is Apache Spark?#

Apache Spark is a distributed computing engine for large-scale data processing. It runs batch and streaming workloads across clusters of machines, processing data in memory for speed. Spark is the de facto standard for big data processing.

Spark Architecture#

┌─────────────┐
│ Driver │ ← Your application code
│ (SparkContext) │
└──────┬──────┘
 │
 ┌────┴────┐
 │ Cluster │ ← YARN, Kubernetes, Standalone, Mesos
 │ Manager │
 └────┬────┘
 │
┌──────┴──────┬──────────────┐
│ Executor 1 │ Executor 2 │ Executor N │
│ (Tasks) │ (Tasks) │ (Tasks) │
│ [Cache] │ [Cache] │ [Cache] │
└─────────────┴──────────────┘

Driver — the main process that creates the SparkSession, defines transformations, and coordinates execution. Executors — worker processes that execute tasks and cache data in memory. Cluster Manager — allocates resources (YARN, Kubernetes, or Spark Standalone).

Workflow Orchestration

Mon, 01 Jan 0001 00:00:00 +0000

Why Orchestration?#

Data pipelines are not single scripts — they are collections of interdependent tasks that must run in a specific order, on a schedule, with retries, monitoring, and alerting. An orchestrator manages this complexity.

Without orchestration, teams resort to cron jobs, manual triggers, and tribal knowledge. This breaks down quickly: no dependency tracking, no retry logic, no visibility into failures, and no way to backfill historical data.

Apache Airflow#

Airflow is the most widely adopted workflow orchestrator. Originally created at Airbnb, it defines pipelines as Python code (DAGs) and provides a web UI, scheduler, and extensible operator framework.

Streaming & Real-Time Data

Mon, 01 Jan 0001 00:00:00 +0000

Why Streaming?#

Batch processing assumes data can wait. Streaming processes data as it arrives — enabling fraud detection within milliseconds, real-time dashboards, live recommendations, and event-driven architectures. Any use case where latency of minutes or hours is unacceptable requires streaming.

Apache Kafka#

Kafka is the dominant distributed event streaming platform. It acts as a durable, high-throughput, fault-tolerant message broker that decouples producers from consumers.

Core Concepts#

Concept	Description
Topic	A named feed of messages (like a table for events)
Partition	A topic is split into partitions for parallelism and ordering
Offset	A sequential ID for each message within a partition
Producer	Publishes messages to topics
Consumer	Reads messages from topics
Consumer Group	A set of consumers that share the work of reading a topic
Broker	A Kafka server; a cluster has multiple brokers
Replication	Each partition is replicated across brokers for fault tolerance

Kafka Architecture#

Producer ──▶ Topic: orders ──▶ Consumer Group: analytics
 │ ├── Consumer 1 (partition 0, 1)
 │ └── Consumer 2 (partition 2, 3)
 │
 └──▶ Consumer Group: fraud-detection
 └── Consumer 3 (all 4 partitions)

Key insight: each consumer group independently tracks its position (offsets). Multiple groups can read the same topic at different speeds without interfering.

Data Lakes & Lakehouses

Mon, 01 Jan 0001 00:00:00 +0000

Data Lake Architecture#

A data lake stores raw data in object storage (S3, GCS, ADLS) at low cost, in any format. The medallion architecture (bronze / silver / gold) organises data by quality level.

Medallion Architecture#

Sources ──▶ Bronze (Raw) ──▶ Silver (Cleaned) ──▶ Gold (Business-ready)

Layer	Purpose	Data Quality	Format	Consumers
Bronze	Raw ingestion, exact copy of source	As-is (may contain errors)	JSON, CSV, Avro, Parquet	Data engineers
Silver	Cleaned, deduplicated, conformed	Validated, typed, deduplicated	Parquet / Delta / Iceberg	Data engineers, analysts
Gold	Aggregated, business-ready models	Trusted, documented, SLA-backed	Parquet / Delta / Iceberg	Analysts, BI tools, ML

Example Directory Layout#

s3://data-lake/
├── bronze/
│ ├── orders/
│ │ ├── year=2025/month=06/day=15/
│ │ │ └── orders_20250615_143022.parquet
│ │ └── _schema/orders_v1.avsc
│ └── customers/
│ └── full_export_20250615.json
├── silver/
│ ├── orders/
│ │ └── year=2025/month=06/
│ │ └── part-00000.parquet
│ └── customers/
│ └── part-00000.parquet
└── gold/
 ├── fact_orders/
 │ └── year=2025/month=06/
 └── dim_customers/
 └── part-00000.parquet

The Problem with Plain Data Lakes#

Raw files on object storage lack features that databases take for granted:

Data Warehouses

Mon, 01 Jan 0001 00:00:00 +0000

What Is a Data Warehouse?#

A data warehouse is a system optimised for analytical queries — aggregations, joins, and scans over large volumes of data. Unlike transactional databases (OLTP), warehouses are designed for read-heavy, column-oriented workloads where queries touch many rows but few columns.

Key Properties#

Property	Description
Columnar storage	Data stored by column, not by row — reads only needed columns
MPP architecture	Massively Parallel Processing — queries split across many nodes
Compression	Columnar data compresses well (similar values together)
Schema-on-write	Data is structured and typed at load time
Separation of storage and compute	Most cloud warehouses decouple these for elastic scaling

Columnar Storage#

Row-based storage (PostgreSQL, MySQL) stores each row contiguously on disk. Columnar storage stores each column contiguously.

dbt (Data Build Tool)

Mon, 01 Jan 0001 00:00:00 +0000

What Is dbt?#

dbt (data build tool) is a SQL-first transformation framework for the T in ELT. It lets data engineers and analytics engineers write SELECT statements that dbt compiles into DDL/DML and executes against the warehouse. dbt manages dependencies, runs tests, generates documentation, and integrates with CI/CD.

Why dbt Exists#

Before dbt, transformations were scattered across stored procedures, Python scripts, and scheduling tools. dbt brings software engineering practices to SQL transformations: version control, testing, documentation, modularity, and code review.

Data Quality & Governance

Mon, 01 Jan 0001 00:00:00 +0000

Why Data Quality and Governance?#

A pipeline that delivers wrong data on time is worse than one that delivers nothing — consumers make decisions based on the output. Data governance ensures data is correct, discoverable, secure, and compliant. Without it, trust erodes and data becomes a liability instead of an asset.

Data Quality Frameworks#

Great Expectations#

Great Expectations (GX) is a Python framework for defining, running, and documenting data quality checks.

Cloud Data Platforms

Mon, 01 Jan 0001 00:00:00 +0000

Cloud Data Platforms Overview#

Every major cloud provider offers a suite of managed services that together form a complete data platform — ingestion, storage, transformation, orchestration, serving, and governance. Choosing the right combination depends on existing cloud investments, team skills, and specific workload requirements.

AWS Data Stack#

AWS has the broadest set of data services, though the landscape can feel fragmented.

Core Services#

Service	Role	Description
S3	Storage	Object storage — the foundation of every AWS data lake
Glue	ETL + Catalogue	Serverless Spark ETL, schema crawler, Data Catalog
Athena	Query	Serverless SQL over S3 (Presto/Trino engine)
Redshift	Warehouse	MPP columnar warehouse (provisioned or serverless)
Lake Formation	Governance	Fine-grained access control, data sharing, lineage
Kinesis	Streaming	Data Streams (Kafka-like), Firehose (delivery), Analytics (Flink)
EMR	Processing	Managed Spark, Hive, Presto, Flink clusters
Step Functions	Orchestration	Serverless workflow orchestration
MWAA	Orchestration	Managed Apache Airflow
MSK	Streaming	Managed Apache Kafka

AWS Data Lake Architecture#

Sources Ingestion Storage Transform Serve
───────── ───────── ─────── ───────── ─────
RDS (PostgreSQL) ──▶ Glue ETL Job ──▶ S3 (bronze) ──▶ Glue ETL / dbt ──▶ S3 (gold) ──▶ Athena
Kinesis Streams ──▶ Firehose ──────▶ S3 (bronze) └──▶ Redshift
Third-party APIs ──▶ Lambda ───────▶ S3 (bronze) └──▶ QuickSight

 Glue Data Catalog (schema registry, table metadata)
 Lake Formation (access control, permissions)

AWS Glue ETL Job (PySpark)#

import sys
from awsglue.context import GlueContext
from awsglue.job import Job
from awsglue.transforms import *
from pyspark.context import SparkContext

sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(sys.argv[0], {"JOB_NAME": "orders_bronze_to_silver"})

# Read from Glue Data Catalog
orders_dyf = glueContext.create_dynamic_frame.from_catalog(
 database="raw_db",
 table_name="raw_orders"
)

# Transform
orders_df = orders_dyf.toDF()
silver_df = (
 orders_df
 .filter(orders_df.order_id.isNotNull())
 .withColumn("amount", orders_df.amount.cast("decimal(10,2)"))
 .withColumn("order_date", orders_df.order_date.cast("date"))
 .dropDuplicates(["order_id"])
)

# Write to S3 as Parquet, partitioned
silver_df.write \
 .mode("overwrite") \
 .partitionBy("year", "month") \
 .parquet("s3://data-lake/silver/orders/")

job.commit()

Athena Query#

-- Query S3 data via Glue Data Catalog — no infrastructure to manage
SELECT
 order_date,
 COUNT(*) AS order_count,
 SUM(amount) AS total_revenue
FROM silver_db.orders
WHERE year = '2025' AND month = '06'
GROUP BY order_date
ORDER BY order_date;

Athena pricing: $5 per TB scanned. Use Parquet (not CSV) and partition tables to minimise cost.