Databases & SQL on Software Engineering Learning Paths

Relational Database Fundamentals

Mon, 01 Jan 0001 00:00:00 +0000

What Is a Database?#

A database is an organized collection of structured data, managed by a Database Management System (DBMS) that handles storage, retrieval, security, and concurrent access. Without a DBMS, applications would need to manage files, locking, indexing, and crash recovery themselves.

Types of Databases#

Type	Model	Examples	Best For
Relational (RDBMS)	Tables with rows/columns	PostgreSQL, MySQL, Oracle, SQL Server	Structured data, complex queries, transactions
Document	JSON/BSON documents	MongoDB, CouchDB, Firestore	Flexible schemas, nested data
Key-Value	Simple key → value pairs	Redis, DynamoDB, Memcached	Caching, sessions, simple lookups
Column-Family	Wide columns, sparse rows	Cassandra, HBase, ScyllaDB	Time-series, high write throughput
Graph	Nodes and edges	Neo4j, Amazon Neptune	Relationships, social networks, recommendations
Time-Series	Timestamped data points	InfluxDB, TimescaleDB, Prometheus	Metrics, IoT, monitoring

The Relational Model#

Proposed by Edgar F. Codd in 1970, the relational model organizes data into relations (tables) with mathematical foundations in set theory and predicate logic.

SQL Basics

Mon, 01 Jan 0001 00:00:00 +0000

What Is SQL?#

SQL (Structured Query Language) is the standard language for interacting with relational databases. Despite variations between vendors, core SQL is remarkably portable. It’s declarative — you describe what you want, not how to get it.

SQL Statement Categories#

Category	Acronym	Statements	Purpose
Data Query Language	DQL	`SELECT`	Read data
Data Manipulation Language	DML	`INSERT`, `UPDATE`, `DELETE`	Modify data
Data Definition Language	DDL	`CREATE`, `ALTER`, `DROP`, `TRUNCATE`	Define structure
Data Control Language	DCL	`GRANT`, `REVOKE`	Manage permissions
Transaction Control	TCL	`BEGIN`, `COMMIT`, `ROLLBACK`	Manage transactions

SELECT — Reading Data#

Basic Syntax#

SELECT columns
FROM table
WHERE condition
ORDER BY column [ASC|DESC]
LIMIT count;

Selecting Columns#

-- All columns
SELECT * FROM employees;

-- Specific columns
SELECT name, salary, department FROM employees;

-- With aliases
SELECT name AS employee_name, salary AS annual_salary FROM employees;

-- Computed columns
SELECT name, salary, salary * 12 AS annual_total FROM employees;

-- Distinct values (remove duplicates)
SELECT DISTINCT department FROM employees;

Filtering with WHERE#

-- Comparison operators
SELECT * FROM employees WHERE salary > 80000;
SELECT * FROM employees WHERE department = 'Engineering';
SELECT * FROM employees WHERE hire_date >= '2021-01-01';

-- Multiple conditions
SELECT * FROM employees WHERE department = 'Engineering' AND salary > 90000;
SELECT * FROM employees WHERE department = 'Sales' OR department = 'Marketing';
SELECT * FROM employees WHERE NOT department = 'Sales';

-- IN (shorthand for multiple OR)
SELECT * FROM employees WHERE department IN ('Engineering', 'Marketing', 'Sales');

-- BETWEEN (inclusive range)
SELECT * FROM employees WHERE salary BETWEEN 70000 AND 100000;

-- LIKE (pattern matching)
SELECT * FROM employees WHERE name LIKE 'A%'; -- Starts with A
SELECT * FROM employees WHERE name LIKE '%son'; -- Ends with son
SELECT * FROM employees WHERE name LIKE '%ar%'; -- Contains ar
SELECT * FROM employees WHERE name LIKE '_ob%'; -- Second and third chars are 'ob'

-- NULL checks (never use = NULL)
SELECT * FROM employees WHERE manager_id IS NULL;
SELECT * FROM employees WHERE manager_id IS NOT NULL;

Sorting with ORDER BY#

-- Ascending (default)
SELECT * FROM employees ORDER BY salary;

-- Descending
SELECT * FROM employees ORDER BY salary DESC;

-- Multiple columns
SELECT * FROM employees ORDER BY department ASC, salary DESC;

-- By column position (less readable)
SELECT name, salary FROM employees ORDER BY 2 DESC;

Limiting Results#

-- First N rows
SELECT * FROM employees ORDER BY salary DESC LIMIT 10;

-- Pagination (skip 20, take 10)
SELECT * FROM employees ORDER BY id LIMIT 10 OFFSET 20;

-- SQL Server uses TOP
SELECT TOP 10 * FROM employees ORDER BY salary DESC;

INSERT — Adding Data#

-- Single row (specify columns)
INSERT INTO employees (name, department, salary, hire_date)
VALUES ('Eve Brown', 'Engineering', 92000, '2023-06-01');

-- Single row (all columns in order — fragile, avoid)
INSERT INTO employees
VALUES (5, 'Eve Brown', 'Engineering', 92000, '2023-06-01');

-- Multiple rows
INSERT INTO employees (name, department, salary, hire_date)
VALUES
 ('Frank Lee', 'Marketing', 75000, '2023-07-15'),
 ('Grace Kim', 'Sales', 71000, '2023-08-01'),
 ('Henry Park', 'Engineering', 98000, '2023-09-10');

-- Insert from SELECT
INSERT INTO archive_employees (name, department, salary)
SELECT name, department, salary
FROM employees
WHERE hire_date < '2020-01-01';

UPDATE — Modifying Data#

-- Update specific rows (always include WHERE!)
UPDATE employees
SET salary = 100000
WHERE id = 3;

-- Update multiple columns
UPDATE employees
SET salary = 95000, department = 'Engineering'
WHERE name = 'Bob Smith';

-- Computed update
UPDATE employees
SET salary = salary * 1.10
WHERE department = 'Engineering';

-- Update from another table
UPDATE employees e
SET department = d.new_department
FROM department_changes d
WHERE e.id = d.employee_id;

⚠️ WARNING: An UPDATE without WHERE modifies ALL rows in the table. Always double-check before executing.

Joins and Subqueries

Mon, 01 Jan 0001 00:00:00 +0000

Why Joins?#

In a normalized database, related data lives in separate tables. Joins combine rows from two or more tables based on a related column, letting you reconstruct the full picture from properly separated data.

Join Types#

Visual Overview#

Assume two tables:

employees (id, name, department_id)
departments (id, name)

flowchart LR
 subgraph "INNER JOIN"
 A1["Only matching<br/>rows from both"]
 end
 subgraph "LEFT JOIN"
 A2["All from left +<br/>matching from right"]
 end
 subgraph "RIGHT JOIN"
 A3["All from right +<br/>matching from left"]
 end
 subgraph "FULL JOIN"
 A4["All from both,<br/>NULLs where no match"]
 end

INNER JOIN#

Returns only rows where there’s a match in both tables:

Aggregations and Grouping

Mon, 01 Jan 0001 00:00:00 +0000

Aggregate Functions#

Aggregate functions compute a single value from a set of rows:

Function	Purpose	Example
`COUNT(*)`	Number of rows	Total employees
`COUNT(column)`	Non-NULL values in column	Employees with a phone
`COUNT(DISTINCT col)`	Unique non-NULL values	Number of departments
`SUM(column)`	Total	Total payroll
`AVG(column)`	Average	Average salary
`MIN(column)`	Smallest	Lowest salary
`MAX(column)`	Largest	Highest salary

SELECT
 COUNT(*) AS total_employees,
 COUNT(DISTINCT department_id) AS departments,
 SUM(salary) AS total_payroll,
 AVG(salary) AS avg_salary,
 MIN(salary) AS min_salary,
 MAX(salary) AS max_salary
FROM employees;

GROUP BY#

Groups rows sharing a value, then applies aggregate functions per group:

Data Definition and Constraints

Mon, 01 Jan 0001 00:00:00 +0000

Creating Tables#

CREATE TABLE employees (
 id SERIAL PRIMARY KEY, -- Auto-increment PK (PostgreSQL)
 name VARCHAR(100) NOT NULL,
 email VARCHAR(255) UNIQUE NOT NULL,
 department_id INTEGER REFERENCES departments(id),
 salary NUMERIC(10, 2) DEFAULT 0.00,
 level VARCHAR(20) CHECK (level IN ('junior', 'mid', 'senior', 'lead')),
 is_active BOOLEAN DEFAULT TRUE,
 hire_date DATE NOT NULL DEFAULT CURRENT_DATE,
 created_at TIMESTAMP DEFAULT NOW()
);

-- MySQL equivalent
CREATE TABLE employees (
 id INT AUTO_INCREMENT PRIMARY KEY,
 name VARCHAR(100) NOT NULL,
 email VARCHAR(255) UNIQUE NOT NULL,
 department_id INT,
 salary DECIMAL(10, 2) DEFAULT 0.00,
 level ENUM('junior', 'mid', 'senior', 'lead'),
 is_active BOOLEAN DEFAULT TRUE,
 hire_date DATE NOT NULL DEFAULT (CURRENT_DATE),
 created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
 FOREIGN KEY (department_id) REFERENCES departments(id)
);

Constraints#

Constraints enforce data integrity rules at the database level — data that violates them is rejected.

Normalization and Schema Design

Mon, 01 Jan 0001 00:00:00 +0000

What Is Normalization?#

Normalization is the process of organizing database tables to reduce redundancy and prevent anomalies (insertion, update, and deletion anomalies). Each normal form builds on the previous one.

Why Normalize?#

Problem	Without Normalization	With Normalization
Update anomaly	Change department name in 500 employee rows	Change it in 1 row in departments table
Insertion anomaly	Can’t add a department with no employees	Department exists independently
Deletion anomaly	Deleting last employee removes department info	Department persists regardless
Storage	Repeated data wastes space	Each fact stored once
Consistency	Same data might disagree across rows	Single source of truth

Normal Forms#

First Normal Form (1NF)#

Rule: Every column contains only atomic (indivisible) values. No repeating groups or arrays.

Indexing and Query Optimization

Mon, 01 Jan 0001 00:00:00 +0000

Why Indexes Matter#

Without an index, the database must scan every row in a table to find matches (full table scan). An index is a separate data structure that allows the database to find rows without reading the entire table — like a book’s index vs reading every page.

Without Index	With Index
O(n) — scan all rows	O(log n) — tree traversal
1M row table → 1M reads	1M row table → ~20 reads
Fine for small tables	Essential for large tables

How B-Tree Indexes Work#

The default index type in all major databases is the B-tree (balanced tree):

Transactions and Concurrency

Mon, 01 Jan 0001 00:00:00 +0000

What Is a Transaction?#

A transaction is a group of SQL statements that execute as a single unit — either all succeed or all fail. Transactions protect data integrity when multiple operations must happen together.

BEGIN;
 UPDATE accounts SET balance = balance - 500 WHERE id = 1; -- Debit sender
 UPDATE accounts SET balance = balance + 500 WHERE id = 2; -- Credit receiver
COMMIT;
-- Both happen, or neither happens

If anything fails between BEGIN and COMMIT:

NoSQL Concepts

Mon, 01 Jan 0001 00:00:00 +0000

What Is NoSQL?#

NoSQL (“Not Only SQL”) databases are non-relational systems designed for specific data models, access patterns, and scale requirements that relational databases handle poorly or inefficiently.

NoSQL isn’t better or worse than SQL — it’s different tradeoffs for different problems.

When to Use NoSQL vs SQL#

Choose SQL (Relational) When	Choose NoSQL When
Data has clear relationships and schema	Schema varies between records
Complex queries with JOINs needed	Access patterns are simple (get by key)
ACID transactions are critical	Eventual consistency is acceptable
Data integrity is paramount	Horizontal scalability is priority
You need ad-hoc analytical queries	You need low-latency at massive scale
Data fits neatly in tables	Data is hierarchical, graph-shaped, or time-series

NoSQL Categories#

Document Stores#

Store data as flexible JSON/BSON documents. Each document can have different fields.

Database Administration

Mon, 01 Jan 0001 00:00:00 +0000

Backups#

Backup Types#

Type	What It Captures	Speed	Restore Time	Use
Logical (pg_dump)	SQL statements to recreate data	Slow (reads all data)	Slow (re-executes SQL)	Small databases, cross-version migration
Physical (pg_basebackup)	Raw data files	Fast (file copy)	Fast (file restore)	Large databases, point-in-time recovery
Continuous (WAL archiving)	Write-Ahead Log stream	Minimal overhead	Fast + PITR	Production (restore to any point in time)

PostgreSQL Backup Commands#

# Logical backup (single database)
pg_dump -h localhost -U postgres mydb > backup.sql
pg_dump -Fc mydb > backup.dump # Custom format (compressed, parallel restore)
pg_dump -Fd mydb -j 4 -f backup_dir/ # Directory format (parallel dump)

# All databases
pg_dumpall > all_databases.sql

# Restore
psql mydb < backup.sql # From SQL
pg_restore -d mydb backup.dump # From custom format
pg_restore -d mydb -j 4 backup_dir/ # Parallel restore

# Physical backup
pg_basebackup -h localhost -U replicator -D /backups/base -Ft -z -P

Backup Best Practices#

Automate backups on a schedule (daily full + continuous WAL)
Test restores regularly — a backup you can’t restore is worthless
Store off-site — different region, different provider
Encrypt backups at rest and in transit
Monitor backup jobs — alert on failure immediately
Define retention — 7 daily, 4 weekly, 12 monthly is a common pattern

Replication#

Purpose#

Goal	Solution
High availability (failover)	Synchronous replica + automatic failover
Read scaling	Async read replicas
Disaster recovery	Cross-region replica
Zero-downtime upgrades	Logical replication to new version

PostgreSQL Streaming Replication#

flowchart LR
 APP["Application"]
 PRIMARY["Primary<br/>(reads + writes)"]
 REPLICA1["Replica 1<br/>(reads only)"]
 REPLICA2["Replica 2<br/>(reads only)"]
 
 APP -->|"writes"| PRIMARY
 APP -->|"reads"| REPLICA1
 APP -->|"reads"| REPLICA2
 PRIMARY -->|"WAL stream"| REPLICA1
 PRIMARY -->|"WAL stream"| REPLICA2

Replication Modes#

Mode	Behavior	Latency	Data Loss Risk
Asynchronous	Primary doesn’t wait for replica confirmation	Lowest	Possible (seconds of data)
Synchronous	Primary waits for at least one replica	Higher	Zero
Logical	Replicates at SQL level (selective tables)	Variable	Possible

Partitioning#

Split large tables into smaller physical pieces for performance:

Practical Patterns

Mon, 01 Jan 0001 00:00:00 +0000

Practical Patterns#

Database theory gets you through schema design and query writing. This section covers the patterns you encounter when databases meet production code — migrations, pagination, connection pooling, multi-tenancy, and the N+1 query problem.

Schema Migrations#

The Problem#

Database schemas evolve. Adding columns, changing types, creating indexes — these changes must be applied consistently across dev, staging, and production.

Migration Tools#

Tool	Language	Approach
Flyway	Java, CLI	Versioned SQL files (`V1__create_users.sql`)
Liquibase	Java, CLI	XML/YAML/SQL changesets
Alembic	Python	Autogenerated from SQLAlchemy models
golang-migrate	Go	Numbered SQL files
Prisma Migrate	TypeScript	Schema-first, auto-generated SQL
Rails Migrations	Ruby	Ruby DSL for schema changes
dbmate	Any	Plain SQL files, lightweight CLI

Migration Best Practices#

Practice	Why
Every change is a migration file	Reproducible, version-controlled
Migrations are forward-only	Never edit a migration that has been applied
Test migrations on a copy of production data	Catch data-dependent failures
Separate schema changes from data changes	Schema DDL is fast; data migration can be slow
Make changes backwards-compatible	Deploy code that works with both old and new schema

Zero-Downtime Migration Pattern#

1. Add new column (nullable) ← migration 1
2. Deploy code that writes to BOTH old and new columns
3. Backfill new column from old column ← migration 2
4. Deploy code that reads from new column only
5. Drop old column ← migration 3

Never rename or remove a column in a single deployment — the old code still running during rollout will break.