Database Internals on Software Engineering Learning Paths

Storage Fundamentals

Mon, 01 Jan 0001 00:00:00 +0000

Why Storage Matters#

Every database operation — reads, writes, joins, index lookups — ultimately translates into I/O against a storage device. Understanding how data is physically organised determines whether a query takes microseconds or minutes.

This section covers the lowest layer of a database: how bytes are arranged on disk, how the database divides storage into fixed-size pages, and how individual rows are packed into those pages.

Disk vs SSD#

Mechanical Hard Drives (HDD)#

HDDs store data on spinning magnetic platters read by a moving head. The cost of an I/O operation is dominated by seek time (moving the head to the right track) and rotational latency (waiting for the platter to spin to the right sector).

B-Trees & B+Trees

Mon, 01 Jan 0001 00:00:00 +0000

The Index Problem#

Given a heap file with millions of rows, finding a specific row without an index requires scanning every page — a full table scan. B-trees and B+trees solve this by maintaining a sorted, balanced tree structure that narrows the search to a handful of page reads.

The Databases & SQL path covered when to use indexes. This section covers how they work internally.

B-Tree Fundamentals#

A B-tree (Bayer & McCreight, 1972) is a self-balancing tree where:

LSM Trees & Log-Structured Storage

Mon, 01 Jan 0001 00:00:00 +0000

The Write Optimisation Problem#

B+trees provide excellent read performance, but writes are expensive: inserting a single key may require reading a leaf page, modifying it, and writing it back — all random I/O. For write-heavy workloads (time-series, event logs, messaging), this becomes a bottleneck.

Log-Structured Merge-trees (LSM-trees) flip the trade-off: absorb all writes in memory, then flush sorted runs to disk sequentially. Reads become more expensive, but writes are dramatically cheaper.

Write-Ahead Logging

Mon, 01 Jan 0001 00:00:00 +0000

The Durability Problem#

A database crash can happen at any moment — power failure, kernel panic, hardware fault. If modified pages exist only in the buffer pool (memory), they are lost. Writing every modified page to disk immediately would be safe but catastrophically slow (random I/O for every change).

Write-Ahead Logging (WAL) solves this: before modifying any data page, write a log record describing the change to a sequential log file. If the system crashes, replay the log to reconstruct any modifications that were not yet flushed to the data files.

Buffer Pool Management

Mon, 01 Jan 0001 00:00:00 +0000

The Memory–Disk Gap#

Disk I/O (even on SSDs) is orders of magnitude slower than memory access. The buffer pool bridges this gap by caching frequently accessed disk pages in memory. A well-tuned buffer pool means most reads never touch disk.

Access latency comparison:
 L1 cache: ~1 ns
 DRAM: ~100 ns
 SSD random: ~50,000 ns (500× slower than RAM)
 HDD random: ~10,000,000 ns (100,000× slower than RAM)

The buffer pool is the single most important performance structure in a database — it determines the cache hit ratio, which directly controls query latency.

MVCC Implementation

Mon, 01 Jan 0001 00:00:00 +0000

Why MVCC?#

Without concurrency control, simultaneous readers and writers would see inconsistent data — partial updates, phantom rows, or torn reads. Lock-based approaches (shared/exclusive locks on every row) are correct but cause readers to block writers and vice versa.

Multi-Version Concurrency Control (MVCC) solves this by keeping multiple versions of each row. Readers see a consistent snapshot without acquiring locks; writers create new versions instead of overwriting in place. The result: readers never block writers, and writers never block readers.

Query Processing

Mon, 01 Jan 0001 00:00:00 +0000

From SQL Text to Result Set#

A SQL query goes through several transformation stages before producing results. Understanding this pipeline explains why some queries are fast and others are slow — regardless of how the SQL is written.

┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
│ Parser │ → │ Analyser │ → │ Rewriter │ → │ Planner/ │ → │ Executor │
│ │ │ │ │ │ │Optimiser │ │ │
│ SQL text │ │ Semantic │ │ View │ │ Cost- │ │ Runs the │
│ → parse │ │ checks, │ │ expansion│ │ based │ │ chosen │
│ tree │ │ resolve │ │ rule │ │ plan │ │ plan │
│ │ │ names │ │ rewrites │ │ selection│ │ │
└──────────┘ └──────────┘ └──────────┘ └──────────┘ └──────────┘

Parsing#

The parser converts SQL text into an abstract syntax tree (AST), also called a parse tree.

Lock Management

Mon, 01 Jan 0001 00:00:00 +0000

Why Locks Still Matter with MVCC#

MVCC handles most read/write concurrency without locks, but locks are still essential for:

Write–write conflicts — two transactions updating the same row need coordination
DDL operations — schema changes must not conflict with concurrent queries
Explicit locking — SELECT ... FOR UPDATE, advisory locks
Internal structures — B+tree page latches, buffer pool frame pins

This section covers the lock manager: the subsystem that grants, queues, detects deadlocks, and escalates locks.

Index Internals

Mon, 01 Jan 0001 00:00:00 +0000

Beyond B+Trees#

Section 2 covered B+trees — the most common index type. But databases support many specialised index structures for different access patterns. This section explores the internal mechanics of hash indexes, GiST, GIN, BRIN, bitmap indexes, and key optimisations like index-only scans and HOT updates.

The Databases & SQL path covered when to create indexes. This section covers how they work under the hood.

Hash Indexes#

Hash indexes map keys to bucket numbers using a hash function. Each bucket is a page (or chain of pages) containing entries with the same hash.

Storage Engine Architecture

Mon, 01 Jan 0001 00:00:00 +0000

What Is a Storage Engine?#

A storage engine is the component responsible for how data is physically stored, retrieved, and modified. It sits below the SQL layer and above the operating system, composing several subsystems covered in earlier sections.

┌─────────────────────────────────────────────┐
│ SQL Layer │
│ (Parser → Planner → Optimiser → Executor) │
├─────────────────────────────────────────────┤
│ Storage Engine │
│ ┌──────────┐ ┌──────────┐ ┌─────────────┐ │
│ │ Access │ │ Buffer │ │ Transaction │ │
│ │ Methods │ │ Pool │ │ Manager │ │
│ │ (B+tree, │ │ │ │ (WAL, MVCC, │ │
│ │ heap, │ │ │ │ locks) │ │
│ │ LSM) │ │ │ │ │ │
│ └──────────┘ └──────────┘ └─────────────┘ │
├─────────────────────────────────────────────┤
│ File System / Direct I/O │
├─────────────────────────────────────────────┤
│ Storage Device │
└─────────────────────────────────────────────┘

Engine Composition#

Every storage engine combines the same fundamental subsystems — the differences lie in the choices each subsystem makes.