Case Studies#

Theory without practice is incomplete. This section examines how six influential distributed systems apply the concepts from this path. Each case study highlights the key architecture decisions, the tradeoffs accepted, and the lessons learned.

Amazon Dynamo#

Paper: “Dynamo: Amazon’s Highly Available Key-value Store” (DeCandia et al., 2007)

Dynamo was built for Amazon’s shopping cart — a system where availability must never be sacrificed, even at the cost of consistency. It pioneered many ideas that later appeared in Cassandra, Riak, and DynamoDB.

Architecture Decisions#

How Writes Work#

Client → any node (coordinator)
    ↓
Coordinator determines preference list: {N1, N2, N3}
    ↓
Write to W nodes (e.g., W=2 of N=3)
    ↓
If a target node is down: write to next node on ring (sloppy quorum)
    Store a hint for the unavailable node (hinted handoff)
    ↓
Return success to client after W acknowledgments

Conflict Resolution: Shopping Cart Example#

User adds item A on Device 1 → version [D1:1]
User adds item B on Device 2 → version [D2:1]

Both writes succeed (different coordinators).
Versions are concurrent: [D1:1] ‖ [D2:1]

Next read returns BOTH versions (siblings).
Application merges: cart = {A, B} (union of both versions)
Writes merged cart back with new version vector.

Lessons#

• Always-writable: Dynamo never rejects a write — this is the core design constraint that drives every other decision
• Push conflict resolution to the client: the application (shopping cart) knows how to merge (union of items); the database cannot make that decision
• Sloppy quorums sacrifice consistency for availability: reads may return stale data, but the system is always available

Google Spanner#

Paper: “Spanner: Google’s Globally-Distributed Database” (Corbett et al., 2012)

Spanner is the opposite of Dynamo — it prioritises consistency over availability and achieves the strongest guarantee possible: external consistency (linearizable + serializable) across the globe.

Architecture Decisions#

Read-Write Transaction Flow#

1. Client starts transaction at coordinator (leader of first shard)
2. Reads acquire read locks on relevant shard leaders
3. Writes are buffered at the client

4. Commit:
   a. If single-shard: Paxos commit (fast path)
   b. If multi-shard: 2PC
      - Coordinator sends Prepare to participant shard leaders
      - Each participant runs Paxos to replicate the Prepare decision
      - All vote YES → Coordinator assigns commit timestamp
      - Commit-wait: wait until TrueTime.now().earliest > commit_ts
      - Send Commit to all participants

5. After commit-wait: transaction is visible to all, globally

Read-Only Transactions (Lock-Free)#

Spanner can serve globally consistent reads without locks by using TrueTime:

1. Client picks a read timestamp t_read = TrueTime.now().latest
2. Each shard serves data as of t_read from its Paxos log
3. No locks needed — snapshot reads from the replicated log
4. Consistent across all shards: same timestamp, same snapshot

Lessons#

• Specialised hardware enables software simplicity: TrueTime (GPS + atomic clocks) reduces clock uncertainty to ~7 ms, making commit-wait practical
• External consistency is achievable at global scale — but requires significant infrastructure investment
• Paxos per shard + 2PC across shards: a layered approach where consensus handles replication and 2PC handles atomicity

Apache Kafka#

Origin: Built at LinkedIn, open-sourced 2011. Now maintained by the Apache Foundation and Confluent.

Kafka is a distributed commit log — it stores ordered, durable streams of records. Its design prioritises throughput, durability, and ordering over low latency.

Architecture Decisions#

In-Sync Replicas (ISR)#

Partition 0: Leader = Broker 1, Replicas = {Broker 1, Broker 2, Broker 3}

ISR = {Broker 1, Broker 2, Broker 3}  ← all caught up

Broker 3 falls behind:
ISR = {Broker 1, Broker 2}            ← Broker 3 removed from ISR

Producer writes with acks=all:
    Wait for ALL ISR members to acknowledge
    → Broker 1 and Broker 2 acknowledge → committed

Broker 3 catches up:
ISR = {Broker 1, Broker 2, Broker 3}  ← Broker 3 rejoined

KRaft: Kafka’s Raft Implementation#

Kafka originally depended on ZooKeeper for metadata management (controller election, partition assignments, topic configuration). KRaft replaces ZooKeeper with a built-in Raft-based metadata quorum.

Before KRaft:
    Kafka brokers ←──── coordination ────► ZooKeeper ensemble
    (separate cluster, additional operational burden)

After KRaft:
    Kafka brokers include controller nodes
    Controller quorum uses Raft for metadata consensus
    No external dependency

Lessons#

• The log is the fundamental abstraction: Kafka’s power comes from treating data as an immutable, append-only, ordered log
• Per-partition ordering is a practical compromise: total ordering across all data is too expensive; per-key ordering within a partition is sufficient for most use cases
• ISR provides flexible durability: acks=all ensures committed records survive leader failure (if ISR has >1 member); acks=1 trades durability for latency

CockroachDB#

Origin: Built by ex-Google engineers, inspired by Spanner. Open-source, designed for global deployment without specialised hardware.

Architecture Decisions#

How CockroachDB Approximates Spanner#

CockroachDB read at timestamp T:
    If data at T has write within the uncertainty interval:
        → Push read timestamp forward OR restart transaction
    If clocks are well-synchronised (NTP ≤ 250ms offset):
        → Restarts are rare in practice

Lessons#

• Spanner’s design is replicable without specialised hardware — HLC + NTP provides a practical approximation
• Raft per range: every shard runs its own Raft group, giving strong consistency with automatic failover
• Parallel commits: reducing 2PC from 2 consensus rounds to 1 in the common case makes distributed transactions practical for OLTP workloads

Apache Cassandra#

Origin: Built at Facebook for inbox search, open-sourced 2008. Combines ideas from Dynamo (partitioning, replication) and Bigtable (data model).

Architecture Decisions#

Tunable Consistency#

CREATE KEYSPACE my_app WITH replication = {
    'class': 'NetworkTopologyStrategy',
    'dc1': 3,
    'dc2': 3
};

-- Per-query consistency:
SELECT * FROM users WHERE id = 42;  -- consistency: ONE (fast, possibly stale)
SELECT * FROM users WHERE id = 42;  -- consistency: QUORUM (majority, consistent)
SELECT * FROM users WHERE id = 42;  -- consistency: LOCAL_QUORUM (majority in local DC)

Lightweight Transactions (LWT)#

For operations requiring linearizability (e.g., conditional inserts), Cassandra uses Paxos:

INSERT INTO users (id, name) VALUES (42, 'Alice')
IF NOT EXISTS;  -- uses Paxos consensus (4 round trips)

LWT is significantly slower than normal writes (4 Paxos round trips vs 1 write). It should be used sparingly.

Lessons#

• Tunable consistency is powerful: letting the application choose per-query avoids one-size-fits-all tradeoffs
• LWW is the default and it loses data: applications must be aware that concurrent writes to the same cell result in silent data loss for the “losing” write
• Phi accrual failure detection adapts to the environment — Cassandra’s detector performs well across diverse network conditions

etcd and Raft#

Origin: Built by CoreOS for Kubernetes, etcd is a distributed key-value store built on Raft. It is the coordination backbone of Kubernetes.

Architecture Decisions#

Linearizable Reads#

etcd offers two read modes:

1. Serializable read (default):
   → Read from any node (may be stale if follower is behind)
   → Fast but not linearizable

2. Linearizable read:
   → Leader confirms it is still the leader (ReadIndex protocol)
   → Ensures the read reflects all committed writes
   → Additional round trip to verify leadership

Watch Mechanism#

etcd’s watch API allows clients to subscribe to changes:

Watch key "/services/web":
    → Event: PUT /services/web/instance-1 = "10.0.1.5:8080"
    → Event: PUT /services/web/instance-2 = "10.0.1.6:8080"
    → Event: DELETE /services/web/instance-1

Implementation: MVCC store keeps revision history
    Watches are served from the revision log
    Clients can resume from a specific revision after disconnect

etcd in Kubernetes#

                    ┌──────────────┐
                    │  API Server   │
                    └──────┬───────┘
                           │ all cluster state
                           ▼
                    ┌──────────────┐
                    │    etcd       │
                    │  (Raft)      │
                    └──────────────┘

Kubernetes stores ALL cluster state in etcd:
    - Pod definitions and status
    - Service endpoints
    - ConfigMaps and Secrets
    - Node registrations
    - Lease objects (node heartbeats)

etcd provides:
    - Linearizable reads/writes for consistency
    - Watch for change notification (controllers)
    - Lease-based TTL for node health

Lessons#

• Small cluster, strong guarantees: etcd does not try to scale to thousands of nodes — it runs 3–5 nodes and provides linearizability, which is exactly what a coordination service needs
• Raft makes consensus practical: etcd’s implementation of Raft is one of the most battle-tested in production, powering every Kubernetes cluster
• Watch + MVCC enables reactive architectures: Kubernetes controllers use etcd watches to react to state changes, forming a reconciliation loop

Cross-System Comparison#

Key Takeaways#

• Dynamo prioritises availability above all else — sloppy quorums, vector clocks, and client-side conflict resolution enable an always-writable system
• Spanner achieves the strongest consistency (external consistency) globally by investing in specialised hardware (TrueTime) and accepting higher write latency (commit-wait)
• Kafka treats the ordered log as the fundamental abstraction — per-partition total ordering with ISR-based replication provides both durability and throughput
• CockroachDB replicates Spanner’s design using commodity hardware — HLC approximates TrueTime, Raft replaces Paxos, and parallel commits optimise 2PC
• Cassandra offers tunable consistency per query — the application decides the tradeoff for each operation, from ONE (fast, weak) to ALL (slow, strong)
• etcd runs a small Raft cluster providing linearizable reads and writes — purpose-built for coordination, not general-purpose storage
• Every system makes explicit tradeoffs between consistency, availability, latency, and throughput — understanding these tradeoffs is the core skill of distributed systems engineering