System Design on Software Engineering Learning Paths

Fundamentals

Mon, 01 Jan 0001 00:00:00 +0000

Why System Design?#

Writing code that works on your laptop is one skill. Designing systems that serve millions of users reliably is another. System design is about making tradeoffs — every choice has consequences for scalability, reliability, cost, and complexity.

Core Metrics#

Latency#

Time from request to response. Measured in milliseconds (ms).

Operation	Typical Latency
L1 cache reference	0.5 ns
RAM access	100 ns
SSD random read	150 μs
HDD seek	10 ms
Same-datacenter roundtrip	0.5 ms
US coast-to-coast roundtrip	40 ms
Europe to US roundtrip	80 ms

P50 vs P95 vs P99: Don’t measure averages — measure percentiles. P99 latency (worst 1% of requests) is what users feel during peak load.

Networking & Protocols

Mon, 01 Jan 0001 00:00:00 +0000

Networking & Protocols#

Understanding how data moves across the network is the foundation of system design. Every architectural decision — from API style to database placement — is constrained by networking realities.

DNS Resolution#

The Domain Name System translates human-readable domain names into IP addresses. A single page load may trigger dozens of DNS lookups.

Resolution Flow#

sequenceDiagram
 participant Client
 participant Resolver as Recursive Resolver
 participant Root as Root NS
 participant TLD as TLD NS (.com)
 participant Auth as Authoritative NS

 Client->>Resolver: Who is api.example.com?
 Resolver->>Root: Who handles .com?
 Root-->>Resolver: Try TLD NS at 192.5.6.30
 Resolver->>TLD: Who handles example.com?
 TLD-->>Resolver: Try Auth NS at 205.251.195.35
 Resolver->>Auth: What is api.example.com?
 Auth-->>Resolver: A record: 93.184.216.34 (TTL 300s)
 Resolver-->>Client: 93.184.216.34

DNS Record Types#

Record	Purpose	Example
A	Maps name to IPv4 address	`api.example.com → 93.184.216.34`
AAAA	Maps name to IPv6 address	`api.example.com → 2606:2800:220:1:...`
CNAME	Alias to another domain name	`www.example.com → example.com`
MX	Mail server routing	`example.com → mail.example.com`
NS	Delegates a zone to nameservers	`example.com → ns1.provider.com`
SRV	Service discovery (host + port)	`_http._tcp.example.com → 8080 web1`
TXT	Arbitrary text (SPF, verification)	`v=spf1 include:_spf.google.com`

DNS Caching Layers#

Browser cache — respects TTL from response
OS resolver cache — shared across all applications
Router/ISP cache — intermediate caching resolver
Recursive resolver — (e.g., 8.8.8.8, 1.1.1.1) caches per TTL

Design implication: Low TTLs (30–60s) enable fast failover but increase lookup latency. High TTLs (3600s+) improve performance but slow propagation of changes.

Load Balancing

Mon, 01 Jan 0001 00:00:00 +0000

Load Balancing#

Load balancers distribute incoming traffic across multiple servers to improve availability, reliability, and performance. They are one of the most fundamental building blocks in scalable system design.

Why Load Balancers?#

Problem	How Load Balancing Solves It
Single point of failure	Traffic reroutes to healthy servers if one dies
Capacity limits	Spread load across multiple machines
Uneven load distribution	Algorithms ensure balanced utilization
Maintenance downtime	Drain connections before taking servers offline
Geographic latency	Route to the nearest healthy endpoint

Layer 4 vs Layer 7 Load Balancing#

graph TD
 subgraph "OSI Model"
 L7[Layer 7 — Application: HTTP, gRPC, WebSocket]
 L4[Layer 4 — Transport: TCP, UDP]
 L3[Layer 3 — Network: IP]
 end

 subgraph "L4 Load Balancer"
 L4LB[Routes based on IP + Port<br/>No payload inspection]
 end

 subgraph "L7 Load Balancer"
 L7LB[Routes based on headers, URL path,<br/>cookies, content type]
 end

Comparison Table#

Feature	L4 Load Balancer	L7 Load Balancer
Decision basis	IP address + TCP/UDP port	HTTP headers, URL, cookies, payload
Performance	Faster (no payload parsing)	Slightly slower (must parse application data)
SSL termination	Typically no (pass-through)	Yes (terminates TLS, re-encrypts optionally)
Content routing	No	Yes (path-based, host-based routing)
WebSocket support	Pass-through (no awareness)	Upgrade-aware, can route by path
Health checks	TCP connect / port check	HTTP status codes, response body checks
Connection pooling	No	Yes (reduces backend connections)
Sticky sessions	IP-based only	Cookie-based, header-based
Use case	High-throughput TCP (databases, gRPC)	Web applications, APIs, microservices
AWS equivalent	Network Load Balancer (NLB)	Application Load Balancer (ALB)

Load Balancing Algorithms#

Round Robin#

Distributes requests sequentially across all servers.

Caching

Mon, 01 Jan 0001 00:00:00 +0000

Caching#

Caching stores copies of data in a faster storage layer so future requests can be served without hitting the slower origin. It is the single most impactful technique for reducing latency and database load in distributed systems.

The Cache Hierarchy#

Every request traverses multiple caching layers. Each layer trades freshness for speed.

flowchart TD
 Client[Client Request] --> Browser[Browser Cache]
 Browser -->|Miss| CDN[CDN Edge Cache]
 CDN -->|Miss| LB[Load Balancer]
 LB --> AppCache[Application Cache<br/>Redis / Memcached]
 AppCache -->|Miss| DB[Database Query Cache]
 DB -->|Miss| Disk[Disk / Storage]

 style Browser fill:#e8f5e1
 style CDN fill:#e8f5e1
 style AppCache fill:#e3f2fd
 style DB fill:#fff3e0

Layer	Latency	Scope	Controlled By
CPU cache (L1/L2/L3)	1–10 ns	Single process	Hardware
In-process cache	10–100 ns	Single instance	Application code
Browser cache	0 ms (local)	Single user	Cache-Control headers
CDN	5–50 ms	Geographic region	CDN config + headers
Application cache (Redis)	1–5 ms	Cluster-wide	Application code
Database query cache	1–10 ms	Database cluster	DB configuration
Database buffer pool	0.1–1 ms	Database instance	DB engine

Caching Strategies#

Cache-Aside (Lazy Loading)#

The application manages the cache explicitly.

Databases at Scale

Mon, 01 Jan 0001 00:00:00 +0000

Databases at Scale#

A single database server eventually becomes a bottleneck — whether for read throughput, write throughput, storage capacity, or availability. Scaling databases requires understanding replication, sharding, and the trade-offs each introduces.

Replication#

Replication maintains copies of data across multiple database nodes for availability, durability, and read scaling.

Synchronous vs Asynchronous Replication#

sequenceDiagram
 participant Client
 participant Primary
 participant Replica1
 participant Replica2

 Note over Client,Replica2: Synchronous Replication
 Client->>Primary: INSERT INTO orders...
 Primary->>Replica1: Replicate (wait for ACK)
 Replica1-->>Primary: ACK
 Primary-->>Client: Commit confirmed

 Note over Client,Replica2: Asynchronous Replication
 Client->>Primary: INSERT INTO orders...
 Primary-->>Client: Commit confirmed (immediate)
 Primary->>Replica2: Replicate (eventual)

Aspect	Synchronous	Asynchronous
Write latency	Higher (wait for replica ACK)	Lower (immediate commit)
Data safety	Zero data loss on primary failure	Potential data loss (replication lag)
Availability	Slower (blocked if replica is down)	Higher (primary independent)
Consistency	Strong (replicas always current)	Eventual (replicas may lag)
Throughput	Lower (bounded by slowest replica)	Higher (primary writes freely)
Use case	Financial transactions, inventory	Analytics, read-heavy workloads

Semi-Synchronous#

A hybrid: the primary waits for at least one replica to acknowledge before confirming to the client. Other replicas receive updates asynchronously.

Message Queues & Event-Driven Architecture

Mon, 01 Jan 0001 00:00:00 +0000

Message Queues & Event-Driven Architecture#

Asynchronous messaging is the backbone of scalable distributed systems. Instead of services waiting for each other synchronously, message queues decouple producers from consumers, enabling independent scaling, fault tolerance, and temporal decoupling.

Why Asynchronous Processing?#

Synchronous communication (request-response) creates tight coupling between services. If the downstream service is slow or down, the upstream service is directly impacted.

Concern	Synchronous	Asynchronous
Coupling	Tight — caller waits for response	Loose — producer doesn’t know consumer
Latency	Cumulative across call chain	Producer returns immediately
Failure handling	Cascading failures	Failures isolated to consumer
Scaling	Both sides must scale together	Producer and consumer scale independently
Ordering	Implicit (request-response)	Must be explicitly managed
Complexity	Simpler mental model	Requires queue infrastructure

When to Use Async#

Task takes > 500ms — don’t make the user wait for email sends, PDF generation, or analytics processing
Spiky traffic — buffer requests during peaks and process at a steady rate
Cross-service coordination — notify multiple services without coupling them
Retry-heavy operations — let the queue handle retries rather than the caller

Message Queue vs Event Stream#

These are fundamentally different communication models despite superficial similarities.

Microservices Architecture

Mon, 01 Jan 0001 00:00:00 +0000

Microservices Architecture#

Microservices decompose a system into small, independently deployable services, each owning a specific business capability. This architectural style enables teams to move fast and scale independently — but introduces significant distributed systems complexity.

Monolith vs Microservices#

Aspect	Monolith	Microservices
Deployment	Single deployable unit	Each service deployed independently
Scaling	Scale entire application	Scale individual services
Technology	Single tech stack	Polyglot (per-service choice)
Data	Shared database	Database per service
Team structure	One team or many teams on one codebase	Small teams per service (2-pizza rule)
Complexity	In-process calls, simple	Network calls, distributed complexity
Consistency	Strong (ACID transactions)	Eventual consistency (sagas)
Testing	End-to-end straightforward	Integration testing complex
Latency	In-process (nanoseconds)	Network (milliseconds)
Debugging	Single process, stack traces	Distributed tracing required
Initial velocity	Faster to start	Slower to start (infrastructure overhead)
Long-term velocity	Slows as codebase grows	Maintains speed per service

The Monolith Is Not the Enemy#

A well-structured monolith (modular monolith) is a valid architecture. Many successful systems start as monoliths and only decompose when specific scaling or team boundaries demand it.

Storage & Content Delivery Networks

Mon, 01 Jan 0001 00:00:00 +0000

Storage & Content Delivery Networks#

Choosing the right storage type and delivery mechanism directly impacts performance, cost, and reliability. This section covers cloud storage models, CDN architecture, and patterns for serving content at global scale.

Storage Types Compared#

Property	Object Storage (S3)	Block Storage (EBS)	File Storage (EFS)
Abstraction	Flat key-value (objects)	Raw disk blocks	Hierarchical file system
Access pattern	HTTP API (GET/PUT)	Attached to single instance	Mounted by multiple instances (NFS)
Scalability	Virtually unlimited	Fixed size (resizable)	Grows automatically
Latency	Higher (network, HTTP)	Very low (attached disk)	Low-moderate (network)
Durability	99.999999999% (11 nines)	99.999% (within AZ)	99.999999999% (multi-AZ)
Use case	Static assets, backups, data lakes	OS disks, databases	Shared config, CMS content
Cost model	Per GB stored + requests	Per GB provisioned	Per GB used
Concurrency	Unlimited concurrent readers	Single writer	Multiple readers/writers

When to Use Each#

graph TD
 Q1{Need shared filesystem?} -->|Yes| EFS[EFS / File Storage]
 Q1 -->|No| Q2{Need low-latency block device?}
 Q2 -->|Yes| EBS[EBS / Block Storage]
 Q2 -->|No| S3[S3 / Object Storage]

Object Storage (S3) Deep Dive#

Key Concepts#

Bucket — container for objects (globally unique name)
Object — file + metadata, identified by a key (path-like string)
Prefix — simulated folder structure (images/2024/photo.jpg)
Versioning — keep multiple versions of every object
Lifecycle policies — automatically transition or delete objects based on age

Storage Classes and Lifecycle#

Storage Class	Access Pattern	Retrieval	Cost (relative)
S3 Standard	Frequent access	Immediate	$$$
S3 Intelligent-Tiering	Unknown/changing patterns	Immediate	$$ (auto-tiers)
S3 Standard-IA	Infrequent (monthly)	Immediate	$$
S3 One Zone-IA	Infrequent, non-critical	Immediate	$
S3 Glacier Instant	Rare, needs instant access	Immediate	$
S3 Glacier Flexible	Archival	Minutes to hours	¢
S3 Glacier Deep Archive	Compliance, long-term	12-48 hours	¢¢

Lifecycle Policy Example#

{
 "Rules": [
 {
 "ID": "ArchiveOldLogs",
 "Status": "Enabled",
 "Prefix": "logs/",
 "Transitions": [
 { "Days": 30, "StorageClass": "STANDARD_IA" },
 { "Days": 90, "StorageClass": "GLACIER" }
 ],
 "Expiration": { "Days": 365 }
 }
 ]
}

CDN Architecture#

A Content Delivery Network (CDN) caches content at geographically distributed edge locations, reducing latency for end users by serving content from a nearby point of presence (PoP).

Reliability & Resilience

Mon, 01 Jan 0001 00:00:00 +0000

Reliability & Resilience#

Failures are inevitable in distributed systems. Reliability engineering is about designing systems that continue operating correctly despite hardware failures, network partitions, software bugs, and operator errors. Resilience is the ability to recover quickly when failures occur.

Fault Tolerance Principles#

Principle	Description
Assume failure	Design every component assuming it will fail
Blast radius	Contain failures to the smallest possible scope
Graceful degradation	Serve partial results rather than complete failure
No single point of failure	Every critical path has redundancy
Fail fast	Detect failures quickly rather than hanging
Design for recovery	Optimise mean time to recovery (MTTR), not just mean time between failures (MTBF)

Failure Categories#

Type	Examples	Mitigation
Hardware	Disk failure, server crash	Replication, multi-AZ
Network	Partition, packet loss, high latency	Timeouts, retries, circuit breakers
Software	Bugs, memory leaks, crashes	Health checks, auto-restart, canary deploys
Dependency	Third-party API down	Circuit breaker, fallbacks, bulkheads
Capacity	Traffic spike, resource exhaustion	Auto-scaling, rate limiting, load shedding
Human	Misconfig, bad deploy	Canary releases, rollback procedures

Redundancy Patterns#

Active-Active#

All instances handle traffic simultaneously. If one fails, the remaining instances absorb its load.

Observability

Mon, 01 Jan 0001 00:00:00 +0000

Observability#

Observability is the ability to understand the internal state of a system by examining its external outputs. In distributed systems, you cannot debug by attaching a debugger — you need telemetry: metrics, logs, and traces working together to answer “why is this broken?”

Observability vs Monitoring#

Aspect	Monitoring	Observability
Approach	Predefined checks against known failures	Explore unknown failures dynamically
Questions	“Is the system up?” (known-knowns)	“Why is it slow for user X?” (unknown-unknowns)
Data	Predefined metrics and thresholds	High-cardinality, high-dimensionality data
Alerts	Threshold-based (CPU > 80%)	SLO burn-rate based
Debug flow	Dashboard → runbook → fix	Query telemetry → form hypothesis → verify

Monitoring tells you something is wrong. Observability helps you understand why.

Security Architecture

Mon, 01 Jan 0001 00:00:00 +0000

Security Architecture#

Security is not a feature you bolt on after launch — it is an architectural concern woven through every layer of the system. This section covers the patterns for authentication at scale, zero-trust networking, and secrets management in distributed systems.

Authentication at Scale#

Token-Based Authentication (JWT)#

Client → Login(credentials) → Auth Service → JWT ← Client stores token
Client → API Request + JWT → API Gateway → Validates JWT → Backend Service

Component	Responsibility
Auth Service	Validates credentials, issues tokens
JWT (access token)	Short-lived (5-15 min), carries claims (user ID, roles)
Refresh token	Long-lived (days/weeks), used to get new access tokens
API Gateway	Validates JWT signature, checks expiration, extracts claims

OAuth 2.0 / OpenID Connect Flows#

Flow	Use Case	Example
Authorization Code + PKCE	Web apps, SPAs, mobile	“Log in with Google”
Client Credentials	Service-to-service	Backend calling another backend
Device Authorization	Smart TVs, CLIs	“Enter this code at example.com/device”

Session vs Token#

	Server-Side Sessions	JWT Tokens
Storage	Server (Redis, DB)	Client (cookie, localStorage)
Revocation	Delete session record	Cannot revoke until expiry (unless denylist)
Scaling	Requires shared session store	Stateless — any server can validate
Size	Session ID (small)	JWT payload (larger, contains claims)

Authorisation Patterns#

RBAC (Role-Based Access Control)#

User → has → Role → has → Permissions
Alice → Admin → [read, write, delete]
Bob → Editor → [read, write]
Carol → Viewer → [read]

Simple, works for most applications. Breaks down when permissions depend on the specific resource.

Search & Analytics

Mon, 01 Jan 0001 00:00:00 +0000

Search & Analytics#

Most applications eventually need full-text search, analytics dashboards, or both. This section covers the systems and patterns for indexing, querying, and analysing large datasets — from Elasticsearch to data lake architectures.

Full-Text Search#

Why Not Just SQL LIKE?#

SELECT * FROM products WHERE name LIKE '%wireless headphones%';

Problem	Explanation
No relevance ranking	Results are unordered — no way to rank “best match”
No fuzzy matching	“headphnes” won’t match “headphones”
No stemming	“running” won’t match “run”
Full table scan	`LIKE '%term%'` cannot use indexes — O(n)
No synonym support	“laptop” won’t match “notebook”

Search Engine Architecture#

Documents → Analyser (tokenise, stem, normalise) → Inverted Index → Query → Ranked Results

Inverted index: maps each term to the list of documents containing it: