Reliability & Resilience#

Failures are inevitable in distributed systems. Reliability engineering is about designing systems that continue operating correctly despite hardware failures, network partitions, software bugs, and operator errors. Resilience is the ability to recover quickly when failures occur.

Fault Tolerance Principles#

Failure Categories#

Redundancy Patterns#

Active-Active#

All instances handle traffic simultaneously. If one fails, the remaining instances absorb its load.

graph LR
    LB[Load Balancer] --> A[Instance A - Active]
    LB --> B[Instance B - Active]
    LB --> C[Instance C - Active]

Active-Passive#

One instance handles traffic (active); the standby takes over only on failure.

graph LR
    LB[Load Balancer] --> A[Instance A - Active]
    LB -.->|Failover| B[Instance B - Passive/Standby]

Circuit Breaker Pattern#

The circuit breaker prevents a service from repeatedly calling a failing dependency, giving it time to recover.

State Diagram#

stateDiagram-v2
    [*] --> Closed
    Closed --> Open : Failure threshold reached
    Open --> HalfOpen : Timeout expires
    HalfOpen --> Closed : Probe succeeds
    HalfOpen --> Open : Probe fails

States Explained#

Configuration Parameters#

Implementation Example#

class CircuitBreaker:
    def __init__(self, failure_threshold=5, timeout=30):
        self.state = "CLOSED"
        self.failure_count = 0
        self.failure_threshold = failure_threshold
        self.timeout = timeout
        self.last_failure_time = None
    
    def call(self, func, *args):
        if self.state == "OPEN":
            if time.time() - self.last_failure_time > self.timeout:
                self.state = "HALF_OPEN"
            else:
                raise CircuitOpenError("Circuit is open")
        
        try:
            result = func(*args)
            self._on_success()
            return result
        except Exception as e:
            self._on_failure()
            raise
    
    def _on_success(self):
        self.failure_count = 0
        self.state = "CLOSED"
    
    def _on_failure(self):
        self.failure_count += 1
        self.last_failure_time = time.time()
        if self.failure_count >= self.failure_threshold:
            self.state = "OPEN"

Retries with Exponential Backoff and Jitter#

Retries handle transient failures (network blips, temporary overload). Without backoff, retries amplify the problem (thundering herd).

Exponential Backoff Formula#

wait_time = base_delay * (2 ^ attempt) + random_jitter

Why Jitter Matters#

Without jitter, all clients retry at exactly the same time (correlated retries), creating periodic thundering herds. Full jitter spreads retries uniformly:

import random
import time

def retry_with_backoff(func, max_retries=4, base_delay=1.0):
    for attempt in range(max_retries):
        try:
            return func()
        except TransientError:
            if attempt == max_retries - 1:
                raise
            # Full jitter: uniform random between 0 and exponential cap
            delay = random.uniform(0, base_delay * (2 ** attempt))
            time.sleep(delay)

Retry Best Practices#

• Only retry on transient errors (5xx, timeouts, connection reset) — never on 4xx
• Set a maximum retry count (3-5 attempts typical)
• Set a maximum total timeout (don’t retry forever)
• Add jitter to prevent thundering herd
• Make operations idempotent before adding retries

Bulkhead Pattern#

Named after ship bulkheads that contain flooding, the bulkhead pattern isolates failures by partitioning resources.

graph TD
    subgraph "Without Bulkhead"
        App1[Application] --> Pool[Shared Thread Pool: 100 threads]
        Pool --> API1[API A - Slow]
        Pool --> API2[API B - Healthy]
        Pool --> API3[API C - Healthy]
    end

If API A becomes slow, it consumes all 100 threads — APIs B and C are also affected.

graph TD
    subgraph "With Bulkhead"
        App2[Application] --> P1[Pool A: 30 threads]
        App2 --> P2[Pool B: 35 threads]
        App2 --> P3[Pool C: 35 threads]
        P1 --> APIA[API A - Slow]
        P2 --> APIB[API B - Healthy]
        P3 --> APIC[API C - Healthy]
    end

API A can only consume its allocated 30 threads. APIs B and C continue functioning normally.

Timeouts#

Every network call must have a timeout. Without timeouts, a hung dependency hangs the caller indefinitely.

Timeout Budget Pattern#

In a chain of services (A → B → C), each hop must have a smaller timeout than its caller:

Client timeout: 10s
  → Service A timeout to B: 5s
    → Service B timeout to C: 2s

If the budget is exhausted at any point, fail fast rather than consuming the remaining budget.

Graceful Degradation#

When a dependency fails, serve reduced functionality rather than failing completely.

Chaos Engineering#

Chaos engineering deliberately injects failures into production (or staging) to discover weaknesses before they manifest in real incidents.

Principles#

1. Define steady state — what does “normal” look like in metrics?
2. Hypothesise — “If we kill this instance, traffic shifts within 30s”
3. Inject failure — kill instance, add latency, partition network
4. Observe — did the system behave as hypothesised?
5. Fix weaknesses — address any unexpected behaviour

Common Experiments#

Tools: Netflix Chaos Monkey, Gremlin, AWS Fault Injection Simulator, LitmusChaos.

Disaster Recovery (RPO / RTO)#

DR Strategies (Increasing Cost and Speed)#

Blue-Green and Canary Deployments#

Blue-Green#

Two identical environments. One (Blue) serves production; the other (Green) receives the new deployment. Switch traffic instantly.

graph LR
    LB[Load Balancer] -->|100%| Blue[Blue - Current v1.2]
    LB -.->|0%| Green[Green - New v1.3]
    
    Note[After validation: switch 100% to Green]

Canary Deployment#

Route a small percentage of traffic to the new version. Gradually increase if metrics are healthy.

graph LR
    LB[Load Balancer] -->|95%| V1[v1.2 - Stable]
    LB -->|5%| V2[v1.3 - Canary]

Progression: 5% → 25% → 50% → 100% (rollback at any stage if errors spike)

Health Checks and Readiness Probes#

Health Check Depth#

Best practice: Load balancer health checks should be shallow (fast, low overhead). Deep checks are for monitoring/alerting, not traffic routing.

Implementation#

@app.get("/health/live")
def liveness():
    """Am I alive? Always return 200 unless process is hung."""
    return {"status": "ok"}

@app.get("/health/ready")
def readiness():
    """Can I serve traffic? Check critical dependencies."""
    db_ok = check_database_connection()
    cache_ok = check_cache_connection()
    if db_ok and cache_ok:
        return {"status": "ready"}
    return JSONResponse(status_code=503, content={"status": "not ready"})

Key Takeaways#

• Design for failure from day one — every dependency will eventually fail; plan your response in advance
• Circuit breakers prevent cascading failures — fail fast and give dependencies time to recover
• Retries must use exponential backoff with jitter — otherwise retries amplify the problem (thundering herd)
• Bulkheads isolate blast radius — one slow dependency shouldn’t bring down unrelated functionality
• Every network call needs a timeout — a missing timeout is a bug waiting to become an incident
• Graceful degradation beats total failure — serve partial results, cached data, or defaults when components fail
• Chaos engineering validates your resilience — test failure handling before real failures test it for you
• RPO and RTO drive DR strategy — know your business requirements before choosing (and paying for) a DR approach
• Canary deployments catch problems early — expose new code to a fraction of traffic before full rollout