Compute Patterns#

Choosing the right compute model is one of the highest-leverage architecture decisions you’ll make. It affects cost, operational complexity, scaling behaviour, and team velocity. This section provides frameworks for making that decision and patterns for getting the most out of each model.

Compute Decision Framework#

Every workload has different characteristics. Match the compute model to the workload, not the other way around.

What is the workload profile?
│
├── Stateless request/response, runs < 15 min
│   ├── < 1M requests/day → FaaS (Lambda / Cloud Functions)
│   └── > 1M requests/day → Containers (cost-optimised)
│
├── Long-running process, always-on
│   ├── Needs OS-level control → VMs
│   └── Doesn't need OS-level control → Containers
│
├── Batch processing, can tolerate interruption
│   └── Spot/preemptible instances (VMs or containers)
│
├── GPU/ML training or inference
│   ├── Training → GPU VMs or managed ML (SageMaker, Vertex AI)
│   └── Inference → GPU containers or serverless inference
│
└── Legacy app, can't be modified
    └── VMs (lift and shift)

Compute Model Comparison#

Auto-Scaling Patterns#

Auto-scaling adjusts compute capacity based on demand. The goal is to match capacity to load with minimal lag and minimal waste.

Scaling Strategies#

Scaling Architecture#

                     Metric Source
                    (CloudWatch / Prometheus / Custom)
                          │
                          ▼
                  ┌───────────────┐
                  │  Scaling       │
                  │  Policy        │
                  │                │
                  │  Target: CPU   │
                  │  at 60%        │
                  └───────┬───────┘
                          │
              ┌───────────┼───────────┐
              ▼           ▼           ▼
         ┌────────┐  ┌────────┐  ┌────────┐
         │ Inst 1 │  │ Inst 2 │  │ Inst 3 │   ← Current: 3 instances
         └────────┘  └────────┘  └────────┘

    CPU spikes to 85% → Policy triggers scale-out

         ┌────────┐  ┌────────┐  ┌────────┐  ┌────────┐  ┌────────┐
         │ Inst 1 │  │ Inst 2 │  │ Inst 3 │  │ Inst 4 │  │ Inst 5 │
         └────────┘  └────────┘  └────────┘  └────────┘  └────────┘
                                                  ↑ new       ↑ new

Scaling Design Rules#

1. Scale out fast, scale in slow — add capacity immediately, remove it after a cooldown to avoid flapping
2. Don’t scale on a single metric — combine CPU, memory, request latency, and queue depth
3. Set minimum instances ≥ 2 — one instance is a single point of failure, even with auto-scaling
4. Pre-warm for known events — if you know Black Friday is coming, schedule capacity in advance
5. Test your scaling — run load tests to verify that scaling triggers fire correctly and fast enough

Cooldown and Stabilisation#

Spot and Preemptible Strategies#

Spot instances (AWS), preemptible VMs (GCP), and spot VMs (Azure) offer 60–90% discounts compared to on-demand pricing. The trade-off: the provider can reclaim them with short notice (2 minutes on AWS, 30 seconds on GCP).

Spot Architecture Patterns#

Mixed Fleet Architecture#

┌──────────────────────────────────────────────────┐
│                  Auto Scaling Group               │
│                                                    │
│  Base (On-Demand): 4 instances                    │
│  ┌────┐ ┌────┐ ┌────┐ ┌────┐                     │
│  │ OD │ │ OD │ │ OD │ │ OD │     ← Always-on     │
│  └────┘ └────┘ └────┘ └────┘       baseline       │
│                                                    │
│  Burst (Spot): 0–8 instances                      │
│  ┌────┐ ┌────┐ ┌────┐ ┌────┐ ┌────┐              │
│  │ SP │ │ SP │ │ SP │ │ SP │ │ SP │  ← Elastic   │
│  └────┘ └────┘ └────┘ └────┘ └────┘    burst      │
│                                                    │
│  Types: m5.xlarge, m5a.xlarge, m6i.xlarge,        │
│         m6a.xlarge (diversified)                   │
└──────────────────────────────────────────────────┘

Spot Interruption Handling#

Spot Interruption Notice (2 min warning)
        │
        ├── 1. Stop accepting new requests (deregister from LB)
        ├── 2. Drain in-flight requests (connection draining)
        ├── 3. Checkpoint state to durable storage (S3/EBS)
        ├── 4. Signal "unhealthy" to orchestrator
        └── 5. Replacement instance launches automatically

Right-Sizing Methodology#

Over-provisioning wastes money. Under-provisioning degrades performance. Right-sizing is the continuous process of matching instance types to actual usage.

Right-Sizing Process#

Instance Family Selection#

Graviton / ARM: ARM-based instances (AWS Graviton, Azure Ampere, GCP Tau T2A) offer 20–40% better price/performance for most workloads. Default to ARM unless your software requires x86.

GPU and ML Compute#

Machine learning workloads have unique compute requirements: massive parallelism for training, low-latency inference for serving.

Training vs Inference#

Inference Architecture Options#

Container Orchestration Decisions#

If you choose containers, you still need to decide how to run them.

Decision Guide#

Do you need Kubernetes-specific features (CRDs, operators, service mesh)?
├── Yes → Managed Kubernetes (EKS, GKE, AKS)
└── No
    ├── Do you need to run on spot/preemptible nodes?
    │   ├── Yes → Managed Kubernetes with spot node pools
    │   └── No
    │       └── Do you want zero infrastructure management?
    │           ├── Yes → Serverless containers (Fargate, Cloud Run)
    │           └── No  → Managed containers (ECS on EC2, ACA)

Key Takeaways#

• Match compute model to workload characteristics: VMs for control, containers for portability, FaaS for event-driven
• Auto-scaling works best when you combine multiple signals (CPU, latency, queue depth) and scale out fast, in slow
• Spot instances provide 60–90% savings but require interruption-tolerant architecture: diversified fleets, checkpointing, and queue-based workers
• Right-size quarterly by analysing actual CPU and memory utilisation against instance capacity
• Default to ARM-based instances unless you have an explicit x86 requirement — the price/performance advantage is significant
• For ML workloads, separate training (spot-friendly, batch) from inference (latency-sensitive, auto-scaled)