Performance Testing#

Performance testing validates that a system meets its speed, scalability, and stability requirements under expected and extreme conditions. It identifies bottlenecks before they reach production and provides data to support capacity planning and SLO validation.


Types of Performance Tests#

TypePurposeDurationLoad Pattern
Load testValidate behavior under expected load5–30 minRamp to target, hold steady
Stress testFind breaking point15–60 minRamp beyond capacity
Soak testDetect memory leaks, resource exhaustion2–24 hoursConstant moderate load
Spike testValidate auto-scaling and recovery5–15 minSudden burst, then drop
Breakpoint testFind maximum capacityVariableIncremental ramp until failure
BenchmarkEstablish baseline for comparison1–5 minFixed load, repeatable

Load Testing Tools Comparison#

Featurek6LocustJMeterGatling
LanguageJavaScriptPythonXML/GUIScala/Java
ProtocolHTTP, gRPC, WebSocketHTTP, customHTTP, JDBC, JMS, FTPHTTP, WebSocket
ScriptingCode-firstCode-firstGUI + codeCode-first (DSL)
Resource usageLow (Go runtime)Medium (Python)High (Java)Medium (JVM)
DistributedBuilt-in (k6 Cloud)Built-in (master/worker)JMeter RemoteDistributed by design
ReportingCLI + JSON + GrafanaWeb UIHTML/XML reportsHTML reports
Learning curveLowLowMediumMedium
Best forCI/CD integration, developersPython teams, custom protocolsEnterprise, complex scenariosHigh-throughput HTTP

k6 — Developer-Friendly Load Testing#

Basic Load Test#

import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  stages: [
    { duration: '1m', target: 50 },   // Ramp up to 50 users
    { duration: '3m', target: 50 },   // Hold at 50 users
    { duration: '1m', target: 100 },  // Ramp to 100 users
    { duration: '3m', target: 100 },  // Hold at 100 users
    { duration: '1m', target: 0 },    // Ramp down
  ],
  thresholds: {
    http_req_duration: ['p(95)<300', 'p(99)<500'],  // SLO validation
    http_req_failed: ['rate<0.01'],                  // <1% error rate
  },
};

export default function () {
  const res = http.get('https://api.example.com/users');

  check(res, {
    'status is 200': (r) => r.status === 200,
    'response time < 300ms': (r) => r.timings.duration < 300,
    'body has users': (r) => JSON.parse(r.body).length > 0,
  });

  sleep(1); // Think time between requests
}

Running k6#

# Local run
k6 run load-test.js

# With custom output
k6 run --out json=results.json load-test.js

# With environment variables
k6 run -e API_URL=https://staging.example.com load-test.js

Locust — Python-Based Load Testing#

from locust import HttpUser, task, between

class WebsiteUser(HttpUser):
    wait_time = between(1, 3)  # Think time: 1-3 seconds

    def on_start(self):
        """Login on start."""
        self.client.post("/login", json={
            "username": "testuser",
            "password": "testpass"
        })

    @task(3)  # Weight: 3x more likely than other tasks
    def view_items(self):
        self.client.get("/api/items")

    @task(1)
    def create_item(self):
        self.client.post("/api/items", json={
            "name": "Test Item",
            "price": 29.99
        })

    @task(2)
    def view_item_detail(self):
        item_id = 42  # In practice, pick from a pool
        self.client.get(f"/api/items/{item_id}")

Running Locust#

# Start with web UI
locust -f locustfile.py --host=https://api.example.com

# Headless mode (for CI)
locust -f locustfile.py --headless -u 100 -r 10 --run-time 5m \
  --host=https://api.example.com

Stress Testing#

Push beyond expected capacity to find the breaking point and observe degradation patterns.

// k6 stress test
export const options = {
  stages: [
    { duration: '2m', target: 100 },   // Normal load
    { duration: '2m', target: 200 },   // High load
    { duration: '2m', target: 500 },   // Very high load
    { duration: '2m', target: 1000 },  // Breaking point?
    { duration: '5m', target: 0 },     // Recovery
  ],
};

What to observe:

  • At what load does latency degrade significantly?
  • Does the system recover after load drops?
  • Are errors graceful (429, 503) or catastrophic (timeouts, crashes)?
  • Does auto-scaling trigger and at what threshold?

Soak Testing#

Run moderate load for an extended period to detect slow resource leaks.

// k6 soak test
export const options = {
  stages: [
    { duration: '5m', target: 50 },     // Ramp up
    { duration: '8h', target: 50 },     // Sustained load
    { duration: '5m', target: 0 },      // Ramp down
  ],
  thresholds: {
    http_req_duration: ['p(95)<500'],
  },
};

Detects:

  • Memory leaks (gradually increasing memory usage)
  • Connection pool exhaustion
  • File descriptor leaks
  • Database connection leaks
  • Log file growth filling disk

Benchmarking#

Establish repeatable baselines for isolated components:

# Python benchmarking with pytest-benchmark
import pytest

def fibonacci(n):
    if n <= 1:
        return n
    return fibonacci(n - 1) + fibonacci(n - 2)

def test_fibonacci_benchmark(benchmark):
    result = benchmark(fibonacci, 20)
    assert result == 6765

HTTP Endpoint Benchmarking#

# Using hey (simple HTTP benchmarker)
hey -n 10000 -c 100 -m GET https://api.example.com/health

# Using wrk (scriptable benchmarker)
wrk -t12 -c400 -d30s https://api.example.com/users

Profiling & Identifying Bottlenecks#

Types of Profiling#

Profiler TypeMeasuresTools
CPU profilerTime spent in functionscProfile, py-spy, async-profiler
Memory profilerAllocations, heap usagememray, tracemalloc, Valgrind
I/O profilerDisk and network waitsstrace, eBPF
Database profilerQuery execution timeEXPLAIN ANALYZE, slow query log
Network profilerLatency, throughputtcpdump, Wireshark

Python CPU Profiling#

import cProfile
import pstats

# Profile a function
profiler = cProfile.Profile()
profiler.enable()
result = expensive_function()
profiler.disable()

# View results
stats = pstats.Stats(profiler)
stats.sort_stats('cumulative')
stats.print_stats(20)  # Top 20 functions

Common Bottleneck Patterns#

SymptomLikely CauseInvestigation
High latency, low CPUI/O bound (DB, network)Profile I/O waits, check query plans
High CPU, high latencyCPU bound (computation)CPU profiler, algorithmic complexity
Gradual degradationResource leakMonitor memory/connections over time
Sudden degradationConnection pool exhaustionCheck pool metrics, max connections
Inconsistent latencyGC pauses or lock contentionGC logs, thread dumps

SLO Validation#

Performance tests should validate your Service Level Objectives:

// k6 thresholds map directly to SLOs
export const options = {
  thresholds: {
    // Latency SLOs
    'http_req_duration{endpoint:search}': ['p95<200'],
    'http_req_duration{endpoint:checkout}': ['p99<1000'],

    // Availability SLO
    'http_req_failed': ['rate<0.001'],  // 99.9% success rate

    // Throughput SLO
    'http_reqs': ['rate>100'],  // At least 100 RPS sustained
  },
};

SLO Metrics to Test#

SLO TypeMetricExample Target
LatencyP50, P95, P99 response timeP95 < 200ms
AvailabilityError rate< 0.1% errors
ThroughputRequests per second> 500 RPS
SaturationResource utilization under loadCPU < 70% at peak

Performance Budgets#

Define acceptable performance boundaries and enforce them in CI:

// k6 budget enforcement
export const options = {
  thresholds: {
    // Page load budget
    'http_req_duration{type:page}': ['p95<2000'],

    // API response budget
    'http_req_duration{type:api}': ['p95<300'],

    // Payload size budget (custom metric)
    'response_size': ['p95<50000'],  // 50KB max
  },
};

// Custom metric for response size
import { Trend } from 'k6/metrics';
const responseSize = new Trend('response_size');

export default function () {
  const res = http.get(url);
  responseSize.add(res.body.length);
}

Budget Categories#

CategoryBudgetRationale
Time to First Byte< 200msServer responsiveness
API response (simple)< 100ms P95User-perceived speed
API response (complex)< 500ms P95Acceptable for dashboards
Search queries< 300ms P95User expectation from Google
File uploads< 5s for 10MBPractical limit
WebSocket connect< 500msReal-time feature readiness

CI/CD Integration#

GitHub Actions Example#

name: Performance Tests
on:
  pull_request:
    branches: [main]

jobs:
  load-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Run k6 load test
        uses: grafana/k6-action@v0.3.1
        with:
          filename: tests/performance/load-test.js
          flags: --out json=results.json

      - name: Upload results
        uses: actions/upload-artifact@v4
        with:
          name: k6-results
          path: results.json

When to Run Performance Tests#

TriggerTest TypeDuration
Every PRSmoke test (minimal load)1–2 min
NightlyFull load test15–30 min
Pre-releaseStress + soak test2–8 hours
Post-deployBaseline comparison5 min

Reporting & Analysis#

Key Metrics to Report#

MetricWhat It Tells You
P50 (median)Typical user experience
P95Experience for most users
P99Worst case for 99% of users
MaxAbsolute worst case (often outlier)
Error rateSystem reliability under load
Throughput (RPS)System capacity
Concurrent usersHow many simultaneous users supported

Comparing Results#

Baseline (main):    P50=45ms  P95=120ms  P99=340ms  Errors=0.02%
This PR:            P50=48ms  P95=180ms  P99=520ms  Errors=0.05%
                    (+7%)     (+50%)     (+53%)     (+150%)

Flag regressions: if P95 increases by more than 20%, the PR requires investigation.


Key Takeaways#

  • Load testing validates SLOs — run against realistic traffic patterns, not just peak numbers
  • k6 is the best choice for developer-centric, CI-integrated testing; Locust wins for Python teams needing custom protocols
  • Stress tests find the ceiling, soak tests find leaks — both are essential for production readiness
  • Profile before optimizing — measure where time is actually spent, don’t guess
  • Performance budgets in CI prevent regressions from reaching production — treat them like test failures
  • P95 and P99 matter more than averages — averages hide the experience of your worst-served users
  • Run smoke performance tests on every PR, full suites nightly or pre-release
  • Always test with realistic data volumes and user patterns — empty databases give misleading results