Statistics#

Statistics is the discipline of learning from data — summarising it, drawing inferences, and making decisions under uncertainty. For engineers, it is the foundation of A/B testing, performance benchmarking, capacity planning, and data-driven decision-making.

Descriptive Statistics#

Measures of Central Tendency#

Example: Response times [12, 15, 14, 13, 850] ms

• Mean = 180.8 ms (misleading — one outlier inflates it)
• Median = 14 ms (representative)
• P99 = 850 ms (captures tail behaviour)

Measures of Spread#

Percentiles#

The pth percentile is the value below which p% of observations fall.

Key insight: Averages hide tail behaviour. If your SLO is P99 < 200ms, an average of 50ms tells you nothing about whether you are meeting it.

Data Distributions in Practice#

Recognising Distributions#

Log-Normal Distribution#

If log(X) is normally distributed, X is log-normal. This models many engineering quantities:

• Response times
• File sizes
• User session lengths
• Error counts

Practical consequence: Use geometric mean or median for log-normal data, not arithmetic mean. Log-transform before applying methods that assume normality.

Inferential Statistics#

Sampling#

The standard error decreases with √n: to halve the uncertainty, quadruple the sample size.

Confidence Intervals#

A range that contains the true parameter with a specified probability.

95% CI for mean = x̄ ± 1.96 × (s / √n)

Common mistake: “95% CI” does NOT mean “95% probability the true value is in this interval.” It means: if you repeated the experiment many times, 95% of the CIs would contain the true value.

Hypothesis Testing#

A structured framework for deciding whether observed data is consistent with a null hypothesis.

The Process#

1. State hypotheses:

   • H₀ (null): no effect, no difference (the default)
   • H₁ (alternative): there is an effect or difference

2. Choose significance level α (usually 0.05)

3. Compute test statistic from the data

4. Calculate p-value: probability of seeing results this extreme (or more) under H₀

5. Decision:

   • p ≤ α → reject H₀ (statistically significant)
   • p > α → fail to reject H₀ (not enough evidence)

Error Types#

Statistical power = 1 - β = probability of detecting a real effect.

Common Tests#

P-Value Misinterpretations#

A/B Testing#

The controlled experiment framework used in product engineering.

Design#

Control (A):  Current version    → measure outcome
Treatment (B): Modified version  → measure outcome

Sample Size Calculation#

To detect a minimum detectable effect (MDE) with power 1-β and significance α:

n ≈ (z_α/2 + z_β)² × 2σ² / MDE²

Rule of thumb: Halving the MDE requires 4× the sample size.

Common Pitfalls#

Regression#

Linear Regression#

Models the relationship between a dependent variable y and independent variable(s) x:

y = β₀ + β₁x₁ + β₂x₂ + ... + ε

Where β₀ is the intercept, β₁…βₙ are coefficients, and ε is the error term.

Ordinary Least Squares (OLS)#

Finds coefficients that minimise the sum of squared residuals:

min Σ (yᵢ - ŷᵢ)²

Evaluating Fit#

Interpreting Coefficients#

β₁ = 2.5 means: for each unit increase in x₁, y increases by 2.5 on average, holding other variables constant.

Correlation vs. Causation#

Correlation (r) measures linear association:

• r = +1: perfect positive linear
• r = 0: no linear relationship
• r = -1: perfect negative linear

r = Σ(xᵢ - x̄)(yᵢ - ȳ) / √(Σ(xᵢ - x̄)² × Σ(yᵢ - ȳ)²)

Correlation does not imply causation. Ice cream sales and drowning deaths are correlated (both increase in summer) but one does not cause the other.

To establish causation, you need:

1. Controlled experiments (A/B tests)
2. Natural experiments
3. Instrumental variables
4. Or very careful observational study with domain knowledge

Practical Statistics for Engineers#

SLO Math#

If your SLO is 99.9% availability per month:

Error budget = 1 - 0.999 = 0.001
Minutes in a month ≈ 43,200
Allowed downtime = 43,200 × 0.001 = 43.2 minutes/month

Capacity Planning with Percentiles#

If P99 latency = 200ms and you have 1000 req/s:

• 10 requests per second experience ≥ 200ms
• Over 24 hours: 864,000 slow requests

Anomaly Detection#

Standard approach: flag values outside μ ± kσ

For non-normal data, use MAD (Median Absolute Deviation) instead of standard deviation.

The Bootstrap#

A resampling technique: sample with replacement from your data to estimate the distribution of any statistic.

import numpy as np

data = [12, 15, 14, 13, 16, 11, 18]
bootstrap_means = [np.mean(np.random.choice(data, len(data))) for _ in range(10000)]
ci_lower = np.percentile(bootstrap_means, 2.5)
ci_upper = np.percentile(bootstrap_means, 97.5)

No assumptions about the underlying distribution. Works for any statistic (mean, median, P99, ratio).

Key Takeaways#

• Use median and percentiles for skewed data (response times, salaries). Mean hides tail behaviour.
• Confidence intervals quantify uncertainty; p-values quantify surprise. Neither proves anything absolutely.
• A/B testing requires pre-calculated sample sizes, fixed analysis points, and correction for multiple comparisons.
• Correlation ≠ causation. Controlled experiments (A/B tests) are the gold standard for causal inference.
• Linear regression is the foundation of prediction models. R² tells you how much variance is explained; RMSE tells you how wrong you are on average.
• The bootstrap is the most practical tool for uncertainty estimation — it requires no distributional assumptions.