Section 06 introduced Big-O notation and common complexity classes. This section goes deeper — covering formal analysis techniques, space complexity, amortised analysis, and practical measurement. The goal is to move from “memorising complexities” to “deriving them yourself.”


Big-O: The Formal Definition#

O(f(n)) describes an upper bound on growth rate:

T(n) is O(f(n)) if there exist constants c > 0 and n₀ ≥ 0 such that T(n) ≤ c·f(n) for all n ≥ n₀.

In practice: drop constants, drop lower-order terms, keep the dominant term.

# T(n) = 3n² + 7n + 42
# Dominant term: n²
# Therefore: O(n²)

# T(n) = 5·log(n) + 1000
# Dominant term: log(n)
# Therefore: O(log n)
NotationMeaningAnalogy
O(f(n))Upper bound (at most)≤
Ω(f(n))Lower bound (at least)≥
Θ(f(n))Tight bound (exactly)=
o(f(n))Strict upper (less than)<
ω(f(n))Strict lower (greater than)>

In interviews and daily work, O(f(n)) is used for everything. Θ is technically more precise when stating exact complexity.


Time Complexity: Deriving It Yourself#

Technique 1: Count Loop Iterations#

# Single loop: O(n)
for i in range(n):
    do_something()  # O(1) work per iteration → n × O(1) = O(n)

# Nested loops: O(n²)
for i in range(n):
    for j in range(n):
        do_something()  # n × n × O(1) = O(n²)

# Nested but dependent: still O(n²)
for i in range(n):
    for j in range(i):     # j runs 0, 1, 2, ..., n-1 times
        do_something()     # Total: 0+1+2+...+(n-1) = n(n-1)/2 = O(n²)

Technique 2: Halving → Logarithmic#

Whenever the problem size is halved each step, the complexity is O(log n):

# Binary search: halves the search space each iteration
def binary_search(arr, target):
    low, high = 0, len(arr) - 1
    while low <= high:          # Loop runs log₂(n) times
        mid = (low + high) // 2
        if arr[mid] == target:
            return mid
        elif arr[mid] < target:
            low = mid + 1       # Halve the space
        else:
            high = mid - 1      # Halve the space
    return -1

Technique 3: Recurrence Relations#

For recursive algorithms, write the recurrence and solve it:

AlgorithmRecurrenceSolution
Binary searchT(n) = T(n/2) + O(1)O(log n)
Merge sortT(n) = 2T(n/2) + O(n)O(n log n)
Naive FibonacciT(n) = T(n-1) + T(n-2) + O(1)O(2ⁿ)
Strassen multiplicationT(n) = 7T(n/2) + O(n²)O(n^2.81)

The Master Theorem (Simplified)#

For recurrences of the form T(n) = aT(n/b) + O(nᵈ):

ConditionResult
d > log_b(a)O(nᵈ)
d = log_b(a)O(nᵈ log n)
d < log_b(a)O(n^(log_b(a)))
Merge sort: T(n) = 2T(n/2) + O(n)
a=2, b=2, d=1 → log₂(2) = 1 = d → O(n log n) ✓

Space Complexity#

Space complexity measures additional memory used beyond the input itself.

Source of spaceExample
VariablesCounters, pointers — O(1)
Data structuresHash maps, arrays — O(n)
Recursion stackRecursive calls — O(depth)
OutputResult array — O(output size)
# O(1) space — sorts in place
def bubble_sort(arr):
    n = len(arr)
    for i in range(n):
        for j in range(n - i - 1):
            if arr[j] > arr[j + 1]:
                arr[j], arr[j + 1] = arr[j + 1], arr[j]

# O(n) space — creates new arrays
def merge_sort(arr):
    if len(arr) <= 1:
        return arr
    mid = len(arr) // 2
    left = merge_sort(arr[:mid])    # New array
    right = merge_sort(arr[mid:])   # New array
    return merge(left, right)       # New merged array

Time-Space Tradeoffs#

StrategyTimeSpaceExample
RecomputeHigherLowerCalculate value each time
Cache/memoiseLowerHigherStore computed results
In-place sortSameLowerQuicksort vs merge sort
Hash table lookupO(1) lookupO(n) storageTrading space for time
graph LR
    A["Time-Space Tradeoff"] --> B["More space → Less time"]
    A --> C["Less space → More time"]
    B --> D["Hash tables, caches, DP tables"]
    C --> E["In-place algorithms, streaming"]

Best, Average, and Worst Case#

The same algorithm can have different performance depending on input.

CaseDefinitionWhen to care
Best caseMinimum operations on any input of size nRarely useful (too optimistic)
Average caseExpected operations over all possible inputsReal-world performance
Worst caseMaximum operations on any input of size nGuarantees and SLA design

Quicksort Example#

def quicksort(arr):
    if len(arr) <= 1:
        return arr
    pivot = arr[0]
    left = [x for x in arr[1:] if x <= pivot]
    right = [x for x in arr[1:] if x > pivot]
    return quicksort(left) + [pivot] + quicksort(right)
CaseWhen it happensComplexity
BestPivot always splits evenlyO(n log n)
AverageRandom inputO(n log n)
WorstAlready sorted + first element pivotO(n²)

Common Algorithms — All Three Cases#

AlgorithmBestAverageWorstWorst space
Linear searchO(1)O(n)O(n)O(1)
Binary searchO(1)O(log n)O(log n)O(1)
Insertion sortO(n)O(n²)O(n²)O(1)
Merge sortO(n log n)O(n log n)O(n log n)O(n)
QuicksortO(n log n)O(n log n)O(n²)O(log n)
Hash table lookupO(1)O(1)O(n)O(n)

Amortised Analysis#

Some operations are expensive occasionally but cheap on average. Amortised analysis gives the average cost per operation over a sequence.

Dynamic Array (Python list append)#

# Python list doubles capacity when full
# Most appends are O(1), but resizing copies n elements = O(n)
# Over n appends: total work = n + n/2 + n/4 + ... ≈ 2n
# Amortised cost per append: O(2n/n) = O(1)
Operation #Array sizeCapacityResize?Cost
111No1
222Yes (copy 1)2
334Yes (copy 2)3
444No1
558Yes (copy 4)5
6-86-88No1 each

Total cost for 8 operations: 1+2+3+1+5+1+1+1 = 15. Average: 15/8 ≈ O(1) per operation.

Other Amortised O(1) Operations#

Data StructureOperationWorst singleAmortised
Dynamic arrayappendO(n)O(1)
Hash tableinsert (with rehash)O(n)O(1)
Splay treefindO(n)O(log n)
Union-Findfind (with path compression)O(log n)O(α(n)) ≈ O(1)

Common Complexity Classes#

graph LR
    A["O(1)"] --> B["O(log n)"]
    B --> C["O(n)"]
    C --> D["O(n log n)"]
    D --> E["O(n²)"]
    E --> F["O(2ⁿ)"]
    F --> G["O(n!)"]
ClassNameExamplen=1000
O(1)ConstantArray index, hash lookup1
O(log n)LogarithmicBinary search10
O(n)LinearLinear search, single pass1,000
O(n log n)LinearithmicMerge sort, good sorts10,000
O(n²)QuadraticNested loops, naive sort1,000,000
O(n³)CubicMatrix multiplication (naive)1,000,000,000
O(2ⁿ)ExponentialSubsets, brute-force10³⁰⁰
O(n!)FactorialPermutations10²⁵⁶⁷

Practical Limits (1 second, ~10⁸ operations)#

Max nAcceptable complexity
≤ 10O(n!), O(2ⁿ)
≤ 20O(2ⁿ)
≤ 500O(n³)
≤ 10⁴O(n²)
≤ 10⁶O(n log n)
≤ 10⁸O(n)
≤ 10¹⁸O(log n), O(1)

Practical Measurement#

Theory and practice can diverge due to constants, cache effects, and implementation details. Measure to validate.

Timing in Python#

import time

def measure(func, *args, repeats=5):
    """Measure average execution time."""
    times = []
    for _ in range(repeats):
        start = time.perf_counter()
        func(*args)
        times.append(time.perf_counter() - start)
    print(f"{func.__name__}: {sum(times)/len(times):.6f}s avg")

Empirical Growth Rate Detection#

def empirical_complexity(func, sizes):
    """Run at different sizes; observe ratio when n doubles."""
    prev_time = None
    for n in sizes:
        data = list(range(n))
        start = time.perf_counter()
        func(data)
        elapsed = time.perf_counter() - start
        ratio = f"{elapsed / prev_time:.1f}x" if prev_time else "-"
        print(f"n={n:>8}  time={elapsed:.4f}s  ratio={ratio}")
        prev_time = elapsed

# If ratio ≈ 2 when n doubles → O(n)
# If ratio ≈ 4 when n doubles → O(n²)
# If ratio ≈ 8 when n doubles → O(n³)

Common Analysis Patterns#

# Shrinking by half → O(log n)
i = n
while i > 0:
    i = i // 2

# Nested loops → O(n²)
for i in range(n):
    for j in range(n):
        pass

# Dependent inner loop → still O(n²): sum 0+1+2+...+(n-1) = n(n-1)/2
for i in range(n):
    for j in range(i):
        pass

# Harmonic series → O(n log n)
for i in range(1, n + 1):
    for j in range(0, n, i):  # n/i iterations per outer loop
        pass

# Two pointers → O(n): each pointer moves at most n times total
left, right = 0, n - 1
while left < right:
    left += 1  # or right -= 1

Key Takeaways#

  1. Big-O captures growth rate, not exact speed — O(n) with a large constant can be slower than O(n²) for small inputs.
  2. Drop constants and lower-order terms — 3n² + 7n + 42 is simply O(n²).
  3. Worst case is the default guarantee — unless stated otherwise, complexities refer to worst case.
  4. Space matters too — an O(n log n) sort that uses O(n) extra space may be worse than an O(n log n) in-place sort for your use case.
  5. Amortised ≠ average — amortised is a guarantee over sequences; average depends on input distribution.
  6. Measure real performance — cache locality, branch prediction, and constant factors matter in practice.
  7. Know the limits — if n = 10⁶, you need O(n log n) or better. This guides algorithm selection.