Calculus Essentials#

Calculus is the mathematics of change and accumulation. This section covers the minimum viable calculus for engineers — enough to understand gradient descent, optimisation, and the mathematical foundations of machine learning without a full analysis course.

Limits#

A limit describes the value a function approaches as its input approaches some point.

lim (x → a) f(x) = L

Means: as x gets arbitrarily close to a, f(x) gets arbitrarily close to L.

Key Limits#

Continuity#

A function is continuous at a point if the limit equals the function value:

lim (x → a) f(x) = f(a)

Discontinuities matter in computing: step functions (ReLU at 0), integer rounding, and hash bucket boundaries are all discontinuous.

Derivatives#

The derivative of a function measures its instantaneous rate of change:

f'(x) = lim (h → 0) [f(x + h) - f(x)] / h

Geometrically: the slope of the tangent line at point x.

Derivative Rules#

Common Derivatives#

The Chain Rule — The Most Important Rule#

The chain rule computes derivatives of composed functions:

d/dx [f(g(x))] = f'(g(x)) · g'(x)

This is exactly what backpropagation in neural networks computes. A neural network is a composition of functions:

output = f₃(f₂(f₁(input)))

To compute how the output changes with respect to the input (or any weight), the chain rule decomposes it into a product of local derivatives — one per layer.

Optimisation#

Finding Minima and Maxima#

At a local minimum or maximum, the derivative is zero:

f'(x) = 0  →  x is a critical point

To classify:

• f’’(x) > 0 → local minimum (concave up)
• f’’(x) < 0 → local maximum (concave down)
• f’’(x) = 0 → inconclusive (test further)

Gradient Descent#

The core optimisation algorithm in machine learning. Given a loss function L(θ), update parameters to reduce the loss:

θ_new = θ_old - α × ∇L(θ_old)

Where:

• α is the learning rate (step size)
• ∇L is the gradient (vector of partial derivatives)

Intuition: The gradient points “uphill.” Subtracting it moves you “downhill” toward a minimum.

Loss landscape (1D):

      ╱╲
     ╱  ╲        ╱╲
    ╱    ╲      ╱  ╲
   ╱      ╲    ╱    ╲
  ╱        ╲  ╱      ╲
 ╱          ╲╱        ╲
             ← gradient descent moves toward valley

Learning Rate Effects#

Variants#

Partial Derivatives#

When a function has multiple variables, the partial derivative measures the rate of change with respect to one variable while holding others constant.

f(x, y) = x² + 3xy + y²

∂f/∂x = 2x + 3y    (treat y as constant)
∂f/∂y = 3x + 2y    (treat x as constant)

The Gradient#

The gradient is the vector of all partial derivatives:

∇f = [∂f/∂x, ∂f/∂y]

For f(x, y) = x² + 3xy + y²:

∇f = [2x + 3y, 3x + 2y]

The gradient points in the direction of steepest ascent. Its magnitude tells you how steep the slope is.

In Neural Networks#

A neural network with millions of parameters θ = [θ₁, θ₂, …, θₙ] has a loss function L(θ). The gradient:

∇L = [∂L/∂θ₁, ∂L/∂θ₂, ..., ∂L/∂θₙ]

Backpropagation computes this gradient efficiently using the chain rule, layer by layer, in O(n) time.

Integrals#

The integral is the reverse of differentiation — it computes accumulated area under a curve.

Definite Integral#

∫ₐᵇ f(x) dx = area under f(x) from a to b

Fundamental Theorem of Calculus#

If F’(x) = f(x), then:

∫ₐᵇ f(x) dx = F(b) - F(a)

The integral of a rate of change gives the total change.

Common Integrals#

Engineering Applications of Integration#

Numerical Integration#

When analytical integration is impossible (most real functions), approximate numerically:

Monte Carlo integration scales well to high dimensions where grid methods fail (the “curse of dimensionality”).

Convexity#

A function is convex if the line segment between any two points on the graph lies above the graph:

f(λx + (1-λ)y) ≤ λf(x) + (1-λ)f(y)   for 0 ≤ λ ≤ 1

Why Convexity Matters#

Convex loss functions: MSE (mean squared error), cross-entropy (for logistic regression), SVMs. These are well-behaved for optimisation.

Non-convex loss functions: Neural network losses. Gradient descent finds “good” local minima but not guaranteed global. In practice, the high-dimensional loss landscape has many equally good minima.

Jensen’s Inequality#

For a convex function f:

f(E[X]) ≤ E[f(X)]

Application: This is why the log-likelihood is used instead of the likelihood in ML — it is numerically more stable and the inequality ensures optimisation still works.

Taylor Series#

Any smooth function can be approximated as a polynomial around a point a:

f(x) = f(a) + f'(a)(x-a) + f''(a)(x-a)²/2! + f'''(a)(x-a)³/3! + ...

Common Taylor Series (around a = 0)#

Why Taylor Series Matter#

1. Linearisation: For small x, eˣ ≈ 1 + x. This approximation is used throughout physics and engineering.
2. Numerical methods: Functions are computed on CPUs using truncated Taylor series (CORDIC, minimax polynomials).
3. Error analysis: The remainder term tells you how accurate your approximation is.
4. Softmax stability: The softmax trick (subtract max) works because of the properties of the exponential Taylor series.

Key Takeaways#

• The derivative measures instantaneous rate of change. The chain rule decomposes derivatives of composed functions — this IS backpropagation.
• Gradient descent iteratively moves toward a minimum by following the negative gradient. The learning rate controls step size.
• Partial derivatives let you compute how a function changes with respect to each variable independently. The gradient collects all partial derivatives into a vector.
• Integrals compute accumulated quantities (area, probability, total work). Monte Carlo integration scales to high dimensions.
• Convex functions have no local minima traps — any local minimum is global. Neural network losses are non-convex but behave well in practice.
• You do not need to be fluent in calculus to use ML — but understanding the chain rule, gradient descent, and convexity gives you the intuition to debug training failures and choose appropriate architectures.