Machine Learning on Software Engineering Learning Paths

Introduction to Machine Learning

Mon, 01 Jan 0001 00:00:00 +0000

Introduction to Machine Learning#

Machine learning is the study of algorithms that improve their performance on a task through experience (data), without being explicitly programmed for that task. This section establishes the taxonomy, core tradeoffs, and workflow that every subsequent section builds upon.

What Is Machine Learning?#

Traditional programming: you write rules, the program applies them to data and produces output.

Machine learning: you provide data and the desired output, and the algorithm learns the rules.

Data Preparation

Mon, 01 Jan 0001 00:00:00 +0000

Data Preparation#

Data preparation consumes 60–80% of a typical ML project’s time. A model trained on poorly prepared data will produce poor results regardless of how sophisticated the algorithm is. This section covers the essential transformations between raw data and model-ready features.

Exploratory Data Analysis (EDA)#

Before any transformation, understand your data:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

df = pd.read_csv("housing.csv")

# Shape, types, memory
print(df.shape)
print(df.dtypes)
print(df.describe())

# Missing values
print(df.isnull().sum().sort_values(ascending=False))

# Distribution of target
df["price"].hist(bins=50)
plt.xlabel("Price")
plt.ylabel("Count")
plt.title("Target Distribution")
plt.show()

# Correlation matrix
corr = df.select_dtypes(include=[np.number]).corr()
sns.heatmap(corr, annot=True, fmt=".2f", cmap="coolwarm", center=0)
plt.title("Feature Correlations")
plt.tight_layout()
plt.show()

EDA Checklist#

Check	What to Look For
Shape	Number of rows and features
Types	Numeric vs categorical vs datetime
Missing values	Percentage per column, patterns (MCAR/MAR/MNAR)
Distributions	Skew, outliers, multimodality
Target balance	Class imbalance for classification
Correlations	Redundant features, leakage signals
Duplicates	Exact or near-duplicate rows

Handling Missing Data#

Missing data is nearly universal. The strategy depends on why the data is missing.

Linear Models

Mon, 01 Jan 0001 00:00:00 +0000

Linear Models#

Linear models are the foundation of machine learning. They are fast, interpretable, and surprisingly effective. Every ML practitioner should understand them thoroughly — both because they serve as strong baselines and because the concepts (gradient descent, regularisation, loss functions) transfer to every other algorithm.

Linear Regression#

The simplest supervised model. It fits a linear function to predict a continuous target:

ŷ = w₁x₁ + w₂x₂ + ... + wₙxₙ + b

Where w are the weights (coefficients), b is the bias (intercept), and ŷ is the prediction.

Decision Trees & Ensembles

Mon, 01 Jan 0001 00:00:00 +0000

Decision Trees & Ensembles#

Decision trees are the foundation of the most powerful classical ML algorithms. A single tree is intuitive and interpretable but prone to overfitting. Ensemble methods — random forests and gradient boosting — combine many trees to achieve state-of-the-art performance on tabular data.

Decision Trees#

A decision tree recursively splits the feature space into regions, making predictions based on the majority class (classification) or mean value (regression) in each region.

Support Vector Machines

Mon, 01 Jan 0001 00:00:00 +0000

Support Vector Machines#

Support Vector Machines (SVMs) find the optimal separating hyperplane between classes by maximising the margin — the distance between the decision boundary and the nearest data points. With the kernel trick, SVMs can learn non-linear boundaries while working in high-dimensional space efficiently.

Linear SVM#

The Maximum Margin Classifier#

Given two linearly separable classes, infinitely many hyperplanes can separate them. SVM chooses the one with the largest margin — the greatest distance to the nearest training points from either class.

Unsupervised Learning

Mon, 01 Jan 0001 00:00:00 +0000

Unsupervised Learning#

Unsupervised learning finds structure in data without labels. Clustering groups similar points together, dimensionality reduction compresses features while preserving structure, and anomaly detection identifies outliers. These techniques are essential for exploration, preprocessing, and problems where labels are expensive or unavailable.

Clustering#

K-Means#

K-Means partitions data into K clusters by iteratively assigning points to the nearest centroid and updating centroids.

Algorithm:

Initialise K centroids (k-means++ for smart initialisation)
Assign each point to the nearest centroid
Recompute centroids as the mean of assigned points
Repeat until convergence

from sklearn.cluster import KMeans
from sklearn.datasets import make_blobs
import matplotlib.pyplot as plt
import numpy as np

X, y_true = make_blobs(n_samples=300, centers=4, cluster_std=0.8, random_state=42)

kmeans = KMeans(n_clusters=4, init="k-means++", n_init=10, random_state=42)
labels = kmeans.fit_predict(X)

plt.scatter(X[:, 0], X[:, 1], c=labels, cmap="viridis", s=20, alpha=0.7)
plt.scatter(kmeans.cluster_centers_[:, 0], kmeans.cluster_centers_[:, 1],
 c="red", marker="X", s=200, edgecolors="black")
plt.title("K-Means Clustering")
plt.show()

Choosing K: The Elbow Method and Silhouette Score#

from sklearn.metrics import silhouette_score

inertias = []
silhouettes = []
K_range = range(2, 10)

for k in K_range:
 km = KMeans(n_clusters=k, n_init=10, random_state=42)
 km.fit(X)
 inertias.append(km.inertia_)
 silhouettes.append(silhouette_score(X, km.labels_))

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4))
ax1.plot(K_range, inertias, "bo-")
ax1.set_xlabel("K")
ax1.set_ylabel("Inertia")
ax1.set_title("Elbow Method")

ax2.plot(K_range, silhouettes, "ro-")
ax2.set_xlabel("K")
ax2.set_ylabel("Silhouette Score")
ax2.set_title("Silhouette Analysis")
plt.tight_layout()
plt.show()

Method	What It Measures	Choose K Where
Elbow	Within-cluster sum of squares (inertia)	Curve “bends” (diminishing returns)
Silhouette	Cohesion vs separation (-1 to 1)	Maximum score
Gap Statistic	Compares inertia to random reference	Largest gap

K-Means Limitations#

Assumes spherical clusters of similar size
Sensitive to initialisation (mitigated by n_init)
Requires K to be specified in advance
Cannot detect non-convex clusters

DBSCAN#

Density-Based Spatial Clustering of Applications with Noise. Finds clusters of arbitrary shape and identifies noise points.

Model Evaluation

Mon, 01 Jan 0001 00:00:00 +0000

Model Evaluation#

A model is only as trustworthy as the method used to evaluate it. This section covers the metrics, validation strategies, and diagnostic tools that tell you whether your model genuinely works — or just memorised the training data.

Classification Metrics#

Confusion Matrix#

The confusion matrix is the foundation of all classification metrics:

 Predicted
 Pos Neg
Actual Pos [ TP | FN ]
 Neg [ FP | TN ]

Term	Meaning
TP (True Positive)	Correctly predicted positive
TN (True Negative)	Correctly predicted negative
FP (False Positive)	Incorrectly predicted positive (Type I error)
FN (False Negative)	Incorrectly predicted negative (Type II error)

from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model = LogisticRegression(max_iter=5000, random_state=42)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

cm = confusion_matrix(y_test, y_pred)
ConfusionMatrixDisplay(cm, display_labels=["Malignant", "Benign"]).plot(cmap="Blues")

Core Metrics#

Metric	Formula	When to Use
Accuracy	(TP+TN) / (TP+TN+FP+FN)	Balanced classes only
Precision	TP / (TP+FP)	Cost of false positives is high (spam filter)
Recall (Sensitivity)	TP / (TP+FN)	Cost of false negatives is high (disease detection)
F1 Score	2 × (Precision × Recall) / (Precision + Recall)	Need balance between precision and recall
Specificity	TN / (TN+FP)	True negative rate matters

from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score

print(f"Accuracy: {accuracy_score(y_test, y_pred):.4f}")
print(f"Precision: {precision_score(y_test, y_pred):.4f}")
print(f"Recall: {recall_score(y_test, y_pred):.4f}")
print(f"F1: {f1_score(y_test, y_pred):.4f}")

Why Accuracy Fails on Imbalanced Data#

If 95% of emails are not spam, a model that always predicts “not spam” achieves 95% accuracy while catching zero spam. Accuracy is misleading when classes are imbalanced.

Hyperparameter Tuning

Mon, 01 Jan 0001 00:00:00 +0000

Hyperparameter Tuning#

Hyperparameters are settings that control the learning process — they are not learned from data. Choosing the right hyperparameters can be the difference between a mediocre model and a production-ready one. This section covers systematic approaches to finding optimal configurations.

Parameters vs Hyperparameters#

Type	Set By	Examples
Parameters	Learned from data during training	Linear regression weights, tree split thresholds
Hyperparameters	Set before training begins	Learning rate, max_depth, C, n_estimators, kernel

Parameters are optimised by the algorithm (e.g., gradient descent minimises the loss). Hyperparameters are optimised by you — using the techniques in this section.

Feature Selection & Engineering

Mon, 01 Jan 0001 00:00:00 +0000

Feature Selection & Engineering#

Features are the raw material of machine learning. Good features make simple models powerful; poor features make complex models weak. This section covers systematic methods for selecting the most informative features and engineering new ones from raw data.

Why Feature Selection Matters#

Benefit	Explanation
Reduced overfitting	Fewer irrelevant features means less noise for the model to memorise
Faster training	Fewer features → smaller matrices → faster computation
Better interpretability	Easier to explain a model with 10 features than 500
Improved accuracy	Removing noise features can improve generalisation

The challenge: with n features, there are 2ⁿ possible subsets. Exhaustive search is intractable for any non-trivial feature count.

Time Series

Mon, 01 Jan 0001 00:00:00 +0000

Time Series#

Time series data has a temporal ordering that standard ML methods ignore. Forecasting future values, detecting seasonality, and engineering lag features require specialised techniques. This section covers classical time series analysis and how to apply ML models to temporal data correctly.

Time Series Fundamentals#

A time series is a sequence of observations ordered by time: y₁, y₂, ..., yₜ.

Components#

Every time series can be decomposed into:

Component	Description	Example
Trend	Long-term increase or decrease	Growing user base
Seasonality	Repeating patterns at fixed intervals	Higher sales in December
Cyclical	Repeating patterns not at fixed intervals	Economic cycles
Residual	Random noise after removing trend and seasonality	Unpredictable variation

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from statsmodels.tsa.seasonal import seasonal_decompose

# Generate example time series
np.random.seed(42)
dates = pd.date_range("2020-01-01", periods=365 * 3, freq="D")
trend = np.linspace(100, 200, len(dates))
seasonal = 20 * np.sin(2 * np.pi * np.arange(len(dates)) / 365)
noise = np.random.normal(0, 5, len(dates))
y = trend + seasonal + noise

ts = pd.Series(y, index=dates)

# Decompose
result = seasonal_decompose(ts, model="additive", period=365)
result.plot()
plt.tight_layout()
plt.show()

Stationarity#

A time series is stationary if its statistical properties (mean, variance, autocorrelation) do not change over time. Most classical forecasting methods require stationarity.

ML Pipelines & scikit-learn

Mon, 01 Jan 0001 00:00:00 +0000

ML Pipelines & scikit-learn#

Production ML code is not a sequence of Jupyter cells — it is a reproducible pipeline that transforms raw data into predictions without manual intervention. Scikit-learn’s Pipeline and ColumnTransformer are the building blocks for clean, leak-free, serialisable ML workflows.

Why Pipelines Matter#

Without pipelines, preprocessing and modelling are separate steps prone to errors:

Problem	Consequence
Fitting scaler on full data before splitting	Data leakage
Forgetting a transform step in production	Wrong predictions
Different code paths for training and inference	Silent bugs
Manual step ordering	Impossible to reproduce

A Pipeline chains transforms and a final estimator into a single object that implements fit(), predict(), and score().

MLOps & Deployment

Mon, 01 Jan 0001 00:00:00 +0000

MLOps & Deployment#

A model that exists only in a notebook delivers zero value. MLOps is the set of practices that takes a trained model from experimentation to production — covering experiment tracking, model versioning, serving, monitoring, and handling the inevitable data drift. This section bridges the gap between data science and production engineering.

MLOps Overview#

Stage	Concern	Tools
Experiment tracking	Log parameters, metrics, artefacts per run	MLflow, Weights & Biases, Neptune
Model versioning	Version control for models and datasets	MLflow Model Registry, DVC
Testing	Validate model behaviour before deployment	pytest, Great Expectations
Serving	Expose the model via an API	Flask, FastAPI, BentoML, SageMaker
Monitoring	Detect performance degradation in production	Evidently, Nannyml, custom dashboards
Retraining	Update the model when data changes	Scheduled pipelines, trigger-based

Experiment Tracking with MLflow#

MLflow tracks experiments — every training run logs its parameters, metrics, and output artefacts.