CI Fundamentals#

Continuous Integration (CI) is the practice of merging all developer working copies to a shared mainline frequently — ideally multiple times per day — and verifying each integration with an automated build and test suite.

What CI Solves#

Without CI, teams encounter these problems:

The CI Contract#

Every team practising CI agrees to:

1. Commit frequently — at least once per day to the mainline (or feature branch)
2. Don’t break the build — if CI fails, fixing it is the top priority
3. Keep the build fast — under 10 minutes is the target
4. Test in a clone of production — CI environment should mirror prod

Pipeline Anatomy#

A CI pipeline is a series of automated stages that code passes through from commit to verified artifact:

graph LR
    A[Trigger] --> B[Checkout]
    B --> C[Install Deps]
    C --> D[Lint & Format]
    D --> E[Build]
    E --> F[Unit Tests]
    F --> G[Integration Tests]
    G --> H[Security Scan]
    H --> I[Artifact]
    
    style A fill:#e3f2fd
    style D fill:#fff3cd
    style F fill:#e8f5e1
    style H fill:#ffebee
    style I fill:#f3e5f5

Stages Breakdown#

Triggers#

Triggers determine when a pipeline runs:

Trigger Strategy#

# Example: Run different stages based on trigger
stages:
  - lint        # Always
  - test        # Always
  - build       # On main or tags
  - deploy-test # On main
  - deploy-prod # On tags only

Build Stage#

The build stage compiles source code and produces deployable artifacts:

Common Build Outputs#

Build Best Practices#

• Reproducible builds — Same commit always produces same output
• No side effects — Build doesn’t modify external state
• Minimal image — Multi-stage Docker builds to reduce image size
• Version stamping — Embed git SHA or tag in the artifact

# Multi-stage build example
FROM node:20-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

FROM node:20-alpine
WORKDIR /app
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
CMD ["node", "dist/server.js"]

Test Stage#

Test Pyramid#

graph TD
    A["E2E Tests (few, slow, expensive)"] 
    B["Integration Tests (moderate)"]
    C["Unit Tests (many, fast, cheap)"]
    
    style A fill:#ffebee
    style B fill:#fff3cd
    style C fill:#e8f5e1

Test Configuration Example#

# pytest configuration for CI
[tool.pytest.ini_options]
testpaths = ["tests"]
markers = [
    "unit: Unit tests (fast, no external deps)",
    "integration: Integration tests (require services)",
    "slow: Long-running tests (nightly only)",
]
addopts = "--cov=src --cov-report=xml --cov-fail-under=80"

Lint Stage#

Linting catches issues without executing code:

Formatting vs Linting#

Parallelism#

Running stages and jobs in parallel dramatically reduces pipeline duration:

graph LR
    subgraph Sequential ["Sequential: 15 min"]
        A1[Lint 3min] --> B1[Unit 5min] --> C1[Integration 7min]
    end
    
    subgraph Parallel ["Parallel: 7 min"]
        A2[Lint 3min]
        B2[Unit 5min]
        C2[Integration 7min]
    end

Parallelism Strategies#

Test Splitting Example#

# Split tests across 4 parallel workers
test:
  parallel: 4
  script:
    - split-tests --total $CI_NODE_TOTAL --index $CI_NODE_INDEX | xargs pytest

Caching#

Caching avoids re-downloading or re-computing unchanged artifacts:

Cache Configuration Example#

# GitLab CI cache example
cache:
  key:
    files:
      - package-lock.json
  paths:
    - node_modules/
  policy: pull-push  # pull on start, push on success

Cache vs Artifacts#

Fail-Fast Principle#

The fail-fast principle means detecting failures as early and cheaply as possible:

Ordering for Speed#

graph LR
    A["Lint (30s)"] --> B["Unit Tests (2min)"] --> C["Build (3min)"] --> D["Integration (5min)"] --> E["E2E (10min)"]
    
    style A fill:#e8f5e1
    style B fill:#e8f5e1
    style C fill:#fff3cd
    style D fill:#fff3e0
    style E fill:#ffebee

Pre-commit Hooks#

Catch issues before they even reach CI:

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.5.0
    hooks:
      - id: trailing-whitespace
      - id: end-of-file-fixer
      - id: check-yaml
  - repo: https://github.com/psf/black
    rev: 24.3.0
    hooks:
      - id: black
  - repo: https://github.com/PyCQA/flake8
    rev: 7.0.0
    hooks:
      - id: flake8

CI Pipeline Design Patterns#

Conditional Pipeline Example#

# Only run frontend tests if frontend files changed
frontend-tests:
  rules:
    - changes:
        - "frontend/**/*"
        - "package.json"
  script:
    - cd frontend && npm test

# Only run backend tests if backend files changed
backend-tests:
  rules:
    - changes:
        - "backend/**/*"
        - "requirements.txt"
  script:
    - cd backend && pytest

Metrics for a Healthy CI#

Key Takeaways#

• CI is a practice (frequent integration) enabled by automation (pipelines)
• A pipeline should give feedback in under 10 minutes — optimise relentlessly
• Fail fast: cheapest checks first, provide clear error messages
• Caching and parallelism are your primary tools for pipeline speed
• The test pyramid guides what to test in CI vs other environments
• Every pipeline should be reproducible — same commit, same result
• Flaky tests undermine trust in CI — track and fix them aggressively