DevOps & CI/CD on Software Engineering Learning Paths

DevOps Principles

Mon, 01 Jan 0001 00:00:00 +0000

DevOps Principles#

DevOps is not a tool, a job title, or a team — it is a set of practices and cultural philosophies that unify software development (Dev) and IT operations (Ops). The goal is to shorten the systems development life cycle while delivering features, fixes, and updates frequently and reliably.

The Three Ways of DevOps#

The foundational principles come from The Phoenix Project and The DevOps Handbook:

Way	Principle	Practice
First Way	Systems thinking (flow)	Optimise left-to-right flow from Dev to Ops to customer. Reduce batch sizes, limit WIP, eliminate waste.
Second Way	Amplify feedback loops	Fast feedback from right to left. Telemetry, monitoring, peer review, testing in production.
Third Way	Continual experimentation and learning	Foster a culture of experimentation, risk-taking, and learning from failure. Blameless postmortems.

DevOps Culture — CALMS#

The CALMS framework describes the cultural pillars of DevOps:

CI Fundamentals

Mon, 01 Jan 0001 00:00:00 +0000

CI Fundamentals#

Continuous Integration (CI) is the practice of merging all developer working copies to a shared mainline frequently — ideally multiple times per day — and verifying each integration with an automated build and test suite.

What CI Solves#

Without CI, teams encounter these problems:

Problem	Symptom	How CI Solves It
Integration hell	Merging weeks of work causes conflicts and breakage	Small, frequent merges reduce conflict surface
“Works on my machine”	Code passes locally but fails elsewhere	Consistent build environment for every commit
Late bug discovery	Bugs found in QA weeks after introduction	Automated tests catch regressions immediately
Slow feedback	Developers wait days for test results	Pipeline runs in minutes on every push
Quality drift	Standards erode without enforcement	Linting and formatting enforced automatically
Release anxiety	Nobody knows if the codebase is deployable	The build is always green (or immediately fixed)

The CI Contract#

Every team practising CI agrees to:

CD & Deployment Strategies

Mon, 01 Jan 0001 00:00:00 +0000

CD & Deployment Strategies#

Continuous Delivery (CD) ensures that code is always in a deployable state. Continuous Deployment takes this further by automatically releasing every change that passes the pipeline. The deployment strategy you choose determines how risk is managed during releases.

Continuous Delivery vs Continuous Deployment#

Aspect	Continuous Delivery	Continuous Deployment
Definition	Every commit can be deployed to production	Every commit is deployed to production
Human gate	Manual approval before prod deploy	No manual gate — fully automated
Prerequisite	Comprehensive automated testing	Extreme confidence in test suite + monitoring
Rollback	Manual or semi-automated	Automated rollback on failure
Adoption	Widespread	Mostly tech-forward companies

graph LR
 A[Commit] --> B[Build & Test]
 B --> C[Deploy to Staging]
 C --> D{Manual Approval?}
 D -->|Continuous Delivery| E[Deploy to Prod]
 D -->|Continuous Deployment| E
 
 style D fill:#fff3cd

Deployment Strategies#

Blue-Green Deployment#

Maintain two identical production environments. Route traffic entirely to one (blue) while deploying to the other (green). Switch when ready.

GitLab CI

Mon, 01 Jan 0001 00:00:00 +0000

GitLab CI#

GitLab CI/CD is GitLab’s built-in continuous integration and delivery system. Configuration lives in a .gitlab-ci.yml file at the repository root, defining pipelines as code that execute on every push, merge request, or schedule.

.gitlab-ci.yml Structure#

A minimal pipeline:

stages:
 - build
 - test
 - deploy

build-app:
 stage: build
 image: node:20-alpine
 script:
 - npm ci
 - npm run build
 artifacts:
 paths:
 - dist/

test-app:
 stage: test
 image: node:20-alpine
 script:
 - npm ci
 - npm test
 coverage: '/Lines\s*:\s*(\d+\.?\d*)%/'

deploy-staging:
 stage: deploy
 script:
 - deploy-to-staging.sh
 environment:
 name: staging
 url: https://staging.example.com
 rules:
 - if: $CI_COMMIT_BRANCH == "main"

Top-Level Keywords#

Keyword	Purpose	Example
`stages`	Define pipeline stages and their order	`[lint, test, build, deploy]`
`variables`	Global environment variables	`NODE_ENV: production`
`default`	Default values for all jobs	`image`, `before_script`, `tags`
`include`	Import external CI configuration	Templates, shared pipelines
`workflow`	Control when entire pipeline runs	Filter by branch, variable

Stages, Jobs, and Artifacts#

Stages#

Stages run sequentially. All jobs within a stage run in parallel by default:

GitHub Actions

Mon, 01 Jan 0001 00:00:00 +0000

GitHub Actions#

GitHub Actions is GitHub’s native CI/CD platform. Workflows are defined in YAML files under .github/workflows/ and triggered by repository events (push, pull request, schedule, and more). Its marketplace of reusable actions and tight GitHub integration make it one of the most popular CI/CD systems.

Workflow Files#

Every workflow lives in .github/workflows/:

# .github/workflows/ci.yml
name: CI Pipeline

on:
 push:
 branches: [main]
 pull_request:
 branches: [main]

permissions:
 contents: read
 checks: write

jobs:
 lint:
 runs-on: ubuntu-latest
 steps:
 - uses: actions/checkout@v4
 - uses: actions/setup-node@v4
 with:
 node-version: 20
 cache: 'npm'
 - run: npm ci
 - run: npm run lint

 test:
 runs-on: ubuntu-latest
 needs: lint
 steps:
 - uses: actions/checkout@v4
 - uses: actions/setup-node@v4
 with:
 node-version: 20
 cache: 'npm'
 - run: npm ci
 - run: npm test -- --coverage
 - uses: actions/upload-artifact@v4
 with:
 name: coverage
 path: coverage/

Workflow Structure#

graph TD
 A[Workflow File] --> B[Events/Triggers]
 A --> C[Permissions]
 A --> D[Environment Variables]
 A --> E[Jobs]
 E --> F[Job 1]
 E --> G[Job 2]
 F --> H[Steps]
 H --> I[Action Step]
 H --> J[Run Step]

Events and Triggers#

Event	Triggers On	Common Use
`push`	Commit pushed to branch/tag	CI on every commit
`pull_request`	PR opened, updated, reopened	Gate before merge
`schedule`	Cron expression	Nightly builds, dependency checks
`workflow_dispatch`	Manual trigger (UI or API)	Ad-hoc runs, prod deploys
`release`	Release published	Build release artifacts
`repository_dispatch`	External webhook	Cross-repo triggers
`workflow_call`	Called by another workflow	Reusable workflows
`merge_group`	Merge queue	Validate before auto-merge

Event Filters#

on:
 push:
 branches:
 - main
 - 'release/**'
 tags:
 - 'v*'
 paths:
 - 'src/**'
 - 'package.json'
 paths-ignore:
 - 'docs/**'
 - '**.md'

 pull_request:
 types: [opened, synchronize, reopened]
 branches: [main]

 schedule:
 - cron: '0 2 * * 1-5' # Weekdays at 2 AM UTC

 workflow_dispatch:
 inputs:
 environment:
 description: 'Deploy target'
 required: true
 type: choice
 options: [staging, production]
 dry_run:
 description: 'Dry run only'
 type: boolean
 default: true

Jobs and Steps#

Job Configuration#

jobs:
 build:
 name: "Build Application"
 runs-on: ubuntu-latest  # Runner environment
 timeout-minutes: 15 # Maximum duration
 continue-on-error: false # Fail workflow on job failure
 concurrency: # Prevent parallel runs
 group: build-${{ github.ref }}
 cancel-in-progress: true
 
 env: # Job-level env vars
 NODE_ENV: production
 
 services: # Sidecar containers
 postgres:
 image: postgres:16
 env:
 POSTGRES_PASSWORD: test
 ports:
 - 5432:5432
 options: >-
 --health-cmd pg_isready
 --health-interval 10s
 --health-timeout 5s
 --health-retries 5
 
 steps:
 - uses: actions/checkout@v4
 - run: echo "Building..."

Step Types#

Type	Syntax	Purpose
Action	`uses: owner/repo@ref`	Run a reusable action
Run	`run: command`	Execute shell commands
Composite	Action with multiple steps	Bundle common step patterns

Step Features#

steps:
 - name: "Conditional step"
 if: github.event_name == 'push'
 run: echo "This is a push event"

 - name: "Step with outputs"
 id: version
 run: echo "tag=v1.2.3" >> $GITHUB_OUTPUT

 - name: "Use output from previous step"
 run: echo "Version is ${{ steps.version.outputs.tag }}"

 - name: "Continue on failure"
 continue-on-error: true
 run: npm run optional-check

 - name: "Set environment variable"
 run: echo "BUILD_DATE=$(date -u +%Y%m%d)" >> $GITHUB_ENV

 - name: "Multi-line script"
 run: |
 echo "Line 1"
 echo "Line 2"
 if [ "${{ github.ref }}" = "refs/heads/main" ]; then
 echo "On main branch"
 fi
 shell: bash

Actions Marketplace#

The GitHub Marketplace provides thousands of reusable actions:

Artifact Management

Mon, 01 Jan 0001 00:00:00 +0000

Artifact Management#

Artifacts — container images, packages, binaries — are the output of your CI pipeline and the input to your CD pipeline. Managing them well means consistent tagging, secure storage, vulnerability scanning, and retention policies.

Container Registries#

A container registry stores and distributes Docker/OCI images. Choose based on your ecosystem:

Registry	Best For	Authentication	Key Features
Amazon ECR	AWS-native workloads	IAM roles	Lifecycle policies, image scanning, cross-region replication
Docker Hub	Open-source, public images	Username/token	Official images, automated builds, rate limits on free tier
GitLab Container Registry	GitLab CI users	CI_JOB_TOKEN	Built-in, zero config, per-project isolation
GitHub Container Registry (ghcr.io)	GitHub Actions users	GITHUB_TOKEN	Free for public images, org-level packages
Google Artifact Registry	GCP workloads	Service accounts	Multi-format (Docker, npm, Maven, Python)
Azure Container Registry	Azure/AKS users	Managed Identity	Geo-replication, ACR Tasks for in-registry builds

ECR Lifecycle Policy Example#

Automatically clean old images to control storage costs:

Infrastructure Automation

Mon, 01 Jan 0001 00:00:00 +0000

Infrastructure Automation#

Infrastructure automation brings the same rigour to infrastructure that CI/CD brings to application code: version control, peer review, automated testing, and repeatable deployments. GitOps, Terraform in pipelines, and policy-as-code are the pillars.

GitOps#

GitOps uses Git as the single source of truth for infrastructure and application state. A controller running in the cluster continuously reconciles the live state against what’s declared in Git.

Core Principles#

Principle	Meaning
Declarative	Desired state is described, not scripted
Versioned and immutable	Git history is the audit trail
Pulled automatically	Controller pulls from Git, not pushed by CI
Continuously reconciled	Drift is detected and corrected automatically

Push vs Pull Deployment#

Push (traditional CI/CD):
 CI Pipeline ──push──▶ kubectl apply / helm upgrade ──▶ Cluster

Pull (GitOps):
 Developer ──commit──▶ Git Repo ◀──poll/watch── Controller ──reconcile──▶ Cluster

ArgoCD#

The most popular GitOps controller for Kubernetes:

Observability in CI/CD

Mon, 01 Jan 0001 00:00:00 +0000

Observability in CI/CD#

You can’t improve what you don’t measure. Observability applied to CI/CD pipelines means tracking DORA metrics, understanding where time is spent, catching regressions early, and verifying deployments automatically.

DORA Metrics#

The four key metrics from the DevOps Research and Assessment (DORA) team that predict software delivery performance:

Metric	Definition	Elite	High	Medium	Low
Deployment Frequency	How often code reaches production	On-demand (multiple/day)	Weekly to monthly	Monthly to 6-monthly	Fewer than once per 6 months
Lead Time for Changes	Commit to production	< 1 hour	1 day to 1 week	1 week to 1 month	> 6 months
Change Failure Rate	% of deployments causing failure	0-15%	16-30%	16-30%	> 30%
Mean Time to Recovery (MTTR)	Time to restore service after failure	< 1 hour	< 1 day	1 day to 1 week	> 6 months

Measuring Deployment Frequency#

# GitLab CI: emit deployment event to Datadog
deploy_prod:
 stage: deploy
 script:
 - ./deploy.sh
 - |
 curl -X POST "https://api.datadoghq.eu/api/v1/events" \
 -H "DD-API-KEY: ${DD_API_KEY}" \
 -d @- <<EOF
 {
 "title": "Production Deployment",
 "text": "Deployed ${CI_PROJECT_NAME} ${CI_COMMIT_TAG}",
 "tags": ["service:${CI_PROJECT_NAME}", "env:prod", "team:${TEAM}"],
 "alert_type": "success"
 }
 EOF

Measuring Lead Time#

Lead time = time from first commit on branch to production deployment: