AI and Large Language Models on Software Engineering Learning Paths

Neural Networks and Deep Learning

Mon, 01 Jan 0001 00:00:00 +0000

This guide covers the foundational concepts of neural networks — from the single perceptron to deep architectures. By the end, you’ll understand how networks learn, why they work, and how to prevent them from failing.

Prerequisites#

Linear algebra basics (vectors, matrices, dot products)
Calculus (derivatives, chain rule)
Basic probability and statistics
Python familiarity

The Perceptron#

The perceptron is the simplest neural unit — a linear classifier inspired by biological neurons.

The Transformer Architecture

Mon, 01 Jan 0001 00:00:00 +0000

The Transformer is the architecture behind virtually all modern language models (GPT, BERT, Claude, Llama, T5). Introduced in “Attention Is All You Need” (Vaswani et al., 2017), it replaced recurrent networks by processing all positions in parallel through a mechanism called self-attention.

This guide breaks down every component — from the attention mechanism to the full architecture variants.

Prerequisites#

Neural network basics (forward/backward pass, loss functions)
Matrix multiplication
Basic understanding of sequence processing (what a “token” is)

The Core Idea: Attention#

The fundamental question attention answers: “When processing token X, how much should I look at every other token in the sequence?”

Language Models

Mon, 01 Jan 0001 00:00:00 +0000

This guide traces the evolution of language models — from counting word sequences to trillion-parameter neural networks that exhibit emergent reasoning. Understanding this history reveals why modern LLMs work the way they do and what trade-offs each generation made.

Prerequisites#

Basic probability (conditional probability, Bayes’ theorem)
Neural network fundamentals (layers, training, loss functions)
Transformer architecture basics (attention, encoder/decoder)

What Is a Language Model?#

A language model assigns probabilities to sequences of tokens:

Training and Fine-Tuning Language Models

Mon, 01 Jan 0001 00:00:00 +0000

This guide covers the full lifecycle of making a language model useful — from pre-training on raw text to aligning it with human preferences. You’ll understand when to fine-tune, when to prompt, and when to use RAG instead.

Prerequisites#

Transformer architecture (attention, decoder-only models)
Neural network training (loss functions, optimizers, backpropagation)
Basic understanding of what language models do (next-token prediction)

The Three Stages of LLM Development#

Stage 1: Pre-training → "Learn language and world knowledge"
Stage 2: Supervised Fine-tuning → "Learn to follow instructions"
Stage 3: Alignment (RLHF/DPO) → "Learn human preferences"

graph LR
 PT[Pre-training<br/>Trillions of tokens<br/>Next token prediction] --> SFT[Supervised Fine-tuning<br/>Thousands of examples<br/>Instruction-response pairs]
 SFT --> ALIGN[Alignment<br/>RLHF or DPO<br/>Human preferences]
 ALIGN --> DEPLOY[Deployed Model<br/>Helpful, harmless<br/>honest]

Stage 1: Pre-training#

Pre-training creates the foundation model — a model that understands language, facts, reasoning patterns, and code, but doesn’t yet know how to be a helpful assistant.

Prompt Engineering

Mon, 01 Jan 0001 00:00:00 +0000

Prompt engineering is the discipline of crafting inputs to large language models to elicit desired outputs. It ranges from simple instruction writing to sophisticated multi-step reasoning strategies. Mastering prompt engineering is the highest-leverage skill for working with LLMs — it requires no training data, no compute, and produces immediate results.

Prerequisites#

Understanding of how language models generate text (autoregressive, token-by-token)
Familiarity with the transformer architecture and attention mechanism
Basic understanding of tokenization

Prompting Fundamentals#

A prompt is the text input provided to an LLM. The model generates a completion conditioned on this input. The quality, specificity, and structure of the prompt directly determine the quality of the output.

Retrieval-Augmented Generation (RAG)

Mon, 01 Jan 0001 00:00:00 +0000

Retrieval-Augmented Generation combines the generative power of large language models with external knowledge retrieval. Instead of relying solely on what the model memorized during training, RAG fetches relevant documents at inference time and includes them in the prompt context. This grounds the model’s responses in factual, up-to-date information.

Prerequisites#

Understanding of how LLMs generate text and their context windows
Basic familiarity with embeddings and vector spaces
Knowledge of prompt engineering fundamentals

Why RAG?#

The Problem with Pure LLMs#

Large language models have fundamental limitations that RAG addresses:

Agents and Tool Use

Mon, 01 Jan 0001 00:00:00 +0000

AI agents are systems where a language model autonomously decides what actions to take, executes those actions via external tools, observes the results, and iterates until a goal is achieved. Unlike simple prompt-response interactions, agents maintain state, plan multi-step strategies, and interact with the real world through APIs, databases, file systems, and other software.

Prerequisites#

Understanding of LLM inference and prompt engineering
Familiarity with function calling and structured output
Basic knowledge of API design and software architecture

Function and Tool Calling#

Tool calling is the mechanism by which an LLM requests execution of external functions. The model doesn’t execute code — it generates structured requests that the application fulfills.

Inference and Deployment

Mon, 01 Jan 0001 00:00:00 +0000

Deploying large language models in production requires understanding the mechanics of text generation, memory constraints, optimization techniques, and the tradeoffs between latency, throughput, and cost. This chapter covers how LLMs generate text at the hardware level and how to serve them efficiently at scale.

Prerequisites#

Understanding of transformer architecture (attention, KV cache)
Basic knowledge of GPU computing and memory hierarchies
Familiarity with model parameters and floating-point representations

Inference Mechanics#

Autoregressive Generation#

LLMs generate text one token at a time. Each new token is conditioned on all previous tokens. This sequential dependency is the fundamental constraint of LLM inference.

Evaluation, Safety, and Alignment

Mon, 01 Jan 0001 00:00:00 +0000

Measuring model quality and ensuring safe, responsible AI deployment are two sides of the same coin. This chapter covers how we evaluate LLMs, detect and mitigate their failure modes, align them with human values, and build guardrails for production systems.

Benchmarks and Evaluation Frameworks#

Why Benchmarks Matter#

Benchmarks provide standardized, reproducible measurements of model capabilities. Without them, comparing models would be purely anecdotal. However, benchmarks are imperfect proxies — a model that scores well on a benchmark may still fail in your specific use case.

Multimodal Models

Mon, 01 Jan 0001 00:00:00 +0000

Multimodal AI systems process and generate content across multiple modalities — text, images, audio, and video. This chapter covers the architectures, capabilities, and practical applications of models that bridge these domains.

Vision-Language Models#

How Vision-Language Models Work#

Vision-language models (VLMs) combine a visual encoder with a language model, enabling understanding of images through natural language.

graph LR
 I[Image] --> VE[Vision Encoder<br/>ViT / SigLIP]
 VE --> P[Projection Layer<br/>Linear or MLP]
 P --> LLM[Language Model<br/>Decoder]
 T[Text Prompt] --> TOK[Tokenizer]
 TOK --> LLM
 LLM --> O[Text Output]

Architecture Patterns#

Model	Vision Encoder	Language Model	Connection Method
GPT-4V / GPT-4o	Proprietary	GPT-4	Native multimodal training
Claude 3.x Vision	Proprietary	Claude	Native multimodal training
LLaVA	CLIP ViT-L/14	Vicuna / LLaMA	Linear projection + instruction tuning
Gemini	Proprietary	Gemini	Native multimodal (all modalities)
Qwen-VL	ViT-bigG	Qwen	Cross-attention + compression
InternVL	InternViT-6B	InternLM	Dynamic resolution + pixel shuffle

Using Vision Models#

from openai import OpenAI
import base64

client = OpenAI()

def analyze_image(image_path: str, question: str) -> str:
 """Analyze an image with GPT-4o."""
 
 with open(image_path, "rb") as f:
 image_data = base64.standard_b64encode(f.read()).decode("utf-8")
 
 response = client.chat.completions.create(
 model="gpt-4o",
 messages=[{
 "role": "user",
 "content": [
 {"type": "text", "text": question},
 {"type": "image_url", "image_url": {
 "url": f"data:image/png;base64,{image_data}",
 "detail": "high" # "low", "high", or "auto"
 }}
 ]
 }],
 max_tokens=1000
 )
 return response.choices[0].message.content


def analyze_image_url(url: str, question: str) -> str:
 """Analyze an image from URL."""
 
 response = client.chat.completions.create(
 model="gpt-4o",
 messages=[{
 "role": "user",
 "content": [
 {"type": "text", "text": question},
 {"type": "image_url", "image_url": {"url": url}}
 ]
 }]
 )
 return response.choices[0].message.content

Vision Model Capabilities#

Capability	Description	Use Cases
Image description	Detailed natural language descriptions	Accessibility, cataloging
OCR / text extraction	Read text from images	Document processing, receipts
Visual reasoning	Answer questions requiring inference	Diagram understanding, charts
Object detection	Identify and locate objects	Inventory, safety monitoring
Spatial understanding	Understand layout and relationships	UI analysis, architecture
Multi-image comparison	Compare or relate multiple images	Before/after, product matching
Chart/graph reading	Extract data from visualizations	Report analysis, data extraction

Practical Considerations#

Token Cost for Images (GPT-4o):

Detail Level Tokens Used Best For
─────────────────────────────────────────────────
low 85 tokens Quick classification, simple questions
high 85-1105 tokens OCR, detailed analysis, small text
auto Model decides General use

High detail: image split into 512x512 tiles
Each tile = 170 tokens + 85 base tokens
Max: 2048x2048 before downscaling

LLaVA Architecture Deep Dive#

LLaVA (Large Language and Vision Assistant) demonstrates a clean, reproducible VLM architecture:

Building LLM Applications

Mon, 01 Jan 0001 00:00:00 +0000

Moving from prototype to production with LLMs requires engineering discipline around reliability, cost, observability, and testing. This chapter covers the patterns and practices for building robust LLM-powered applications.

API Integration Patterns#

Streaming Responses#

Streaming delivers tokens as they’re generated, reducing perceived latency from seconds to milliseconds for the first token:

from openai import OpenAI

client = OpenAI()

def stream_response(messages: list[dict]) -> str:
 """Stream a response, yielding tokens as they arrive."""
 
 stream = client.chat.completions.create(
 model="gpt-4o",
 messages=messages,
 stream=True
 )
 
 full_response = ""
 for chunk in stream:
 if chunk.choices[0].delta.content:
 token = chunk.choices[0].delta.content
 full_response += token
 print(token, end="", flush=True)
 
 return full_response

Server-Sent Events (SSE) for web applications:

AI-Assisted Development

Mon, 01 Jan 0001 00:00:00 +0000

AI coding assistants have become standard tools in software engineering. This chapter covers how to use them effectively, understand their limitations, and integrate them into development workflows without compromising code quality or security.

Code Generation Tools#

The Landscape#

Tool	Type	Model	Integration
GitHub Copilot	Inline completion + chat	GPT-4o / Claude	VS Code, JetBrains, Neovim
Cursor	AI-native IDE	Claude, GPT-4o, custom	Fork of VS Code
Claude (Anthropic)	Chat + artifacts + computer use	Claude 4 family	Web, API, IDE extensions
ChatGPT	Chat interface	GPT-4o	Web, API, plugins
Amazon Q Developer	Inline + chat + agents	Proprietary	VS Code, JetBrains, CLI
Codeium / Windsurf	Inline + chat	Proprietary	Multi-IDE
Aider	Terminal-based pair programming	Any (OpenAI, Anthropic, local)	CLI, git-integrated
Kiro	Spec-driven development	Claude	VS Code

How Code Completion Works#

Code Completion Pipeline:

1: Context Gathering
 ├── Current file (cursor position)
 ├── Open tabs / related files
 ├── Import statements (dependency hints)
 ├── Recent edits (intent signals)
 └── Project structure (file tree)

2: Prompt Construction
 ├── Prefix (code before cursor)
 ├── Suffix (code after cursor)
 └── Metadata (language, framework, file path)

3: Model Inference
 └── Generate completion tokens

4: Post-Processing
 ├── Filter low-confidence suggestions
 ├── Deduplicate
 ├── Format to match style
 └── Present to user (inline ghost text)

IDE Integration Patterns#

Pattern	Description	Example
Inline ghost text	Grayed-out suggestion at cursor	Copilot tab-to-accept
Chat panel	Conversational coding assistant	Cursor chat, Copilot chat
Inline edit	Select code → describe change	Cursor Cmd+K
Agent mode	Multi-file autonomous changes	Cursor composer, Copilot agent
Terminal assist	Command suggestions in terminal	Amazon Q, GitHub Copilot CLI

Effective Prompting for Code#

Principles#

Principle	Bad Example	Good Example
Be specific	“Make a function”	“Write a Python function that validates email addresses using regex, returning True/False”
Provide context	“Fix this bug”	“This function should return sorted results but returns unsorted when the input has duplicates”
Specify constraints	“Make it fast”	“Optimize for O(n log n) time complexity, input size up to 1M elements”
Include examples	“Parse dates”	“Parse dates like ‘2024-01-15’, ‘Jan 15, 2024’, ‘15/01/2024’ into ISO format”
State the stack	“Build an API”	“Build a REST endpoint using FastAPI with Pydantic validation and SQLAlchemy ORM”

Comment-Driven Development#

Write comments describing what you want, then let the AI generate the implementation:

Using AI Effectively

Mon, 01 Jan 0001 00:00:00 +0000

This chapter is about practical wisdom — how to get the most value from AI tools as an end user, whether you’re writing, researching, analyzing data, or building workflows. The goal is to develop intuition for when AI helps, when it misleads, and how to collaborate with it productively.

Choosing the Right Model#

Capability vs Cost Matrix#

Not every task needs the most powerful model. Matching task complexity to model capability saves money and often improves speed.

Agents, Skills, Steering, and System Prompts

Mon, 01 Jan 0001 00:00:00 +0000

What Are AI Agents?#

An AI agent is a system that uses a language model as its reasoning core, enhanced with the ability to take actions in the world — calling tools, reading files, querying databases, browsing the web, or interacting with APIs. Unlike a simple chatbot that only generates text, an agent observes, reasons, acts, and iterates.

flowchart LR
 USER["User<br/>Task / Goal"] --> AGENT["Agent<br/>(LLM + Tools + Memory)"]
 AGENT --> THINK["Think:<br/>Plan next step"]
 THINK --> ACT["Act:<br/>Call tool or API"]
 ACT --> OBSERVE["Observe:<br/>Process result"]
 OBSERVE --> THINK
 THINK --> RESPOND["Respond:<br/>Final answer"]

Core Agent Loop#

Most agent frameworks implement some variant of this loop:

MCP (Model Context Protocol)

Mon, 01 Jan 0001 00:00:00 +0000

What Is MCP?#

The Model Context Protocol (MCP) is an open standard for connecting AI models to external data sources and tools. Created by Anthropic and open-sourced in late 2024, it provides a universal interface between AI applications (clients) and integrations (servers), much like USB-C standardized device connectivity.

The Problem MCP Solves#

Before MCP, every AI application had to build custom integrations for each tool:

flowchart LR
 subgraph Before["Before MCP: N×M integrations"]
 A1["App 1"] --> T1["Tool 1"]
 A1 --> T2["Tool 2"]
 A1 --> T3["Tool 3"]
 A2["App 2"] --> T1
 A2 --> T2
 A2 --> T3
 A3["App 3"] --> T1
 A3 --> T2
 A3 --> T3
 end

flowchart LR
 subgraph After["After MCP: N+M integrations"]
 A1["App 1"] --> MCP["MCP Protocol"]
 A2["App 2"] --> MCP
 A3["App 3"] --> MCP
 MCP --> T1["Server 1"]
 MCP --> T2["Server 2"]
 MCP --> T3["Server 3"]
 end

N apps × M tools = N×M custom integrations becomes N + M — each app implements the client protocol once, each tool implements the server protocol once.

ACP (Agent Communication Protocol)

Mon, 01 Jan 0001 00:00:00 +0000

What Is ACP?#

The Agent Communication Protocol (ACP) — also known as A2A (Agent-to-Agent) in Google’s implementation — is an emerging open standard for agent-to-agent communication. While MCP connects an AI agent to tools and data sources, ACP connects agents to each other, enabling them to discover capabilities, delegate tasks, and collaborate without knowing each other’s internal implementation.

MCP vs ACP: Different Problems#

flowchart TB
 subgraph MCP["MCP: Agent ↔ Tools"]
 AGENT1["Agent"] <-->|"MCP"| TOOL1["File System"]
 AGENT1 <-->|"MCP"| TOOL2["Database"]
 AGENT1 <-->|"MCP"| TOOL3["API"]
 end
 
 subgraph ACP["ACP: Agent ↔ Agent"]
 AGENT2["Research Agent"] <-->|"ACP"| AGENT3["Coding Agent"]
 AGENT3 <-->|"ACP"| AGENT4["Review Agent"]
 AGENT2 <-->|"ACP"| AGENT4
 end

Protocol	Connects	Primary Use
MCP	Agent ↔ Tools/Data	Give an agent access to external capabilities
ACP/A2A	Agent ↔ Agent	Let agents collaborate, delegate, and communicate

Why Agent-to-Agent Communication?#

As AI systems become more complex, no single agent can do everything well. Multi-agent architectures emerge: