This guide traces the evolution of language models — from counting word sequences to trillion-parameter neural networks that exhibit emergent reasoning. Understanding this history reveals why modern LLMs work the way they do and what trade-offs each generation made.

Prerequisites#

• Basic probability (conditional probability, Bayes’ theorem)
• Neural network fundamentals (layers, training, loss functions)
• Transformer architecture basics (attention, encoder/decoder)

What Is a Language Model?#

A language model assigns probabilities to sequences of tokens:

P("The cat sat on the mat") = ?

Equivalently, it predicts the next token given context:

P(next_token | previous_tokens)

Every generation of language model — from n-grams to GPT-4 — is fundamentally doing this. The difference is how well they capture the structure of language.

N-Gram Models (1980s–2000s)#

The simplest approach: estimate probabilities by counting occurrences in a corpus.

Bigram model (n=2): P(word | previous_word)

P("mat" | "the") = Count("the mat") / Count("the")

Trigram model (n=3): P(word | two_previous_words)

P("sat" | "cat", "the") = Count("the cat sat") / Count("the cat")

from collections import defaultdict

class BigramModel:
    def __init__(self):
        self.counts = defaultdict(lambda: defaultdict(int))
        self.totals = defaultdict(int)

    def train(self, corpus):
        for sentence in corpus:
            for w1, w2 in zip(sentence, sentence[1:]):
                self.counts[w1][w2] += 1
                self.totals[w1] += 1

    def probability(self, word, context):
        if self.totals[context] == 0:
            return 1e-10
        return self.counts[context][word] / self.totals[context]

Limitations#

N-gram models dominated speech recognition and machine translation until ~2013. They’re still used as baselines and in hybrid systems.

Word Embeddings: Word2Vec and GloVe (2013–2014)#

The breakthrough insight: represent words as dense vectors where similar words have similar vectors.

Word2Vec (Mikolov et al., 2013)#

Two architectures:

• CBOW (Continuous Bag of Words): Predict center word from surrounding context
• Skip-gram: Predict surrounding words from center word

Skip-gram example:
Input: "The [cat] sat on the mat"
Predict: "The", "sat" given "cat"

The trained weight matrix becomes the embedding — each row is a word’s vector.

# Famous analogy: king - man + woman ≈ queen
embedding["king"] - embedding["man"] + embedding["woman"] ≈ embedding["queen"]

GloVe (Pennington et al., 2014)#

Global Vectors — combines the benefits of count-based methods (like LSA) with prediction-based methods (like Word2Vec).

Trains on the co-occurrence matrix: how often words appear near each other across the entire corpus.

Objective: w_i · w_j + b_i + b_j = log(X_ij)
(dot product of word vectors should approximate log co-occurrence count)

Properties of Word Embeddings#

Limitation: Static Embeddings#

Word2Vec/GloVe assign one vector per word regardless of context:

"I went to the bank to deposit money"  ← bank = financial institution
"I sat on the river bank"              ← bank = river edge

Same vector for "bank" in both! This is wrong.

This motivated contextual embeddings.

ELMo: Contextual Embeddings (2018)#

ELMo (Embeddings from Language Models) by Peters et al. was the first widely successful contextual embedding model.

Architecture: Two-layer bidirectional LSTM trained as a language model.

Forward LSTM:  → reads left to right, predicts next word
Backward LSTM: ← reads right to left, predicts previous word

ELMo embedding = weighted combination of all LSTM layers

Key innovation: The same word gets different embeddings depending on context. “Bank” in a financial sentence gets a different vector than “bank” near a river.

Impact: Adding ELMo embeddings to existing models improved performance on virtually every NLP benchmark by 5-25%.

Limitation: Still sequential (LSTM), so training is slow and context is limited.

BERT: Bidirectional Encoder Representations (2018)#

BERT (Devlin et al., Google) applied the Transformer encoder to create deep bidirectional contextual embeddings.

Pre-training Objectives#

Masked Language Modeling (MLM)#

Randomly mask 15% of tokens and predict them:

Input:  "The cat [MASK] on the [MASK]"
Target: "The cat  sat   on the  mat"

This forces bidirectional understanding — the model must use both left and right context.

Next Sentence Prediction (NSP)#

Given two sentences, predict if B follows A:

[CLS] The cat sat on the mat [SEP] It was a sunny day [SEP] → IsNext
[CLS] The cat sat on the mat [SEP] Quantum physics is complex [SEP] → NotNext

(NSP was later shown to be less important — RoBERTa dropped it.)

Architecture#

• Encoder-only Transformer (no causal mask — full bidirectional attention)
• BERT-base: 12 layers, 768 hidden, 12 heads, 110M parameters
• BERT-large: 24 layers, 1024 hidden, 16 heads, 340M parameters
• Vocabulary: 30,000 WordPiece tokens
• Max sequence: 512 tokens

Fine-tuning Pattern#

Pre-trained BERT → Add task-specific head → Fine-tune on labeled data

Classification: [CLS] embedding → linear layer → softmax
NER: Each token embedding → linear layer → tag
QA: Start/end token prediction

Impact#

BERT dominated NLP benchmarks from 2018–2020. It established the pre-train then fine-tune paradigm that all subsequent models follow.

BERT Variants#

GPT Series: Autoregressive Language Models#

GPT-1 (2018)#

Radford et al. (OpenAI) showed that a decoder-only Transformer pre-trained on next-token prediction could be fine-tuned for downstream tasks.

• 12 layers, 768 hidden, 117M parameters
• Pre-training: predict next token (autoregressive)
• Fine-tuning: add linear head per task

GPT-2 (2019)#

Scaled up GPT-1 and demonstrated zero-shot task performance — no fine-tuning needed.

• 48 layers, 1600 hidden, 1.5B parameters
• Trained on WebText (40GB of web pages)
• Key insight: “Language models are unsupervised multitask learners” — framing tasks as text completion works

Translation (zero-shot):
"Translate English to French: cheese =>"  → "fromage"

Summarization (zero-shot):
"Article: [text] TL;DR:"  → summary

GPT-3 (2020)#

The scaling breakthrough — showed that scale alone produces qualitatively new capabilities.

• 96 layers, 12288 hidden, 175B parameters
• Trained on 300B tokens (Common Crawl, books, Wikipedia)
• Introduced in-context learning: provide examples in the prompt, no gradient updates

Few-shot prompting:
"Translate English to French:
sea otter => loutre de mer
cheese => fromage
plush giraffe =>"  → "girafe en peluche"

GPT-4 (2023)#

• Estimated ~1.8T parameters (Mixture of Experts)
• Multimodal (text + images)
• Dramatically improved reasoning, instruction following, safety
• Architecture details not published

The GPT Scaling Story#

T5: Text-to-Text Transfer Transformer (2019)#

Google’s T5 unified all NLP tasks into a single format: text in, text out.

Classification: "classify: This movie is great"     → "positive"
Translation:    "translate English to German: Hello" → "Hallo"
Summarization:  "summarize: [long text]"             → "short summary"
QA:             "question: What is X? context: ..."  → "answer"

Architecture: Encoder-decoder Transformer (like the original).

Key contribution: Systematic study of pre-training objectives, model sizes, data, and transfer approaches. Showed that the text-to-text framing is universal.

Scaling Laws (2020–2022)#

Kaplan et al. (2020) — OpenAI Scaling Laws#

Discovered power-law relationships between model performance and:

• Number of parameters (N)
• Dataset size (D)
• Compute budget (C)

Loss ∝ N^(-0.076)  (more parameters → lower loss)
Loss ∝ D^(-0.095)  (more data → lower loss)
Loss ∝ C^(-0.050)  (more compute → lower loss)

Implication: For a fixed compute budget, it’s better to train a larger model for fewer steps than a smaller model for more steps.

Chinchilla (2022) — DeepMind#

Challenged the “bigger model is always better” assumption. Found that most models were undertrained — they needed more data relative to their size.

Chinchilla optimal: Training tokens ≈ 20× parameter count.

Post-Chinchilla, the field shifted toward training smaller models on more data — better inference cost for equivalent performance.

Emergent Abilities#

As models scale, they exhibit capabilities that don’t exist in smaller models — phase transitions rather than gradual improvement.

Caveat: Some researchers argue emergent abilities are artifacts of evaluation metrics (discontinuous metrics create apparent phase transitions). The debate is ongoing.

Tokenization#

Language models don’t process raw text — they operate on tokens (subword units).

Why Not Characters or Words?#

Byte Pair Encoding (BPE)#

The dominant tokenization algorithm. Iteratively merges the most frequent pair of tokens:

Corpus: "low lower lowest"

Step 0: l o w _ l o w e r _ l o w e s t
Step 1: merge (l, o) → lo:  lo w _ lo w e r _ lo w e s t
Step 2: merge (lo, w) → low: low _ low e r _ low e s t
Step 3: merge (e, r) → er:  low _ low er _ low e s t
Step 4: merge (e, s) → es:  low _ low er _ low es t
Step 5: merge (es, t) → est: low _ low er _ low est
...

Result: Common words become single tokens; rare words split into subwords.

# Example tokenization (GPT-style)
"unhappiness" → ["un", "happiness"]
"transformers" → ["transform", "ers"]
"ChatGPT" → ["Chat", "G", "PT"]

SentencePiece#

Google’s tokenizer that operates on raw text (no pre-tokenization). Treats the input as a sequence of Unicode characters and learns subword units directly. Language-agnostic — works for any script.

Used by: T5, Llama, Mistral, multilingual models.

tiktoken#

OpenAI’s fast BPE implementation used by GPT-3.5/4. Key difference: uses a regex-based pre-tokenizer to split on word boundaries before BPE, preventing merges across word boundaries.

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4")
tokens = enc.encode("Hello, world!")
# [9906, 11, 1917, 0]
print(enc.decode(tokens))
# "Hello, world!"
print(len(enc.encode("This sentence has")), "tokens")

Tokenization Comparison#

Tokenization Gotchas#

• Arithmetic: “123 + 456” might tokenize as [“123”, " +", " 456"] or [“12”, “3”, " +", " 4", “56”] — inconsistent number representation hurts math
• Multilingual: Non-English text often requires 2-4× more tokens per word
• Code: Whitespace-sensitive languages (Python) need careful handling
• Token boundaries: Models can’t “see inside” tokens — they process whole tokens as atomic units

Context Windows#

The context window is the maximum number of tokens a model can process at once.

Why Context Length Matters#

• Longer context = more information available for reasoning
• Enables processing entire codebases, books, or conversation histories
• But: attention is O(n²) — doubling context quadruples compute
• Quality often degrades in the middle of long contexts (“lost in the middle” phenomenon)

Extending Context#

Open vs Closed Models#

Closed-Source (API-only)#

Open-Weight Models#

Open vs Closed Trade-offs#

The Modern LLM Landscape (2024–2025)#

graph TD
    subgraph Closed Source
        GPT4[GPT-4o<br/>OpenAI]
        Claude[Claude 3.5<br/>Anthropic]
        Gemini[Gemini 1.5<br/>Google]
    end

    subgraph Open Weights
        Llama[Llama 3.1<br/>Meta]
        Mistral[Mistral Large<br/>Mistral AI]
        Qwen[Qwen 2.5<br/>Alibaba]
        DS[DeepSeek V3<br/>DeepSeek]
    end

    subgraph Small and Efficient
        Phi[Phi-3<br/>Microsoft]
        Gemma[Gemma 2<br/>Google]
        Llama8[Llama 3.1 8B<br/>Meta]
    end

    subgraph Specialized
        Code[Code Models<br/>StarCoder, CodeLlama]
        Embed[Embedding Models<br/>E5, BGE, Cohere]
        Multi[Multimodal<br/>LLaVA, GPT-4V]
    end

Trends#

1. Open models closing the gap — Llama 3.1 405B matches GPT-4 on many benchmarks
2. Small models getting better — Phi-3 3.8B outperforms GPT-3.5 on some tasks
3. MoE becoming standard — More parameters, same inference cost
4. Longer contexts — 128K–2M tokens becoming common
5. Multimodal — Text + image + audio + video in single models
6. Reasoning models — o1, DeepSeek-R1 with explicit chain-of-thought

Key Takeaways#

1. Language modeling is next-token prediction at scale — from n-grams counting co-occurrences to transformers with trillions of parameters, the core task is the same. The revolution is in how well models capture the structure of language.

2. Contextual embeddings were the paradigm shift — static embeddings (Word2Vec) give one vector per word; contextual models (ELMo → BERT → GPT) give different representations based on surrounding context. This is why modern models understand polysemy and nuance.

3. BERT proved pre-train + fine-tune; GPT proved pre-train + prompt — BERT showed that unsupervised pre-training creates powerful representations. GPT showed that with enough scale, you don’t even need fine-tuning — just describe the task in natural language.

4. Scaling laws are predictable — model performance follows power laws with compute, data, and parameters. Chinchilla showed the optimal balance: ~20 tokens per parameter. This guides billion-dollar training decisions.

5. Tokenization is a hidden bottleneck — BPE/SentencePiece determine what the model can “see.” Poor tokenization hurts arithmetic, multilingual performance, and code understanding. It’s the least glamorous but most practically important preprocessing step.

6. Open models have democratized LLMs — Llama, Mistral, and others provide near-frontier performance with full control over deployment, privacy, and customization. The gap between open and closed narrows with each release.

7. Context length is the new frontier — from 512 tokens (BERT) to 2M tokens (Gemini), longer context enables new applications (full codebase understanding, book-length analysis) but introduces challenges in attention efficiency and retrieval quality.