What Are Regular Expressions?#

Regular expressions (regex) are patterns that describe sets of strings. They’re a universal skill — the same concepts work in grep, sed, awk, Python, JavaScript, and every text editor.

flowchart LR
    Pattern["Pattern: ^[0-9]{4}-[0-9]{2}"] --> Engine["Regex Engine"]
    Input["Input: 2024-01-15 Error occurred"] --> Engine
    Engine --> Match["Match: 2024-01"]

Basic Syntax#

Literal Characters and Metacharacters#

PatternMatches
abcLiteral “abc”
.Any single character (except newline)
\Escape next character

Anchors#

PatternMatches
^Start of line
$End of line
\bWord boundary (Perl/PCRE)
grep "^ERROR" log          # lines starting with ERROR
grep "\.py$" filelist      # lines ending with .py
grep -P "\berror\b" log    # "error" as whole word

Character Classes#

PatternMatches
[abc]Any one of a, b, c
[^abc]Any character NOT a, b, c
[a-z]Any lowercase letter
[A-Z]Any uppercase letter
[0-9]Any digit
[a-zA-Z0-9]Any alphanumeric

POSIX Character Classes#

ClassEquivalentMatches
[:alpha:][a-zA-Z]Letters
[:digit:][0-9]Digits
[:alnum:][a-zA-Z0-9]Alphanumeric
[:space:][ \t\n\r\f\v]Whitespace
[:upper:][A-Z]Uppercase
[:lower:][a-z]Lowercase
grep "[[:digit:]]" file    # lines with at least one digit
grep "[[:space:]]$" file   # lines with trailing whitespace

Quantifiers#

PatternMatches
*Zero or more of preceding
+One or more of preceding (ERE)
?Zero or one of preceding (ERE)
{n}Exactly n times
{n,}n or more times
{n,m}Between n and m times
grep -E "a{3}" file        # "aaa"
grep -E "[0-9]{1,3}" file  # 1-3 digit numbers
grep -E "https?" file      # "http" or "https"

Grouping and Alternation#

PatternMeaning
(abc)Group (capture)
a|ba OR b (BRE)
a|ba OR b (ERE, literal pipe)
grep -E "(error|warning|critical)" log
grep -E "^(GET|POST|PUT|DELETE) " access.log

BRE vs ERE vs PCRE#

FeatureBRE (Basic)ERE (Extended)PCRE (Perl)
Default toolgrep, sedgrep -E, awkgrep -P
+, ?, {, |, (Must escapeLiteralLiteral
\d, \w, \s❌❌✅
Lookahead/behind❌❌✅
Non-greedy *?❌❌✅
Backreferences\1\1\1

Recommendation: Use ERE (grep -E, sed -E) for most tasks. Use PCRE (grep -P) when you need \d, \w, or lookaheads.


Practical Patterns#

Validation#

# IP address (basic — doesn't validate ranges)
grep -E '^([0-9]{1,3}\.){3}[0-9]{1,3}$'

# Email (simplified)
grep -E '^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$'

# ISO date (YYYY-MM-DD)
grep -E '^[0-9]{4}-(0[1-9]|1[0-2])-(0[1-9]|[12][0-9]|3[01])$'

# Semantic version
grep -E '^v?[0-9]+\.[0-9]+\.[0-9]+(-[a-zA-Z0-9.]+)?$'

# UUID
grep -E '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$'

Extraction#

# Extract URLs
grep -oE 'https?://[^ >"]+' file

# Extract IP addresses
grep -oP '\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}' file

# Extract quoted strings
grep -oE '"[^"]*"' file

# Extract key=value pairs
grep -oE '[A-Z_]+=[^ ]+' file

Transformation with sed#

# Reformat date: YYYY-MM-DD → DD/MM/YYYY
sed -E 's/([0-9]{4})-([0-9]{2})-([0-9]{2})/\3\/\2\/\1/g'

# Extract domain from URL
sed -E 's|https?://([^/]+).*|\1|'

# Remove HTML tags
sed -E 's/<[^>]+>//g'

# Normalize whitespace
sed -E 's/[[:space:]]+/ /g; s/^ //; s/ $//'

Regex in Bash#

[[ =~ ]] — Bash Regex Matching#

if [[ "$input" =~ ^[0-9]+$ ]]; then
    echo "It's a number"
fi

# Capture groups via BASH_REMATCH
if [[ "$date" =~ ^([0-9]{4})-([0-9]{2})-([0-9]{2})$ ]]; then
    year="${BASH_REMATCH[1]}"
    month="${BASH_REMATCH[2]}"
    day="${BASH_REMATCH[3]}"
fi

Important: Don’t quote the regex pattern in [[ =~ ]] — quoting makes it a literal string match.


Common Mistakes#

MistakeProblemFix
grep ".*"Matches everythingBe more specific
Forgetting anchorsPartial matchesUse ^...$ for full-line
Greedy matching<.*> matches too muchUse <[^>]*> or non-greedy <.*?> (PCRE)
Unescaped dots. matches ANY charUse \. for literal dot
Wrong regex flavor\d in grep (no PCRE)Use [0-9] or grep -P

Lookaheads and Lookbehinds#

Lookaheads and lookbehinds (collectively “lookarounds”) assert that a pattern exists ahead of or behind the current position without consuming characters. They are zero-width assertions — they check but do not include the match.

Available in PCRE only (grep -P, Python re, JavaScript RegExp, Java Pattern).

Syntax#

TypeSyntaxMeaning
Positive lookahead(?=...)What follows must match
Negative lookahead(?!...)What follows must NOT match
Positive lookbehind(?<=...)What precedes must match
Negative lookbehind(?<!...)What precedes must NOT match

Examples#

# Match "price" only if followed by a number
grep -P 'price(?=\s*\d)' file
# "price 42" ✓  "price tag" ✗

# Match a number NOT followed by "px"
grep -P '\d+(?!px)' file
# "100em" ✓  "100px" ✗ (the "100" part)

# Match a number preceded by "$"
grep -P '(?<=\$)\d+' file
# "$42" → matches "42"  "€42" ✗

# Match a word NOT preceded by "un"
grep -P '(?<!un)happy' file
# "happy" ✓  "unhappy" ✗

Practical Recipes#

# Password validation: 8+ chars, at least one digit, one uppercase, one lowercase
grep -P '^(?=.*[a-z])(?=.*[A-Z])(?=.*\d).{8,}$'

# Extract values from key=value, but only the value (lookbehind)
grep -oP '(?<=API_KEY=)[^\s]+'

# Find "TODO" comments not followed by an assignee
grep -P 'TODO(?!\s*\([A-Za-z]+\))' *.py

# Match a domain, but not "localhost"
grep -P '(?<!local)host\.' config

Lookbehind Limitations#

  • Fixed-width only in most engines (Python, Java, PCRE): (?<=ab) works, (?<=a+) does not
  • JavaScript supports variable-length lookbehind since ES2018
  • Workaround for variable-length: use \K in PCRE to reset the match start
# \K discards everything matched so far (PCRE alternative to lookbehind)
grep -oP 'price:\s*\K\d+' file
# "price: 42" → matches "42"

Catastrophic Backtracking and ReDoS#

The Problem#

Regex engines use backtracking to try different ways of matching a pattern. Certain patterns cause exponential backtracking on non-matching input — the engine tries every possible combination before giving up.

The Classic Dangerous Pattern#

(a+)+$

Against input aaaaaaaaaaaaaaaaX:

  • The engine tries matching all as in group 1, fails at X
  • Backtracks: tries (n-1) as in first group, 1 in second, fails
  • Tries (n-2), (1, 1), fails; tries (n-2), (2), fails…
  • 2ⁿ combinations before concluding no match

With 25 as, this can take seconds. With 30+, it can hang the process.

Patterns That Cause Backtracking#

The recipe for catastrophic backtracking is nested quantifiers on overlapping alternatives:

DangerousWhySafe Alternative
(a+)+Nested quantifiersa+
(a|aa)+Overlapping alternativesa+
(.*a){10}Greedy .* with required char([^a]*a){10}
(\s+|,)+Overlapping with whitespace[\s,]+
(x+x+)+yNested quantifiersx{2,}y

ReDoS (Regular Expression Denial of Service)#

If user-supplied input is matched against a vulnerable regex, an attacker can craft input that hangs the server:

# VULNERABLE — user input matched against dangerous pattern
import re
pattern = re.compile(r'^(([a-z])+\.)+[A-Z]{2,}$')
# Attacker sends: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!"
# → server hangs

Real-world incidents: Cloudflare outage (2019), Stack Overflow outage (2016), npm package slug vulnerability.

Prevention#

StrategyHow
Avoid nested quantifiersFlatten (a+)+ to a+
Use atomic groups(?>a+) — no backtracking into group (PCRE/Java)
Use possessive quantifiersa++ — same as atomic, shorthand (PCRE/Java)
Set timeoutPython: use regex library with timeout; JS: built-in timeout (Node 20+)
Use RE2Google’s RE2 engine guarantees linear time (no backtracking) — used by Go, Rust
Lint your patternsTools: redos-checker, safe-regex, rxxr2
Avoid user-supplied regexIf you must accept them, use RE2 or sandbox with timeouts
# SAFE — using atomic group equivalent via possessive quantifier
# Python's re module doesn't support possessive, but the regex library does:
import regex
pattern = regex.compile(r'^([a-z]++\.)+[A-Z]{2,}$')

Testing for Vulnerable Patterns#

# Quick test: time the match against a long non-matching string
time echo "aaaaaaaaaaaaaaaaaaaaaaaaaaaa!" | grep -P '(a+)+$'
# If it takes more than 0.01s, the pattern is vulnerable

Regex in Programming Languages#

The Bash/grep/sed patterns above transfer directly to programming languages, but each has its own API and quirks.

Python (re module)#

import re

# Compile for reuse
pattern = re.compile(r'\d{4}-\d{2}-\d{2}')

# Search (first match)
match = pattern.search("Date: 2024-01-15 and 2024-02-20")
if match:
    print(match.group())     # "2024-01-15"
    print(match.start())     # 6

# Find all
dates = pattern.findall("2024-01-15 and 2024-02-20")
# ["2024-01-15", "2024-02-20"]

# Named groups
m = re.match(r'(?P<year>\d{4})-(?P<month>\d{2})-(?P<day>\d{2})', "2024-01-15")
m.group('year')   # "2024"
m.groupdict()     # {'year': '2024', 'month': '01', 'day': '15'}

# Substitution
re.sub(r'\bfoo\b', 'bar', "foo foobar foo")  # "bar foobar bar"

# Split
re.split(r'[,;\s]+', "a, b; c  d")  # ['a', 'b', 'c', 'd']

Python gotcha: Use raw strings (r'...') to avoid double-escaping backslashes.

JavaScript (RegExp)#

// Literal syntax (preferred)
const pattern = /\d{4}-\d{2}-\d{2}/g;

// Constructor (for dynamic patterns)
const dynamic = new RegExp(`user_${userId}`, 'i');

// Test (boolean)
/^\d+$/.test("42");              // true

// Match (array or null)
"2024-01-15".match(/(\d{4})-(\d{2})-(\d{2})/);
// ["2024-01-15", "2024", "01", "15"]

// Named groups (ES2018+)
const m = "2024-01-15".match(/(?<year>\d{4})-(?<month>\d{2})-(?<day>\d{2})/);
m.groups.year;                   // "2024"

// matchAll (ES2020+ — requires /g flag)
for (const m of text.matchAll(/\d+/g)) {
    console.log(m[0], m.index);
}

// Replace
"foo bar foo".replace(/foo/g, "baz");  // "baz bar baz"
"2024-01-15".replace(/(\d{4})-(\d{2})-(\d{2})/, "$3/$2/$1");  // "15/01/2024"

JS gotcha: The /g flag makes the regex stateful — regex.lastIndex advances between exec() calls. Reuse carefully.

Java (Pattern / Matcher)#

import java.util.regex.*;

Pattern pattern = Pattern.compile("\\d{4}-\\d{2}-\\d{2}");
Matcher matcher = pattern.matcher("Date: 2024-01-15");

if (matcher.find()) {
    System.out.println(matcher.group());   // "2024-01-15"
    System.out.println(matcher.start());   // 6
}

// Named groups
Pattern dated = Pattern.compile("(?<year>\\d{4})-(?<month>\\d{2})-(?<day>\\d{2})");
Matcher dm = dated.matcher("2024-01-15");
if (dm.matches()) {
    dm.group("year");    // "2024"
}

// Replace
"foo bar foo".replaceAll("\\bfoo\\b", "baz");  // "baz bar baz"

// Split
"a, b; c  d".split("[,;\\s]+");  // ["a", "b", "c", "d"]

Java gotcha: Backslashes must be double-escaped in strings (\\d not \d).

Quick Comparison#

FeaturePython reJavaScript RegExpJava Pattern
Literal syntaxNo — use r'...'Yes — /pattern/flagsNo — use "\\\\..."
Named groups(?P<name>...)(?<name>...)(?<name>...)
Lookaheads✓✓✓
Lookbehinds✓ (fixed-width)✓ (variable since ES2018)✓ (fixed-width)
Possessive quantifiersregex library only✗✓
Atomic groupsregex library only✗✓ (via (?>...))
Unicode support✓ (re.UNICODE default in Python 3)✓ (/u flag)✓ (Pattern.UNICODE_CHARACTER_CLASS)
Timeout/safetyregex library timeout paramNode 20+ --experimental-vm-modulesManual via Thread.interrupt()

Key Takeaways#

  1. Start simple — anchors (^$), character classes ([0-9]), quantifiers (+*?)
  2. Use ERE (grep -E, sed -E) — avoids escaping +, ?, |, ()
  3. . matches anything — escape it (\.) when you mean a literal dot
  4. Anchors prevent partial matches — ^pattern$ for full-line matching
  5. Lookaheads/lookbehinds check without consuming — essential for password validation, context-sensitive extraction, and complex replacements
  6. Beware nested quantifiers — (a+)+ causes exponential backtracking. Flatten, use atomic groups, or switch to RE2
  7. Never run user-supplied regex without a timeout or a safe engine (RE2)
  8. BASH_REMATCH captures groups from [[ =~ ]]; named groups work in Python, JS, and Java
  9. Test interactively — use grep -E "pattern" <<< "test string" to verify
  10. Regex is universal — same concepts in every language and tool, only the API and escaping differ