Scripting on Software Engineering Learning Paths

Shell Basics

Mon, 01 Jan 0001 00:00:00 +0000

What is a Shell?#

A shell is a command-line interpreter — a program that reads text commands, interprets them, and executes them by calling the operating system’s kernel. It serves two roles: an interactive environment for running commands one at a time, and a programming language for automating sequences of commands.

flowchart LR
 User["User Input"] --> Shell["Shell (bash, zsh)"]
 Shell --> Kernel["OS Kernel"]
 Kernel --> HW["Hardware"]
 Kernel --> FS["File System"]
 Kernel --> Net["Network"]
 Shell --> Output["Output (stdout/stderr)"]

The shell is NOT the terminal. The terminal is the window application (iTerm2, GNOME Terminal). The shell is the program running inside it.

Variables, Quoting, and Expansion

Mon, 01 Jan 0001 00:00:00 +0000

Variables#

Shell variables are untyped — everything is a string. The shell interprets context to decide if something is a number or text.

Assignment Rules#

# Correct — NO spaces around =
name="Alice"
count=42
path="/usr/local/bin"

# WRONG — spaces cause errors
name = "Alice" # Error: "name: command not found"

Variable Usage#

name="world"
echo "Hello, $name" # Hello, world
echo "Hello, ${name}" # Same — braces for clarity
echo "${name}_backup" # world_backup (braces needed here)
echo "$name_backup" # Empty! (looks for variable name_backup)

Variable Types#

# Regular variable (local to current shell/script)
greeting="Hello"

# Environment variable (inherited by child processes)
export API_KEY="abc123"

# Readonly (constant)
readonly MAX_RETRIES=3
MAX_RETRIES=5 # Error: readonly variable

# Unset
unset greeting # removes the variable

Special Variables#

Variable	Meaning
`$0`	Script name
`$1`-`$9`	Positional parameters
`$#`	Number of arguments
`$@`	All arguments (as separate words)
`$*`	All arguments (as single string)
`$?`	Exit code of last command
`$$`	Current process PID
`$!`	PID of last background process

Quoting#

Quoting is the #1 source of shell scripting bugs. Understanding it prevents most issues.

Control Structures

Mon, 01 Jan 0001 00:00:00 +0000

Conditionals: if/elif/else#

if [[ "$status" == "active" ]]; then
 echo "Service is running"
elif [[ "$status" == "stopped" ]]; then
 echo "Service is stopped"
else
 echo "Unknown status: $status"
fi

The semicolon before then replaces a newline — these are equivalent:

if [[ condition ]]; then command; fi

if [[ condition ]]
then
 command
fi

Test Commands: `[[ ]]` vs `[ ]`#

`[[ ]]` — Bash Extended Test (Preferred)#

Category	Operator	Meaning
String	`==`	Equal
String	`!=`	Not equal
String	`=~`	Regex match
String	`-z "$s"`	String is empty
String	`-n "$s"`	String is non-empty
Numeric	`-eq`, `-ne`	Equal, not equal
Numeric	`-lt`, `-le`	Less than, less or equal
Numeric	`-gt`, `-ge`	Greater than, greater or equal
File	`-f`	Is regular file
File	`-d`	Is directory
File	`-e`	Exists (any type)
File	`-r`, `-w`, `-x`	Readable, writable, executable
File	`-s`	Exists and non-empty
Logic	`&&`	AND
Logic	`\|\|`	OR
Logic	`!`	NOT

`[ ]` vs `[[ ]]` Comparison#

Feature	`[ ]` (POSIX)	`[[ ]]` (Bash)
Word splitting on vars	Yes (must quote!)	No
Glob expansion	Yes	No
Regex (`=~`)	❌	✅
Pattern matching (`==` with `*`)	❌	✅
`&&`, `\|\|` inside	❌	✅
Portability	All shells	Bash/Zsh only

Rule: Use [[ ]] in bash scripts. Use [ ] only for POSIX portability.

Functions

Mon, 01 Jan 0001 00:00:00 +0000

Defining Functions#

# Standard syntax
greet() {
 local name="${1:?Error: name required}"
 local greeting="${2:-Hello}"
 echo "${greeting}, ${name}!"
}

# Alternative syntax (less common)
function greet {
 echo "Hello, $1"
}

The function keyword is optional in bash. Prefer the name() syntax — it’s POSIX-compatible.

Parameters and Return Values#

Positional Parameters#

create_user() {
 local username="$1"
 local email="$2"
 local role="${3:-viewer}" # default value
 
 echo "Creating $username ($email) with role: $role"
}

create_user "alice" "alice@example.com" "admin"
create_user "bob" "bob@example.com" # role defaults to "viewer"

Parameter Validation#

deploy() {
 local env="${1:?Error: environment required (test|pre|prod)}"
 local version="${2:?Error: version required}"
 
 [[ "$env" =~ ^(test|pre|prod)$ ]] || {
 echo "Invalid environment: $env" >&2
 return 1
 }
 
 echo "Deploying v$version to $env"
}

Return Values#

Functions return exit codes (0-255), not data. Use echo + command substitution for data:

Text Processing

Mon, 01 Jan 0001 00:00:00 +0000

The Big Three: grep, sed, awk#

These three tools handle 90% of text processing needs on any Unix system:

Tool	Purpose	Think of it as
`grep`	Find lines matching a pattern	Filter
`sed`	Transform text (find & replace)	Editor
`awk`	Process structured/columnar data	Spreadsheet

flowchart LR
 Input["Input Text"] --> grep["grep: FIND"]
 Input --> sed["sed: TRANSFORM"]
 Input --> awk["awk: PROCESS"]
 grep --> Lines["Matching lines"]
 sed --> Modified["Modified text"]
 awk --> Computed["Computed results"]

grep — Search for Patterns#

Basic Usage#

grep "error" logfile.txt # lines containing "error"
grep -i "error" logfile.txt # case-insensitive
grep -n "error" logfile.txt # show line numbers
grep -c "error" logfile.txt # count matches
grep -l "TODO" src/*.py # list files with matches
grep -L "test" src/*.py # list files WITHOUT matches
grep -v "debug" logfile.txt # invert (exclude matches)
grep -r "TODO" src/ # recursive search
grep -w "error" logfile.txt # whole word only

Extended Regex (`-E`)#

grep -E "error|warning|critical" log # alternation (OR)
grep -E "^[0-9]{4}-[0-9]{2}" log # lines starting with date
grep -E "HTTP/[12]\.[01]\" [45]" log # 4xx/5xx responses
grep -oE "[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+" log # extract IPs

Common Patterns#

# Find function definitions in Python
grep -rn "^def " src/

# Find TODO/FIXME comments
grep -rn "TODO\|FIXME\|HACK" --include="*.py" .

# Count errors per log file
grep -c "ERROR" /var/log/*.log

# Lines between two patterns
grep -A5 "Exception" log # 5 lines After match
grep -B3 "Exception" log # 3 lines Before match
grep -C2 "Exception" log # 2 lines Context (before + after)

sed — Stream Editor#

Substitution (Most Common Use)#

# Replace first occurrence per line
sed 's/old/new/' file

# Replace ALL occurrences per line
sed 's/old/new/g' file

# Case-insensitive replace
sed 's/error/ERROR/gi' file

# In-place editing
sed -i 's/old/new/g' file # Linux
sed -i '' 's/old/new/g' file # macOS (requires empty string)

Line Operations#

# Delete lines
sed '/^#/d' file # delete comment lines
sed '/^$/d' file # delete empty lines
sed '1,5d' file # delete lines 1-5

# Print specific lines
sed -n '10,20p' file # print lines 10-20
sed -n '/START/,/END/p' file # print between patterns

# Insert/append
sed '1i\Header line' file # insert before line 1
sed '$a\Footer line' file # append after last line

Practical Examples#

# Remove trailing whitespace
sed 's/[[:space:]]*$//' file

# Replace environment variable placeholders
sed "s/\${DB_HOST}/$DB_HOST/g" template.conf > config.conf

# Comment out a line
sed -i '/dangerous_setting/s/^/# /' config.ini

# Extract value from key=value
sed -n 's/^version=//p' config.ini

Multiple Operations#

# Chain with -e
sed -e 's/foo/bar/g' -e 's/baz/qux/g' file

# Or use semicolons
sed 's/foo/bar/g; s/baz/qux/g' file

awk — Pattern Scanning and Processing#

Column Extraction#

# Print specific columns (whitespace-delimited)
awk '{print $1, $3}' file

# Custom delimiter
awk -F',' '{print $2}' data.csv # CSV
awk -F':' '{print $1}' /etc/passwd # colon-separated

Filtering#

# Where clause (column conditions)
awk '$3 > 100 {print $1, $3}' file
awk '$2 == "ERROR" {print}' log
awk 'NF > 3' file # lines with more than 3 fields
awk 'length > 80' file # lines longer than 80 chars

Built-in Variables#

Variable	Meaning
`$0`	Entire line
`$1`-`$N`	Field N
`NR`	Current line number
`NF`	Number of fields in current line
`FS`	Field separator (input)
`OFS`	Output field separator

# Number lines
awk '{print NR": "$0}' file

# Print last field
awk '{print $NF}' file

# Print second-to-last field
awk '{print $(NF-1)}' file

Aggregation#

# Sum a column
awk '{sum += $2} END {print "Total:", sum}' sales.txt

# Average
awk '{sum += $2; n++} END {print "Avg:", sum/n}' data.txt

# Count by category
awk '{count[$1]++} END {for (k in count) print k, count[k]}' log

# Max value
awk 'BEGIN{max=0} $3>max{max=$3} END{print "Max:", max}' data.txt

BEGIN and END Blocks#

awk '
BEGIN { FS=","; OFS="\t"; print "Name\tAge\tCity" }
NR > 1 { print $1, $2, $3 }
END { print "---"; print NR-1, "records processed" }
' data.csv

cut, sort, uniq — Simple but Powerful#

cut — Extract Columns#

cut -d',' -f1,3 data.csv # fields 1 and 3, comma-delimited
cut -d':' -f1 /etc/passwd # usernames
cut -c1-10 file # characters 1-10

sort#

sort file # alphabetical
sort -n file # numeric
sort -k2 -t',' file # by field 2, comma-delimited
sort -r file # reverse
sort -u file # unique (deduplicate)
sort -h file # human-readable numbers (1K, 2M)

uniq (requires sorted input)#

sort file | uniq # remove duplicates
sort file | uniq -c # count occurrences
sort file | uniq -d # show only duplicates
sort file | uniq -c | sort -rn # frequency ranking

Combining Tools#

# Top 10 IP addresses in access log
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -10

# Count HTTP status codes
awk '{print $9}' access.log | sort | uniq -c | sort -rn

# Find largest files
find . -type f -exec du -h {} \; | sort -rh | head -20

# Extract unique error messages
grep "ERROR" app.log | sed 's/.*ERROR //' | sort -u

Key Takeaways#

grep finds, sed transforms, awk processes — know which to reach for
grep -E for extended regex (alternation, quantifiers)
sed ’s/old/new/g’ for find-and-replace (add -i for in-place)
awk ‘{print $N}’ for column extraction
sort | uniq -c | sort -rn is the frequency analysis pattern
Combine tools with pipes — each does one thing well

Pipes, Redirection, and Process Substitution

Mon, 01 Jan 0001 00:00:00 +0000

The Unix Philosophy#

Write programs that do one thing and do it well. Write programs to work together.

Pipes are how you connect small tools into powerful workflows.

flowchart LR
 A["cat log"] -->|pipe| B["grep ERROR"]
 B -->|pipe| C["awk '{print $4}'"]
 C -->|pipe| D["sort"]
 D -->|pipe| E["uniq -c"]
 E -->|pipe| F["sort -rn"]
 F -->|pipe| G["head -10"]

Pipes (`|`)#

Connect stdout of one command to stdin of the next:

# Top 10 error sources from a log
cat access.log | grep "ERROR" | awk '{print $4}' | sort | uniq -c | sort -rn | head -10

# Count unique users currently logged in
who | awk '{print $1}' | sort -u | wc -l

# Find largest directories
du -sh */ | sort -rh | head -5

Pipe Behavior#

Each command in a pipe runs as a separate process
They run concurrently (not sequentially)
Data flows as a stream — no temp files needed
If any command in the pipe exits, the others receive SIGPIPE

File Descriptors#

Every process has three standard streams:

File Operations and Permissions

Mon, 01 Jan 0001 00:00:00 +0000

File Operations#

Creating Files and Directories#

touch file.txt # create empty file (or update timestamp)
mkdir mydir # create directory
mkdir -p path/to/nested/dir # create parent dirs as needed

Copying, Moving, Deleting#

# Copy
cp source.txt dest.txt
cp -r source_dir/ dest_dir/ # recursive (directories)
cp -p file.txt backup/ # preserve permissions and timestamps

# Move / Rename
mv old_name.txt new_name.txt
mv file.txt /other/directory/

# Delete
rm file.txt
rm -r directory/ # recursive
rm -f file.txt # force (no error if missing)
rm -rf directory/ # recursive + force (DANGEROUS)

Safety with `rm`#

# NEVER do this with unquoted or unset variables
rm -rf "$DIR/" # if DIR is empty → rm -rf / !!!

# Safe pattern
[[ -n "$DIR" ]] || die "DIR is empty"
[[ "$DIR" != "/" ]] || die "Refusing to delete /"
rm -rf "${DIR:?ERROR: DIR not set}"

find — The Swiss Army Knife#

Basic Searches#

# By name
find . -name "*.py" # exact glob match
find . -iname "*.py" # case-insensitive
find . -name "*.py" -o -name "*.js" # OR

# By type
find . -type f # regular files only
find . -type d # directories only
find . -type l # symlinks only

# By size
find . -size +100M # larger than 100MB
find . -size -1k # smaller than 1KB

# By time
find . -mtime -7 # modified in last 7 days
find . -mtime +30 # modified more than 30 days ago
find . -newer reference.txt # newer than reference file

Actions#

# Execute command on each result
find . -name "*.log" -exec rm {} \;
find . -name "*.py" -exec grep -l "TODO" {} \;

# Batch execution (more efficient)
find . -name "*.log" -exec rm {} +

# Print with custom format
find . -type f -printf "%s %p\n" # size and path

# Delete directly (faster than -exec rm)
find . -name "*.tmp" -delete

Combining Conditions#

# AND (implicit)
find . -name "*.log" -size +10M

# OR
find . \( -name "*.log" -o -name "*.tmp" \) -mtime +7

# NOT
find . -type f ! -name "*.py"

# Practical: find large old logs
find /var/log -name "*.log" -size +50M -mtime +30 -exec ls -lh {} \;

Permissions#

Understanding Permission Notation#

-rwxr-xr-- 1 alice developers 4096 Jan 1 12:00 script.sh
│├─┤├─┤├─┤
│ │ │ └── Others: read only (r--)
│ │ └───── Group: read + execute (r-x)
│ └───────── Owner: read + write + execute (rwx)
└─────────── Type: - = file, d = directory, l = symlink

Octal (Numeric) Permissions#

Octal	Binary	Permission
0	000	— (none)
1	001	–x (execute)
2	010	-w- (write)
3	011	-wx (write + execute)
4	100	r– (read)
5	101	r-x (read + execute)
6	110	rw- (read + write)
7	111	rwx (all)

Common Permission Patterns#

Octal	Meaning	Use Case
`755`	rwxr-xr-x	Executable scripts, directories
`644`	rw-r–r–	Regular files
`600`	rw——-	Private files (keys, secrets)
`700`	rwx——	Private directories
`775`	rwxrwxr-x	Shared group directories
`664`	rw-rw-r–	Shared group files

chmod — Change Permissions#

# Numeric
chmod 755 script.sh
chmod 644 config.yml
chmod 600 ~/.ssh/id_rsa

# Symbolic
chmod +x script.sh # add execute for all
chmod u+w file.txt # add write for owner
chmod go-r file.txt # remove read for group and others
chmod a+r file.txt # add read for all
chmod u=rwx,go=rx script.sh # set explicitly

chown — Change Ownership#

chown alice file.txt # change owner
chown alice:developers file.txt # change owner and group
chown -R alice:developers dir/ # recursive
chown :developers file.txt # change group only

Directory Permissions#

Directories have different semantics for the same bits:

Regular Expressions

Mon, 01 Jan 0001 00:00:00 +0000

What Are Regular Expressions?#

Regular expressions (regex) are patterns that describe sets of strings. They’re a universal skill — the same concepts work in grep, sed, awk, Python, JavaScript, and every text editor.

flowchart LR
 Pattern["Pattern: ^[0-9]{4}-[0-9]{2}"] --> Engine["Regex Engine"]
 Input["Input: 2024-01-15 Error occurred"] --> Engine
 Engine --> Match["Match: 2024-01"]

Basic Syntax#

Literal Characters and Metacharacters#

Pattern	Matches
`abc`	Literal “abc”
`.`	Any single character (except newline)
`\`	Escape next character

Anchors#

Pattern	Matches
`^`	Start of line
`$`	End of line
`\b`	Word boundary (Perl/PCRE)

grep "^ERROR" log # lines starting with ERROR
grep "\.py$" filelist # lines ending with .py
grep -P "\berror\b" log # "error" as whole word

Character Classes#

Pattern	Matches
`[abc]`	Any one of a, b, c
`[^abc]`	Any character NOT a, b, c
`[a-z]`	Any lowercase letter
`[A-Z]`	Any uppercase letter
`[0-9]`	Any digit
`[a-zA-Z0-9]`	Any alphanumeric

POSIX Character Classes#

Class	Equivalent	Matches
`[:alpha:]`	`[a-zA-Z]`	Letters
`[:digit:]`	`[0-9]`	Digits
`[:alnum:]`	`[a-zA-Z0-9]`	Alphanumeric
`[:space:]`	`[ \t\n\r\f\v]`	Whitespace
`[:upper:]`	`[A-Z]`	Uppercase
`[:lower:]`	`[a-z]`	Lowercase

grep "[[:digit:]]" file # lines with at least one digit
grep "[[:space:]]$" file # lines with trailing whitespace

Quantifiers#

Pattern	Matches
`*`	Zero or more of preceding
`+`	One or more of preceding (ERE)
`?`	Zero or one of preceding (ERE)
`{n}`	Exactly n times
`{n,}`	n or more times
`{n,m}`	Between n and m times

grep -E "a{3}" file # "aaa"
grep -E "[0-9]{1,3}" file # 1-3 digit numbers
grep -E "https?" file # "http" or "https"

Grouping and Alternation#

Pattern	Meaning
`(abc)`	Group (capture)
`a\|b`	a OR b (BRE)
`a\|b`	a OR b (ERE, literal pipe)

grep -E "(error|warning|critical)" log
grep -E "^(GET|POST|PUT|DELETE) " access.log

BRE vs ERE vs PCRE#

Feature	BRE (Basic)	ERE (Extended)	PCRE (Perl)
Default tool	`grep`, `sed`	`grep -E`, `awk`	`grep -P`
`+`, `?`, `{`, `\|`, `(`	Must escape	Literal	Literal
`\d`, `\w`, `\s`	❌	❌	✅
Lookahead/behind	❌	❌	✅
Non-greedy `*?`	❌	❌	✅
Backreferences	`\1`	`\1`	`\1`

Recommendation: Use ERE (grep -E, sed -E) for most tasks. Use PCRE (grep -P) when you need \d, \w, or lookaheads.

Process Management

Mon, 01 Jan 0001 00:00:00 +0000

What is a Process?#

A process is a running instance of a program. Every command you run creates a process with its own PID (Process ID), memory space, and file descriptors.

flowchart TD
 Init["init/systemd (PID 1)"]
 Init --> Shell["bash (PID 1234)"]
 Shell --> Script["./deploy.sh (PID 5678)"]
 Script --> Curl["curl (PID 5679)"]
 Script --> Grep["grep (PID 5680)"]

Process States#

State	Meaning
Running (R)	Actively executing on CPU
Sleeping (S)	Waiting for I/O or event
Stopped (T)	Suspended (Ctrl+Z)
Zombie (Z)	Finished but parent hasn’t collected exit status

Viewing Processes#

# Current user's processes
ps aux | grep "[m]yapp"

# All processes, full format
ps -ef

# Process tree
pstree -p

# Find PID by name
pgrep -f "python app.py"
pidof nginx

# Real-time monitoring
top # built-in
htop # better (if installed)

Useful `ps` Patterns#

# Top CPU consumers
ps aux --sort=-%cpu | head -10

# Top memory consumers
ps aux --sort=-%mem | head -10

# Processes by a specific user
ps -u alice

# Show process hierarchy
ps -ejH

Background Jobs#

# Run in background
long_command &
echo "PID: $!" # PID of last background process

# List background jobs
jobs # shows job numbers
jobs -l # includes PIDs

# Bring to foreground
fg %1 # job number 1

# Suspend and background
# Press Ctrl+Z (suspends foreground process)
bg %1 # resume in background

# Detach from terminal
nohup long_command & # survives terminal close
disown %1 # detach already-running job

Wait for Background Processes#

# Wait for specific PID
long_task_1 &
pid1=$!
long_task_2 &
pid2=$!

wait $pid1 $pid2
echo "Both tasks complete"

# Wait for all background jobs
wait

Signals#

Signals are software interrupts sent to processes:

Script Debugging and Error Handling

Mon, 01 Jan 0001 00:00:00 +0000

The Problem: Silent Failures#

Without explicit error handling, shell scripts fail silently and continue executing — often causing more damage than the original error:

# Dangerous: if cd fails, rm deletes from WRONG directory
cd /data/temp
rm -rf *

# Safe: exit if cd fails
cd /data/temp || exit 1
rm -rf *

Strict Mode: `set -euo pipefail`#

The single most important line in any bash script:

#!/bin/bash
set -euo pipefail

What Each Flag Does#

Flag	Without It	With It
`-e` (errexit)	Script continues after failed commands	Script exits immediately on failure
`-u` (nounset)	Unset variables expand to empty string	Unset variables cause an error
`-o pipefail`	Pipe exit code = last command’s code	Pipe exit code = first failure’s code

`-e` in Detail#

set -e

cp important.dat /backup/ # if this fails...
rm important.dat # ...this NEVER runs (script exits)
echo "Done" # also never runs

Commands that legitimately return non-zero need special handling:

Cron, Systemd Timers, and Automation

Mon, 01 Jan 0001 00:00:00 +0000

Scheduled Task Execution#

Automation means running scripts without human intervention — on a schedule, in response to events, or as part of a pipeline.

flowchart TD
 Trigger["Trigger"]
 Trigger --> Cron["Cron (time-based)"]
 Trigger --> Systemd["Systemd Timer (time-based, managed)"]
 Trigger --> Event["Event (file change, webhook)"]
 Trigger --> CI["CI/CD Pipeline (code push)"]
 
 Cron --> Script["Script Execution"]
 Systemd --> Script
 Event --> Script
 CI --> Script

Cron#

The traditional Unix scheduler. Runs commands at specified times.

Portable Scripting and POSIX Compliance

Mon, 01 Jan 0001 00:00:00 +0000

Why Portability Matters#

Not every system has bash. Alpine Docker images use ash, BSD systems use sh, embedded systems have minimal shells. If your script must run across different environments, you need POSIX compliance.

flowchart TD
 Script["Your Script"]
 Script --> Q{"Where does it run?"}
 Q -->|"Linux servers (bash guaranteed)"| Bash["Use #!/bin/bash"]
 Q -->|"Docker (Alpine, minimal)"| POSIX["Use #!/bin/sh"]
 Q -->|"macOS + Linux"| Env["Use #!/usr/bin/env bash"]
 Q -->|"CI/CD (controlled)"| Bash
 Q -->|"Multi-OS distribution"| POSIX

POSIX vs Bash Features#

Feature	POSIX (`sh`)	Bash
`[[ ]]` extended test	❌	✅
Arrays	❌	✅
`local` keyword	Not guaranteed	✅
`$(( ))` arithmetic	✅	✅
`$(command)` substitution	✅	✅
`{1..10}` brace expansion	❌	✅
`<<<` here string	❌	✅
Process substitution `<()`	❌	✅
`=~` regex matching	❌	✅
`==` in `[[ ]]`	❌	✅
`source` keyword	❌ (use `.`)	✅
`function` keyword	❌	✅
`${var,,}` case change	❌	✅ (bash 4+)
`set -o pipefail`	❌	✅
`$RANDOM`	❌	✅
`read -p` (prompt)	❌	✅

Writing POSIX-Compliant Scripts#

Test Commands#

# Bash (not portable)
if [[ "$var" == "value" ]]; then ...
if [[ "$var" =~ ^[0-9]+$ ]]; then ...

# POSIX equivalent
if [ "$var" = "value" ]; then ...
# No regex — use case or grep
case "$var" in
 [0-9]*) echo "starts with digit" ;;
esac

String Operations#

# Bash
[[ -z "$var" ]]
[[ "$a" == "$b" ]]

# POSIX
[ -z "$var" ]
[ "$a" = "$b" ] # single = for equality

Functions#

# Bash
function greet {
 local name="$1"
 echo "Hello, $name"
}

# POSIX
greet() {
 name="$1" # no guaranteed 'local'
 echo "Hello, $name"
}

Loops Without Brace Expansion#

# Bash
for i in {1..10}; do echo "$i"; done

# POSIX
i=1
while [ "$i" -le 10 ]; do
 echo "$i"
 i=$((i + 1))
done

No Arrays — Alternatives#

# Bash arrays
files=("one.txt" "two.txt" "three.txt")
for f in "${files[@]}"; do echo "$f"; done

# POSIX — use positional parameters
set -- "one.txt" "two.txt" "three.txt"
for f in "$@"; do echo "$f"; done

# POSIX — use newline-separated string
files="one.txt
two.txt
three.txt"
echo "$files" | while IFS= read -r f; do echo "$f"; done

No Process Substitution#

# Bash
diff <(sort file1) <(sort file2)

# POSIX — use temp files
sort file1 > /tmp/sorted1
sort file2 > /tmp/sorted2
diff /tmp/sorted1 /tmp/sorted2
rm /tmp/sorted1 /tmp/sorted2

No Pipefail#

# Bash
set -o pipefail
cmd1 | cmd2

# POSIX — check PIPESTATUS manually (not available)
# Alternative: use intermediate files or check each command
cmd1 > /tmp/intermediate
cmd2 < /tmp/intermediate

When to Use POSIX#

Scenario	Use POSIX	Use Bash
Docker (Alpine, scratch)	✅	❌
System init scripts	✅	❌
Embedded/IoT	✅	❌
Cross-platform distribution	✅	❌
Linux servers (you control)	❌	✅
CI/CD pipelines	❌	✅
Developer tooling	❌	✅
Complex scripts (arrays, regex)	❌	✅

Portability Checklist#

#!/bin/sh
# POSIX-compliant script checklist:

# ✅ Shebang: #!/bin/sh (not #!/bin/bash)
# ✅ Tests: [ ] not [[ ]]
# ✅ Equality: = not ==
# ✅ No arrays
# ✅ No local (or accept it's not guaranteed)
# ✅ No brace expansion {1..10}
# ✅ No here strings <<<
# ✅ No process substitution <()
# ✅ No pipefail
# ✅ No source (use . instead)
# ✅ No function keyword (use name() { })
# ✅ No [[ =~ ]] regex
# ✅ printf over echo (echo behavior varies!)

The `echo` Problem#

echo behavior varies across systems:

Scripting on Software Engineering Learning Paths

Shell Basics

What is a Shell?#

Variables, Quoting, and Expansion

Variables#

Assignment Rules#

Variable Usage#

Variable Types#

Special Variables#

Quoting#

Control Structures

Conditionals: if/elif/else#

Test Commands: [[ ]] vs [ ]#

[[ ]] — Bash Extended Test (Preferred)#

[ ] vs [[ ]] Comparison#

Functions

Defining Functions#

Parameters and Return Values#

Positional Parameters#

Parameter Validation#

Return Values#

Text Processing

The Big Three: grep, sed, awk#

grep — Search for Patterns#

Basic Usage#

Extended Regex (-E)#

Common Patterns#

sed — Stream Editor#

Substitution (Most Common Use)#

Line Operations#

Practical Examples#

Multiple Operations#

awk — Pattern Scanning and Processing#

Column Extraction#

Filtering#

Built-in Variables#

Aggregation#

BEGIN and END Blocks#

cut, sort, uniq — Simple but Powerful#

cut — Extract Columns#

sort#

uniq (requires sorted input)#

Combining Tools#

Key Takeaways#

Pipes, Redirection, and Process Substitution

The Unix Philosophy#

Pipes (|)#

Pipe Behavior#

File Descriptors#

File Operations and Permissions

File Operations#

Creating Files and Directories#

Copying, Moving, Deleting#

Safety with rm#

find — The Swiss Army Knife#

Basic Searches#

Actions#

Combining Conditions#

Permissions#

Understanding Permission Notation#

Octal (Numeric) Permissions#

Common Permission Patterns#

chmod — Change Permissions#

chown — Change Ownership#

Directory Permissions#

Regular Expressions

What Are Regular Expressions?#

Basic Syntax#

Literal Characters and Metacharacters#

Anchors#

Character Classes#

POSIX Character Classes#

Quantifiers#

Grouping and Alternation#

BRE vs ERE vs PCRE#

Process Management

What is a Process?#

Process States#

Viewing Processes#

Useful ps Patterns#

Test Commands: `[[ ]]` vs `[ ]`#

`[[ ]]` — Bash Extended Test (Preferred)#

`[ ]` vs `[[ ]]` Comparison#

Extended Regex (`-E`)#

Pipes (`|`)#

Safety with `rm`#

Useful `ps` Patterns#

Strict Mode: `set -euo pipefail`#

`-e` in Detail#

The `echo` Problem#