Python on Software Engineering Learning Paths

Python: Syntax, Types, and the Interpreter

Mon, 01 Jan 0001 00:00:00 +0000

How Python Executes Code#

The CPython Pipeline#

flowchart LR
 A["source.py"] --> B["Lexer/Tokenizer"]
 B --> C["Parser (AST)"]
 C --> D["Compiler"]
 D --> E["Bytecode (.pyc)"]
 E --> F["CPython VM"]
 F --> G["Execution"]

Lexer — breaks source into tokens (def, foo, (, ), :)
Parser — builds an Abstract Syntax Tree (AST)
Compiler — generates bytecode (stored in __pycache__/)
VM — executes bytecode instruction by instruction

Inspecting Bytecode#

import dis

def add(a, b):
 return a + b

dis.dis(add)
# Output:
# 2 0 LOAD_FAST 0 (a)
# 2 LOAD_FAST 1 (b)
# 4 BINARY_ADD
# 6 RETURN_VALUE

Python Implementations#

Implementation	Language	Use Case
CPython	C	Default, reference implementation
PyPy	Python/RPython	JIT-compiled, 4-10x faster for long-running code
Jython	Java	JVM integration
MicroPython	C	Microcontrollers, embedded systems
GraalPy	Java	GraalVM polyglot

The Type System#

Everything is an Object#

# Even integers are objects
x = 42
type(x) # <class 'int'>
x.bit_length() # 6 (42 in binary = 101010)
id(x) # memory address

# Functions are objects
def greet():
 pass

type(greet) # <class 'function'>
greet.__name__ # 'greet'
greet.__code__.co_varnames # ()

Object Identity, Type, and Value#

a = [1, 2, 3]
b = a # b references the SAME object
c = [1, 2, 3] # c is a DIFFERENT object with same value

a is b # True — same identity (same object in memory)
a is c # False — different identity
a == c # True — same value

id(a) == id(b) # True
id(a) == id(c) # False

Mutability#

Mutable	Immutable
`list`	`tuple`
`dict`	`frozenset`
`set`	`str`
`bytearray`	`bytes`
Custom objects (default)	`int`, `float`, `bool`

Why it matters:

Python: Control Flow, Functions, and Modules

Mon, 01 Jan 0001 00:00:00 +0000

Control Flow#

Conditionals — Beyond Basics#

# Walrus operator (Python 3.8+) — assign and test in one expression
if (n := len(data)) > 100:
 print(f"Processing {n} items in batches")

# Chained comparisons
if 0 <= score <= 100:
 print("Valid score")

# Match statement (Python 3.10+) — structural pattern matching
def handle_command(command):
 match command.split():
 case ["quit"]:
 return "Goodbye"
 case ["hello", name]:
 return f"Hello, {name}!"
 case ["add", *numbers] if all(n.isdigit() for n in numbers):
 return sum(int(n) for n in numbers)
 case _:
 return "Unknown command"

# Pattern matching with objects
match event:
 case {"type": "click", "x": x, "y": y}:
 handle_click(x, y)
 case {"type": "keypress", "key": key}:
 handle_key(key)
 case {"type": "scroll", "direction": "up" | "down" as direction}:
 handle_scroll(direction)

Iteration Patterns#

# zip — iterate multiple sequences in parallel
names = ["Alice", "Bob", "Charlie"]
scores = [95, 87, 92]
for name, score in zip(names, scores):
 print(f"{name}: {score}")

# zip_longest — don't stop at shortest
from itertools import zip_longest
for a, b in zip_longest([1, 2, 3], [10, 20], fillvalue=0):
 print(a, b) # (1,10), (2,20), (3,0)

# enumerate with start
for i, line in enumerate(lines, start=1):
 print(f"Line {i}: {line}")

# reversed and sorted
for item in reversed(items):
 process(item)

for item in sorted(items, key=lambda x: x.priority, reverse=True):
 process(item)

# Dictionary iteration patterns
for key in sorted(config.keys()):
 print(f"{key} = {config[key]}")

# Nested loop with product
from itertools import product
for x, y in product(range(3), range(3)):
 print(f"({x}, {y})")

Comprehensions — Advanced#

# Nested comprehension (flatten)
matrix = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
flat = [x for row in matrix for x in row] # [1, 2, 3, 4, 5, 6, 7, 8, 9]

# Conditional expression in comprehension
labels = ["PASS" if score >= 60 else "FAIL" for score in scores]

# Dictionary comprehension with filtering
env_vars = {k: v for k, v in os.environ.items() if k.startswith("APP_")}

# Set comprehension
unique_domains = {email.split("@")[1] for email in emails}

# Walrus in comprehension (filter + transform)
results = [cleaned for raw in data if (cleaned := clean(raw)) is not None]

Functions — Advanced Patterns#

Parameter Types#

def complex_function(
 required_arg, # positional or keyword
 *args, # variable positional
 keyword_only, # must be passed by name
 default_kwarg="default", # keyword with default
 **kwargs # variable keyword
):
 pass

# Positional-only parameters (Python 3.8+)
def divmod(a, b, /): # a and b MUST be positional
 return a // b, a % b

# Combined
def api_call(url, /, *, method="GET", headers=None, **params):
 # url is positional-only
 # method, headers are keyword-only
 # params catches extra keyword args
 pass

Closures and Factories#

def make_validator(min_val, max_val, required=True):
 """Factory: creates a validation function with captured config."""
 def validate(value):
 if value is None:
 return not required
 return min_val <= value <= max_val
 return validate

validate_age = make_validator(0, 150)
validate_score = make_validator(0, 100)
validate_optional_rating = make_validator(1, 5, required=False)

validate_age(25) # True
validate_age(200) # False
validate_optional_rating(None) # True

Partial Application#

from functools import partial

def power(base, exponent):
 return base ** exponent

square = partial(power, exponent=2)
cube = partial(power, exponent=3)

square(5) # 25
cube(3) # 27

# Practical: configure a logger
import logging
debug = partial(logging.log, logging.DEBUG)
error = partial(logging.log, logging.ERROR)
debug("This is a debug message")

Function Annotations and Overloading#

from typing import overload

@overload
def process(data: str) -> list[str]: ...
@overload
def process(data: list[str]) -> str: ...

def process(data):
 if isinstance(data, str):
 return data.split()
 elif isinstance(data, list):
 return " ".join(data)

Modules and Packages#

Module System#

# Every .py file is a module
# __name__ is "__main__" when run directly, module name when imported

# my_module.py
def helper():
 return "I'm a helper"

_private_var = "not exported by convention"

if __name__ == "__main__":
 # Only runs when executed directly, not when imported
 print(helper())

Package Structure#

my_package/
├── __init__.py # Makes it a package, runs on import
├── core.py
├── utils/
│ ├── __init__.py
│ ├── strings.py
│ └── numbers.py
└── _internal.py # Convention: private module

# __init__.py — control what's exported
from .core import MainClass, main_function
from .utils.strings import clean_text

__all__ = ["MainClass", "main_function", "clean_text"]
# Controls what `from my_package import *` exports

Import Mechanics#

# Python searches for modules in this order:
# 1. sys.modules cache (already imported)
# 2. Built-in modules
# 3. sys.path (current dir, PYTHONPATH, site-packages)

import sys
print(sys.path) # shows search order

# Lazy imports (defer heavy imports)
def process_image(path):
 from PIL import Image # only imported when function is called
 return Image.open(path)

# Conditional imports
try:
 import ujson as json # faster JSON library
except ImportError:
 import json # fallback to stdlib

Hypothetical Use Cases#

Use Case: CLI Argument Parser#

def parse_args(argv: list[str]) -> dict:
 """Simple CLI argument parser using pattern matching."""
 args = {"verbose": False, "output": "stdout", "files": []}
 
 i = 0
 while i < len(argv):
 match argv[i]:
 case "-v" | "--verbose":
 args["verbose"] = True
 case "-o" | "--output":
 i += 1
 args["output"] = argv[i]
 case arg if not arg.startswith("-"):
 args["files"].append(arg)
 case unknown:
 raise ValueError(f"Unknown argument: {unknown}")
 i += 1
 
 return args

Use Case: Plugin System#

import importlib
from pathlib import Path

def load_plugins(plugin_dir: str) -> dict:
 """Dynamically load all plugins from a directory."""
 plugins = {}
 
 for path in Path(plugin_dir).glob("*.py"):
 if path.name.startswith("_"):
 continue
 
 module_name = path.stem
 spec = importlib.util.spec_from_file_location(module_name, path)
 module = importlib.util.module_from_spec(spec)
 spec.loader.exec_module(module)
 
 # Each plugin must have a register() function
 if hasattr(module, "register"):
 plugin_info = module.register()
 plugins[plugin_info["name"]] = module
 
 return plugins

Key Takeaways#

Pattern matching (3.10+) replaces complex if/elif chains for structural data
Comprehensions are idiomatic Python — prefer them over manual loops for transforms
Closures and factories create configured functions without classes
Positional-only and keyword-only parameters make APIs clearer
__init__.py controls package exports — use __all__ for explicit public API
Lazy imports improve startup time for heavy dependencies

Python: Object-Oriented Programming

Mon, 01 Jan 0001 00:00:00 +0000

Classes — The Fundamentals#

Anatomy of a Class#

class BankAccount:
 """A simple bank account with deposit/withdraw operations."""
 
 # Class variable — shared across ALL instances
 bank_name = "Python National Bank"
 _total_accounts = 0
 
 def __init__(self, owner: str, balance: float = 0.0):
 """Instance initializer (NOT a constructor — __new__ is the constructor)."""
 # Instance variables — unique to each instance
 self.owner = owner
 self._balance = balance # Convention: "protected"
 self.__id = id(self) # Name-mangled: becomes _BankAccount__id
 BankAccount._total_accounts += 1
 
 @property
 def balance(self) -> float:
 """Read-only property — controlled access to _balance."""
 return self._balance
 
 @balance.setter
 def balance(self, value: float):
 if value < 0:
 raise ValueError("Balance cannot be negative")
 self._balance = value
 
 def deposit(self, amount: float) -> float:
 """Instance method — operates on self."""
 if amount <= 0:
 raise ValueError("Deposit must be positive")
 self._balance += amount
 return self._balance
 
 def withdraw(self, amount: float) -> float:
 if amount > self._balance:
 raise InsufficientFundsError(self._balance, amount)
 self._balance -= amount
 return self._balance
 
 @classmethod
 def from_dict(cls, data: dict) -> "BankAccount":
 """Alternative constructor — creates instance from dictionary."""
 return cls(owner=data["owner"], balance=data.get("balance", 0))
 
 @classmethod
 def get_total_accounts(cls) -> int:
 """Access class-level state."""
 return cls._total_accounts
 
 @staticmethod
 def validate_amount(amount: float) -> bool:
 """Utility — no access to instance or class state."""
 return isinstance(amount, (int, float)) and amount > 0
 
 def __repr__(self) -> str:
 """Unambiguous representation (for developers)."""
 return f"BankAccount(owner={self.owner!r}, balance={self._balance})"
 
 def __str__(self) -> str:
 """Human-readable representation."""
 return f"{self.owner}'s account: ${self._balance:,.2f}"

Instance vs Class vs Static Methods#

flowchart TD
 A["Method Types"] --> B["Instance Method"]
 A --> C["Class Method"]
 A --> D["Static Method"]
 
 B --> B1["def method(self)"]
 B --> B2["Access instance + class state"]
 B --> B3["Most common"]
 
 C --> C1["@classmethod<br>def method(cls)"]
 C --> C2["Access class state only"]
 C --> C3["Alternative constructors"]
 
 D --> D1["@staticmethod<br>def method()"]
 D --> D2["No access to instance or class"]
 D --> D3["Utility functions"]

The Data Model — Dunder Methods#

Making Objects Behave Like Built-in Types#

class Vector:
 """2D vector with full operator support."""
 
 __slots__ = ('x', 'y') # Memory optimization — no __dict__
 
 def __init__(self, x: float, y: float):
 self.x = x
 self.y = y
 
 # Arithmetic operators
 def __add__(self, other: "Vector") -> "Vector":
 return Vector(self.x + other.x, self.y + other.y)
 
 def __sub__(self, other: "Vector") -> "Vector":
 return Vector(self.x - other.x, self.y - other.y)
 
 def __mul__(self, scalar: float) -> "Vector":
 return Vector(self.x * scalar, self.y * scalar)
 
 def __rmul__(self, scalar: float) -> "Vector":
 """Handles: 3 * vector (scalar on the left)."""
 return self.__mul__(scalar)
 
 def __abs__(self) -> float:
 """Magnitude: abs(vector)."""
 return (self.x ** 2 + self.y ** 2) ** 0.5
 
 # Comparison
 def __eq__(self, other: object) -> bool:
 if not isinstance(other, Vector):
 return NotImplemented
 return self.x == other.x and self.y == other.y
 
 def __lt__(self, other: "Vector") -> bool:
 return abs(self) < abs(other)
 
 # Container protocol
 def __len__(self) -> int:
 return 2
 
 def __getitem__(self, index: int) -> float:
 match index:
 case 0: return self.x
 case 1: return self.y
 case _: raise IndexError(f"Vector index {index} out of range")
 
 def __iter__(self):
 yield self.x
 yield self.y
 
 # Hashing (required for use in sets/dict keys)
 def __hash__(self) -> int:
 return hash((self.x, self.y))
 
 def __repr__(self) -> str:
 return f"Vector({self.x}, {self.y})"
 
 def __format__(self, spec: str) -> str:
 if spec == "polar":
 import math
 r = abs(self)
 theta = math.atan2(self.y, self.x)
 return f"({r:.4f}, {math.degrees(theta):.2f}°)"
 return f"({self.x}, {self.y})"


# Usage
v1 = Vector(3, 4)
v2 = Vector(1, 2)

v3 = v1 + v2 # Vector(4, 6)
v4 = v1 * 2 # Vector(6, 8)
v5 = 3 * v1 # Vector(9, 12) — uses __rmul__
magnitude = abs(v1) # 5.0
x, y = v1 # unpacking via __iter__
print(f"{v1:polar}") # (5.0000, 53.13°)

Key Dunder Methods Reference#

Category	Methods	Purpose
Creation	`__new__`, `__init__`, `__del__`	Object lifecycle
Representation	`__repr__`, `__str__`, `__format__`	String conversion
Comparison	`__eq__`, `__lt__`, `__le__`, `__gt__`, `__ge__`	Ordering
Arithmetic	`__add__`, `__sub__`, `__mul__`, `__truediv__`	Operators
Container	`__len__`, `__getitem__`, `__setitem__`, `__contains__`	Sequence/mapping
Callable	`__call__`	`obj()` syntax
Context	`__enter__`, `__exit__`	`with` statement
Attribute	`__getattr__`, `__setattr__`, `__delattr__`	Attribute access
Hashing	`__hash__`	Sets, dict keys

Inheritance and Composition#

Method Resolution Order (MRO)#

class A:
 def method(self):
 return "A"

class B(A):
 def method(self):
 return "B"

class C(A):
 def method(self):
 return "C"

class D(B, C):
 pass

# MRO: D → B → C → A → object
print(D.__mro__)
# (<class 'D'>, <class 'B'>, <class 'C'>, <class 'A'>, <class 'object'>)

D().method() # "B" — follows MRO left to right

graph BT
 D --> B
 D --> C
 B --> A
 C --> A
 A --> object
 
 style D fill:#e1f5fe
 style B fill:#fff3e0
 style C fill:#fff3e0
 style A fill:#f3e5f5

Cooperative Multiple Inheritance with super()#

class Serializable:
 def serialize(self) -> dict:
 return {"type": type(self).__name__}

class Timestamped:
 def __init__(self, **kwargs):
 super().__init__(**kwargs)
 from datetime import datetime
 self.created_at = datetime.now()
 
 def serialize(self) -> dict:
 data = super().serialize()
 data["created_at"] = self.created_at.isoformat()
 return data

class Identifiable:
 def __init__(self, **kwargs):
 super().__init__(**kwargs)
 import uuid
 self.id = str(uuid.uuid4())
 
 def serialize(self) -> dict:
 data = super().serialize()
 data["id"] = self.id
 return data

class Event(Identifiable, Timestamped, Serializable):
 def __init__(self, name: str, **kwargs):
 super().__init__(**kwargs)
 self.name = name
 
 def serialize(self) -> dict:
 data = super().serialize()
 data["name"] = self.name
 return data

event = Event("user_signup")
event.serialize()
# {"type": "Event", "created_at": "...", "id": "...", "name": "user_signup"}

Composition Over Inheritance#

# BAD: Deep inheritance hierarchy
class Animal:
 def eat(self): ...
class FlyingAnimal(Animal):
 def fly(self): ...
class SwimmingAnimal(Animal):
 def swim(self): ...
class Duck(FlyingAnimal, SwimmingAnimal): # Diamond problem
 pass

# GOOD: Composition with protocols
from typing import Protocol

class CanFly(Protocol):
 def fly(self) -> str: ...

class CanSwim(Protocol):
 def swim(self) -> str: ...

class Wings:
 def fly(self) -> str:
 return "Flapping wings"

class Fins:
 def swim(self) -> str:
 return "Using fins"

class Webbed:
 def swim(self) -> str:
 return "Paddling with webbed feet"

class Duck:
 def __init__(self):
 self._flying = Wings()
 self._swimming = Webbed()
 
 def fly(self) -> str:
 return self._flying.fly()
 
 def swim(self) -> str:
 return self._swimming.swim()

Abstract Base Classes#

from abc import ABC, abstractmethod

class Shape(ABC):
 """Abstract base class — cannot be instantiated directly."""
 
 @abstractmethod
 def area(self) -> float:
 """Subclasses MUST implement this."""
 ...
 
 @abstractmethod
 def perimeter(self) -> float:
 ...
 
 def describe(self) -> str:
 """Concrete method — inherited by all subclasses."""
 return f"{type(self).__name__}: area={self.area():.2f}, perimeter={self.perimeter():.2f}"

class Circle(Shape):
 def __init__(self, radius: float):
 self.radius = radius
 
 def area(self) -> float:
 import math
 return math.pi * self.radius ** 2
 
 def perimeter(self) -> float:
 import math
 return 2 * math.pi * self.radius

# Shape() # TypeError: Can't instantiate abstract class
c = Circle(5)
c.describe() # "Circle: area=78.54, perimeter=31.42"

Dataclasses — Modern Python OOP#

from dataclasses import dataclass, field, asdict, astuple
from typing import ClassVar

@dataclass(frozen=True, slots=True) # Immutable + memory efficient
class Coordinate:
 latitude: float
 longitude: float
 
 def distance_to(self, other: "Coordinate") -> float:
 """Haversine distance in km."""
 import math
 R = 6371 # Earth radius in km
 dlat = math.radians(other.latitude - self.latitude)
 dlon = math.radians(other.longitude - self.longitude)
 a = (math.sin(dlat/2)**2 + 
 math.cos(math.radians(self.latitude)) * 
 math.cos(math.radians(other.latitude)) * 
 math.sin(dlon/2)**2)
 return R * 2 * math.asin(math.sqrt(a))

@dataclass
class Order:
 id: str
 customer: str
 items: list[str] = field(default_factory=list) # Mutable default
 total: float = 0.0
 _internal: str = field(default="", repr=False, compare=False)
 
 # Class variable (not a field)
 MAX_ITEMS: ClassVar[int] = 100
 
 def __post_init__(self):
 """Validation after __init__."""
 if self.total < 0:
 raise ValueError("Total cannot be negative")
 if len(self.items) > self.MAX_ITEMS:
 raise ValueError(f"Max {self.MAX_ITEMS} items per order")

# Auto-generated: __init__, __repr__, __eq__, __hash__ (if frozen)
paris = Coordinate(48.8566, 2.3522)
london = Coordinate(51.5074, -0.1278)
paris.distance_to(london) # ~343.5 km

order = Order(id="ORD-001", customer="Alice", items=["Widget"])
asdict(order) # Convert to dictionary

Descriptors — The Mechanism Behind Properties#

class Validated:
 """Descriptor that validates values on assignment."""
 
 def __init__(self, validator, error_msg="Invalid value"):
 self.validator = validator
 self.error_msg = error_msg
 self.attr_name = None # Set by __set_name__
 
 def __set_name__(self, owner, name):
 self.attr_name = f"_{name}"
 
 def __get__(self, obj, objtype=None):
 if obj is None:
 return self
 return getattr(obj, self.attr_name, None)
 
 def __set__(self, obj, value):
 if not self.validator(value):
 raise ValueError(f"{self.error_msg}: {value!r}")
 setattr(obj, self.attr_name, value)

class PositiveNumber(Validated):
 def __init__(self):
 super().__init__(
 validator=lambda x: isinstance(x, (int, float)) and x > 0,
 error_msg="Must be a positive number"
 )

class NonEmptyString(Validated):
 def __init__(self):
 super().__init__(
 validator=lambda x: isinstance(x, str) and len(x.strip()) > 0,
 error_msg="Must be a non-empty string"
 )

class Product:
 name = NonEmptyString()
 price = PositiveNumber()
 quantity = PositiveNumber()
 
 def __init__(self, name: str, price: float, quantity: int):
 self.name = name # Triggers NonEmptyString.__set__
 self.price = price # Triggers PositiveNumber.__set__
 self.quantity = quantity

p = Product("Widget", 9.99, 100) # OK
# Product("", 9.99, 100) # ValueError: Must be a non-empty string
# Product("X", -1, 100) # ValueError: Must be a positive number

Metaclasses — Classes That Create Classes#

class SingletonMeta(type):
 """Metaclass that ensures only one instance of a class exists."""
 _instances = {}
 
 def __call__(cls, *args, **kwargs):
 if cls not in cls._instances:
 cls._instances[cls] = super().__call__(*args, **kwargs)
 return cls._instances[cls]

class DatabaseConnection(metaclass=SingletonMeta):
 def __init__(self, url: str = "localhost:5432"):
 self.url = url
 self.connected = False
 
 def connect(self):
 self.connected = True

# Always returns the same instance
db1 = DatabaseConnection("prod:5432")
db2 = DatabaseConnection("other:5432") # Ignored — returns existing
db1 is db2 # True

flowchart TD
 A["type (metaclass)"] -->|"creates"| B["MyClass (class)"]
 B -->|"creates"| C["instance (object)"]
 
 A -->|"type(MyClass) → type"| A
 B -->|"type(instance) → MyClass"| B
 C -->|"isinstance(instance, MyClass) → True"| B

Hypothetical Use Cases#

Use Case: Domain Model with Validation#

from dataclasses import dataclass, field
from datetime import datetime, date
from enum import Enum, auto

class OrderStatus(Enum):
 PENDING = auto()
 CONFIRMED = auto()
 SHIPPED = auto()
 DELIVERED = auto()
 CANCELLED = auto()

@dataclass
class OrderItem:
 product_id: str
 name: str
 quantity: int
 unit_price: float
 
 @property
 def subtotal(self) -> float:
 return self.quantity * self.unit_price

@dataclass
class Order:
 id: str
 customer_id: str
 items: list[OrderItem] = field(default_factory=list)
 status: OrderStatus = OrderStatus.PENDING
 created_at: datetime = field(default_factory=datetime.now)
 shipped_at: datetime | None = None
 
 @property
 def total(self) -> float:
 return sum(item.subtotal for item in self.items)
 
 def add_item(self, item: OrderItem) -> None:
 if self.status != OrderStatus.PENDING:
 raise InvalidOperationError(f"Cannot modify {self.status.name} order")
 self.items.append(item)
 
 def confirm(self) -> None:
 if not self.items:
 raise InvalidOperationError("Cannot confirm empty order")
 self._transition(OrderStatus.CONFIRMED)
 
 def ship(self) -> None:
 self._transition(OrderStatus.SHIPPED)
 self.shipped_at = datetime.now()
 
 def cancel(self) -> None:
 if self.status in (OrderStatus.SHIPPED, OrderStatus.DELIVERED):
 raise InvalidOperationError("Cannot cancel shipped/delivered order")
 self._transition(OrderStatus.CANCELLED)
 
 def _transition(self, new_status: OrderStatus) -> None:
 valid_transitions = {
 OrderStatus.PENDING: {OrderStatus.CONFIRMED, OrderStatus.CANCELLED},
 OrderStatus.CONFIRMED: {OrderStatus.SHIPPED, OrderStatus.CANCELLED},
 OrderStatus.SHIPPED: {OrderStatus.DELIVERED},
 }
 allowed = valid_transitions.get(self.status, set())
 if new_status not in allowed:
 raise InvalidOperationError(
 f"Cannot transition from {self.status.name} to {new_status.name}"
 )
 self.status = new_status

Use Case: Registry Pattern with Metaclass#

class PluginRegistry(type):
 """Auto-registers all subclasses of a base class."""
 _registry: dict[str, type] = {}
 
 def __init_subclass__(cls, **kwargs):
 super().__init_subclass__(**kwargs)
 if hasattr(cls, "name"):
 PluginRegistry._registry[cls.name] = cls
 
 @classmethod
 def get(mcs, name: str):
 return mcs._registry.get(name)
 
 @classmethod
 def list_plugins(mcs) -> list[str]:
 return list(mcs._registry.keys())

class Exporter(metaclass=PluginRegistry):
 """Base class — subclasses auto-register."""
 def export(self, data) -> bytes:
 raise NotImplementedError

class JSONExporter(Exporter):
 name = "json"
 def export(self, data) -> bytes:
 import json
 return json.dumps(data).encode()

class CSVExporter(Exporter):
 name = "csv"
 def export(self, data) -> bytes:
 # ... CSV export logic
 pass

# Usage
exporter_cls = PluginRegistry.get("json")
exporter = exporter_cls()
result = exporter.export({"key": "value"})

Key Takeaways#

Properties provide controlled attribute access without changing the API
__slots__ saves memory by eliminating per-instance __dict__
Dataclasses eliminate boilerplate for data-holding classes
Composition > inheritance — prefer has-a over is-a relationships
Descriptors are the mechanism behind property, classmethod, staticmethod
Metaclasses are rarely needed — use __init_subclass__ for most registration patterns
ABC enforces interface contracts at instantiation time, not at call time

Python: The Standard Library

Mon, 01 Jan 0001 00:00:00 +0000

File I/O and Pathlib#

pathlib — Object-Oriented Filesystem Paths#

from pathlib import Path

# Creating paths (OS-independent)
home = Path.home() # /Users/username
project = Path.cwd() / "src" / "main.py" # Operator / joins paths
config = Path("/etc/nginx/nginx.conf")

# Path inspection
project.name # "main.py"
project.stem # "main"
project.suffix # ".py"
project.parent # PosixPath('/Users/.../src')
project.parts # ('/', 'Users', '...', 'src', 'main.py')
project.is_absolute() # True

# File operations
path = Path("data/output.json")
path.parent.mkdir(parents=True, exist_ok=True) # Create directories
path.write_text('{"key": "value"}', encoding="utf-8")
content = path.read_text(encoding="utf-8")
path.write_bytes(b'\x00\x01\x02')

# Globbing
for py_file in Path("src").rglob("*.py"): # Recursive
 print(py_file)

# Temporary files
import tempfile
with tempfile.NamedTemporaryFile(suffix=".json", delete=False) as f:
 f.write(b'{"temp": true}')
 temp_path = Path(f.name)

File I/O Patterns#

# Reading large files line by line (memory efficient)
def count_errors(log_path: Path) -> int:
 count = 0
 with open(log_path, "r", encoding="utf-8") as f:
 for line in f: # Lazy iteration — one line in memory at a time
 if "ERROR" in line:
 count += 1
 return count

# Binary file handling
def copy_file(src: Path, dst: Path, chunk_size: int = 8192) -> None:
 with open(src, "rb") as fsrc, open(dst, "wb") as fdst:
 while chunk := fsrc.read(chunk_size):
 fdst.write(chunk)

Collections Module#

from collections import (
 defaultdict, Counter, deque, OrderedDict, ChainMap, namedtuple
)

# defaultdict — auto-creates missing keys
word_index = defaultdict(list)
for i, word in enumerate(words):
 word_index[word].append(i) # No KeyError if word is new

graph = defaultdict(set)
graph["A"].add("B")
graph["A"].add("C")

# Counter — frequency counting
text = "the quick brown fox jumps over the lazy dog"
word_counts = Counter(text.split())
word_counts.most_common(3) # [('the', 2), ('quick', 1), ('brown', 1)]

# Arithmetic on Counters
inventory = Counter(apples=5, oranges=3)
sold = Counter(apples=2, oranges=1)
remaining = inventory - sold # Counter({'apples': 3, 'oranges': 2})

# deque — O(1) append/pop from both ends
from collections import deque
buffer = deque(maxlen=100) # Fixed-size circular buffer
buffer.append("event_1")
buffer.appendleft("priority_event")
buffer.popleft() # O(1) vs list.pop(0) which is O(n)

# ChainMap — layered dictionaries (first match wins)
defaults = {"color": "blue", "size": "medium", "debug": False}
user_prefs = {"color": "red"}
env_config = {"debug": True}
config = ChainMap(env_config, user_prefs, defaults)
config["color"] # "red" (from user_prefs)
config["debug"] # True (from env_config)
config["size"] # "medium" (from defaults)

itertools — Efficient Iteration#

import itertools

# Infinite iterators
counter = itertools.count(start=1, step=2) # 1, 3, 5, 7, ...
cycler = itertools.cycle(["red", "green", "blue"]) # repeats forever
repeater = itertools.repeat("hello", times=3) # "hello" x3

# Combinatorics
list(itertools.permutations([1, 2, 3], r=2))
# [(1,2), (1,3), (2,1), (2,3), (3,1), (3,2)]

list(itertools.combinations([1, 2, 3, 4], r=2))
# [(1,2), (1,3), (1,4), (2,3), (2,4), (3,4)]

list(itertools.product("AB", "12"))
# [('A','1'), ('A','2'), ('B','1'), ('B','2')]

# Grouping
data = [
 {"dept": "eng", "name": "Alice"},
 {"dept": "eng", "name": "Bob"},
 {"dept": "sales", "name": "Charlie"},
 {"dept": "sales", "name": "Diana"},
]
# MUST be sorted by key first!
data.sort(key=lambda x: x["dept"])
for dept, members in itertools.groupby(data, key=lambda x: x["dept"]):
 print(f"{dept}: {[m['name'] for m in members]}")

# Chain — flatten iterables
all_items = list(itertools.chain([1, 2], [3, 4], [5, 6])) # [1,2,3,4,5,6]
all_items = list(itertools.chain.from_iterable(list_of_lists))

# Slicing iterators
first_10 = list(itertools.islice(infinite_gen(), 10))

# Accumulate (running totals)
list(itertools.accumulate([1, 2, 3, 4, 5])) # [1, 3, 6, 10, 15]
list(itertools.accumulate([3, 1, 4, 1, 5], max)) # [3, 3, 4, 4, 5]

functools — Higher-Order Functions#

from functools import lru_cache, cache, reduce, wraps, total_ordering

# Caching expensive computations
@lru_cache(maxsize=128)
def fibonacci(n: int) -> int:
 if n < 2:
 return n
 return fibonacci(n - 1) + fibonacci(n - 2)

fibonacci(100) # Instant — without cache this would take forever
fibonacci.cache_info() # CacheInfo(hits=98, misses=101, maxsize=128, currsize=101)

# @cache — unlimited cache (Python 3.9+)
@cache
def expensive_lookup(key: str) -> dict:
 # Simulates slow database/API call
 return fetch_from_database(key)

# reduce — fold a sequence into a single value
from functools import reduce
product = reduce(lambda a, b: a * b, [1, 2, 3, 4, 5]) # 120

# total_ordering — generate comparison methods from __eq__ and one other
@total_ordering
class Version:
 def __init__(self, major, minor, patch):
 self.major, self.minor, self.patch = major, minor, patch
 
 def __eq__(self, other):
 return (self.major, self.minor, self.patch) == (other.major, other.minor, other.patch)
 
 def __lt__(self, other):
 return (self.major, self.minor, self.patch) < (other.major, other.minor, other.patch)

# Now Version also has __le__, __gt__, __ge__ automatically

datetime and Time#

from datetime import datetime, date, timedelta, timezone
from zoneinfo import ZoneInfo # Python 3.9+

# Current time
now = datetime.now() # Naive (no timezone)
now_utc = datetime.now(timezone.utc) # Aware (UTC)
now_madrid = datetime.now(ZoneInfo("Europe/Madrid"))

# Parsing and formatting
dt = datetime.fromisoformat("2026-05-29T12:00:00+02:00")
dt.strftime("%Y-%m-%d %H:%M:%S %Z") # "2026-05-29 12:00:00 CEST"

# Arithmetic
tomorrow = now + timedelta(days=1)
next_week = now + timedelta(weeks=1)
duration = datetime(2026, 12, 31) - datetime(2026, 1, 1)
duration.days # 364

# Timezone conversion
utc_time = datetime(2026, 5, 29, 10, 0, tzinfo=timezone.utc)
tokyo_time = utc_time.astimezone(ZoneInfo("Asia/Tokyo"))
# 2026-05-29 19:00:00+09:00

JSON, CSV, and Data Serialization#

import json
import csv
from dataclasses import dataclass, asdict

# JSON with custom serialization
class CustomEncoder(json.JSONEncoder):
 def default(self, obj):
 if isinstance(obj, datetime):
 return obj.isoformat()
 if isinstance(obj, set):
 return list(obj)
 if hasattr(obj, "__dict__"):
 return obj.__dict__
 return super().default(obj)

data = {"timestamp": datetime.now(), "tags": {"python", "tutorial"}}
json.dumps(data, cls=CustomEncoder, indent=2)

# CSV reading/writing
def read_csv_as_dicts(path: Path) -> list[dict]:
 with open(path, newline="", encoding="utf-8") as f:
 reader = csv.DictReader(f)
 return list(reader)

def write_csv(path: Path, rows: list[dict]) -> None:
 if not rows:
 return
 with open(path, "w", newline="", encoding="utf-8") as f:
 writer = csv.DictWriter(f, fieldnames=rows[0].keys())
 writer.writeheader()
 writer.writerows(rows)

subprocess — Running External Commands#

import subprocess

# Simple command execution
result = subprocess.run(
 ["git", "status", "--short"],
 capture_output=True,
 text=True,
 check=True, # Raises CalledProcessError on non-zero exit
 timeout=30,
)
print(result.stdout)

# Piping commands
ps = subprocess.run(["ps", "aux"], capture_output=True, text=True)
grep = subprocess.run(
 ["grep", "python"],
 input=ps.stdout,
 capture_output=True,
 text=True,
)

# Streaming output
process = subprocess.Popen(
 ["tail", "-f", "/var/log/app.log"],
 stdout=subprocess.PIPE,
 text=True,
)
for line in process.stdout:
 if "ERROR" in line:
 alert(line)

logging — Structured Logging#

import logging
import logging.handlers
import json

# Basic configuration
logging.basicConfig(
 level=logging.INFO,
 format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
 datefmt="%Y-%m-%d %H:%M:%S",
)

# Module-level logger (best practice)
logger = logging.getLogger(__name__)

# Structured JSON logging
class JSONFormatter(logging.Formatter):
 def format(self, record):
 log_data = {
 "timestamp": self.formatTime(record),
 "level": record.levelname,
 "logger": record.name,
 "message": record.getMessage(),
 }
 if record.exc_info:
 log_data["exception"] = self.formatException(record.exc_info)
 if hasattr(record, "extra_data"):
 log_data.update(record.extra_data)
 return json.dumps(log_data)

# Rotating file handler
handler = logging.handlers.RotatingFileHandler(
 "app.log", maxBytes=10_000_000, backupCount=5
)
handler.setFormatter(JSONFormatter())
logger.addHandler(handler)

# Usage with context
logger.info("Order processed", extra={"extra_data": {"order_id": "ORD-123", "total": 99.99}})

re — Regular Expressions#

import re

# Common patterns
email_pattern = re.compile(r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}")
ip_pattern = re.compile(r"\b(?:\d{1,3}\.){3}\d{1,3}\b")
url_pattern = re.compile(r"https?://[^\s<>\"]+")

# Named groups
log_pattern = re.compile(
 r"(?P<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})"
 r" \[(?P<level>\w+)\]"
 r" (?P<message>.*)"
)

line = "2026-05-29 12:00:00 [ERROR] Connection timeout"
match = log_pattern.match(line)
if match:
 match.group("level") # "ERROR"
 match.group("message") # "Connection timeout"
 match.groupdict() # {"timestamp": "...", "level": "ERROR", "message": "..."}

# Substitution
cleaned = re.sub(r"\s+", " ", messy_text) # Collapse whitespace
redacted = re.sub(r"\b\d{4}-\d{4}-\d{4}-\d{4}\b", "XXXX-XXXX-XXXX-XXXX", text)

# findall vs finditer
emails = re.findall(email_pattern, document) # List of all matches
for match in re.finditer(email_pattern, document): # Iterator of Match objects
 print(f"Found {match.group()} at position {match.start()}")

Hypothetical Use Cases#

Use Case: Log Analyzer#

from pathlib import Path
from collections import Counter, defaultdict
from datetime import datetime
import re

def analyze_logs(log_path: Path) -> dict:
 """Parse and analyze application logs."""
 pattern = re.compile(
 r"(?P<ts>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2})"
 r" (?P<level>DEBUG|INFO|WARN|ERROR|FATAL)"
 r" \[(?P<service>[\w-]+)\]"
 r" (?P<msg>.*)"
 )
 
 level_counts = Counter()
 errors_by_service = defaultdict(list)
 hourly_distribution = Counter()
 
 with open(log_path, encoding="utf-8") as f:
 for line in f:
 if m := pattern.match(line):
 level = m.group("level")
 service = m.group("service")
 ts = datetime.fromisoformat(m.group("ts"))
 
 level_counts[level] += 1
 hourly_distribution[ts.hour] += 1
 
 if level in ("ERROR", "FATAL"):
 errors_by_service[service].append({
 "timestamp": ts,
 "message": m.group("msg"),
 })
 
 return {
 "total_lines": sum(level_counts.values()),
 "level_distribution": dict(level_counts),
 "peak_hour": hourly_distribution.most_common(1)[0],
 "error_services": {
 svc: len(errs) for svc, errs in errors_by_service.items()
 },
 }

Key Takeaways#

pathlib replaces os.path — use / operator for path joining
defaultdict eliminates key-existence checks; Counter handles frequency counting
itertools provides memory-efficient iteration — never build full lists when you can iterate
lru_cache gives free memoization for pure functions
Always use encoding="utf-8" explicitly in file operations
logging over print — configure once at application entry point, use getLogger(__name__) everywhere
subprocess.run with check=True and capture_output=True is the modern way to run commands

Python: Error Handling and Exceptions

Mon, 01 Jan 0001 00:00:00 +0000

The Exception Hierarchy#

graph TD
 BE["BaseException"] --> SE["SystemExit"]
 BE --> KI["KeyboardInterrupt"]
 BE --> GE["GeneratorExit"]
 BE --> EX["Exception"]
 
 EX --> AE["ArithmeticError"]
 EX --> LE["LookupError"]
 EX --> OSE["OSError"]
 EX --> VE["ValueError"]
 EX --> TE["TypeError"]
 EX --> RE["RuntimeError"]
 EX --> ATE["AttributeError"]
 
 AE --> ZDE["ZeroDivisionError"]
 AE --> OFE["OverflowError"]
 
 LE --> IE["IndexError"]
 LE --> KE["KeyError"]
 
 OSE --> FNF["FileNotFoundError"]
 OSE --> PE["PermissionError"]
 OSE --> CE["ConnectionError"]
 
 CE --> CRE["ConnectionRefusedError"]
 CE --> CTO["TimeoutError"]

Rule: Never catch BaseException — it swallows SystemExit and KeyboardInterrupt.

Python: Decorators, Generators, and Context Managers

Mon, 01 Jan 0001 00:00:00 +0000

Decorators — Functions That Modify Functions#

How Decorators Work#

# A decorator is a function that takes a function and returns a function
def my_decorator(func):
 def wrapper(*args, **kwargs):
 # Before the original function
 print(f"Calling {func.__name__}")
 result = func(*args, **kwargs)
 # After the original function
 print(f"{func.__name__} returned {result}")
 return result
 return wrapper

@my_decorator
def add(a, b):
 return a + b

# @my_decorator is syntactic sugar for:
# add = my_decorator(add)

add(3, 4)
# Output:
# Calling add
# Returned 7

sequenceDiagram
 participant Caller
 participant Wrapper
 participant Original
 
 Caller->>Wrapper: add(3, 4)
 Note over Wrapper: Pre-processing
 Wrapper->>Original: func(3, 4)
 Original-->>Wrapper: return 7
 Note over Wrapper: Post-processing
 Wrapper-->>Caller: return 7

Preserving Function Metadata#

from functools import wraps

def timer(func):
 """Decorator that measures execution time."""
 @wraps(func) # Preserves __name__, __doc__, __module__
 def wrapper(*args, **kwargs):
 import time
 start = time.perf_counter()
 result = func(*args, **kwargs)
 elapsed = time.perf_counter() - start
 print(f"{func.__name__} took {elapsed:.4f}s")
 return result
 return wrapper

@timer
def slow_function():
 """This docstring is preserved thanks to @wraps."""
 import time
 time.sleep(1)

slow_function.__name__ # "slow_function" (not "wrapper")
slow_function.__doc__ # "This docstring is preserved..."

Decorators with Arguments#

from functools import wraps

def retry(max_attempts: int = 3, exceptions: tuple = (Exception,)):
 """Decorator factory — returns a decorator configured with arguments."""
 def decorator(func):
 @wraps(func)
 def wrapper(*args, **kwargs):
 for attempt in range(1, max_attempts + 1):
 try:
 return func(*args, **kwargs)
 except exceptions as e:
 if attempt == max_attempts:
 raise
 print(f"Attempt {attempt} failed: {e}")
 return wrapper
 return decorator

@retry(max_attempts=5, exceptions=(ConnectionError, TimeoutError))
def fetch_data(url):
 ...

# Three levels:
# retry(max_attempts=5) → returns decorator
# decorator(fetch_data) → returns wrapper
# wrapper(url) → calls fetch_data with retry logic

Class-Based Decorators#

class CacheResult:
 """Decorator as a class — useful when you need state."""
 
 def __init__(self, ttl_seconds: int = 300):
 self.ttl = ttl_seconds
 self.cache = {}
 
 def __call__(self, func):
 @wraps(func)
 def wrapper(*args, **kwargs):
 import time
 key = (args, tuple(sorted(kwargs.items())))
 
 if key in self.cache:
 result, timestamp = self.cache[key]
 if time.time() - timestamp < self.ttl:
 return result
 
 result = func(*args, **kwargs)
 self.cache[key] = (result, time.time())
 return result
 
 wrapper.clear_cache = self.cache.clear
 return wrapper
 
 def __repr__(self):
 return f"CacheResult(ttl={self.ttl}, entries={len(self.cache)})"

@CacheResult(ttl_seconds=60)
def expensive_query(user_id: int) -> dict:
 ...

expensive_query.clear_cache() # Exposed method

Stacking Decorators#

# Decorators apply bottom-up (closest to function first)
@log_calls # 3rd: wraps the authenticated+validated function
@authenticate # 2nd: wraps the validated function
@validate_input # 1st: wraps the original function
def create_user(data: dict) -> User:
 ...

# Equivalent to:
# create_user = log_calls(authenticate(validate_input(create_user)))

Generators — Lazy Evaluation#

Generator Functions#

def fibonacci():
 """Infinite Fibonacci sequence — generates values on demand."""
 a, b = 0, 1
 while True:
 yield a
 a, b = b, a + b

# Only computes values as requested
fib = fibonacci()
next(fib) # 0
next(fib) # 1
next(fib) # 1
next(fib) # 2

# Take first 10
from itertools import islice
first_10 = list(islice(fibonacci(), 10))
# [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]

Generator vs List — Memory Comparison#

import sys

# List: ALL values in memory at once
squares_list = [x**2 for x in range(1_000_000)]
sys.getsizeof(squares_list) # ~8.4 MB

# Generator: ONE value at a time
squares_gen = (x**2 for x in range(1_000_000))
sys.getsizeof(squares_gen) # ~200 bytes (constant!)

# Both produce the same results
sum(x**2 for x in range(1_000_000)) # Generator expression in function call

yield from — Delegating to Sub-generators#

def flatten(nested):
 """Recursively flatten any nested iterable."""
 for item in nested:
 if isinstance(item, (list, tuple, set)):
 yield from flatten(item) # Delegate to recursive call
 else:
 yield item

list(flatten([1, [2, 3], [4, [5, 6]], 7]))
# [1, 2, 3, 4, 5, 6, 7]

# yield from also forwards .send() and .throw() to sub-generator
def chain(*iterables):
 for it in iterables:
 yield from it

list(chain([1, 2], [3, 4], [5, 6])) # [1, 2, 3, 4, 5, 6]

Generators as Coroutines (send/throw)#

def running_average():
 """Generator that accepts values via .send() and yields running average."""
 total = 0.0
 count = 0
 average = None
 
 while True:
 value = yield average # Receive value, yield current average
 if value is None:
 break
 total += value
 count += 1
 average = total / count

avg = running_average()
next(avg) # Prime the generator (advance to first yield)
avg.send(10) # 10.0
avg.send(20) # 15.0
avg.send(30) # 20.0
avg.send(None) # StopIteration

Practical Generator Patterns#

def read_in_chunks(file_path: str, chunk_size: int = 8192):
 """Read a large file in chunks without loading it all into memory."""
 with open(file_path, "rb") as f:
 while chunk := f.read(chunk_size):
 yield chunk

def batch(iterable, size: int):
 """Split an iterable into fixed-size batches."""
 from itertools import islice
 iterator = iter(iterable)
 while batch := list(islice(iterator, size)):
 yield batch

# Process 1M records in batches of 100
for record_batch in batch(range(1_000_000), 100):
 process_batch(record_batch)

def sliding_window(iterable, size: int):
 """Sliding window over an iterable."""
 from collections import deque
 window = deque(maxlen=size)
 for item in iterable:
 window.append(item)
 if len(window) == size:
 yield tuple(window)

list(sliding_window([1, 2, 3, 4, 5], 3))
# [(1,2,3), (2,3,4), (3,4,5)]

Context Managers — Resource Management#

The Protocol#

class ManagedResource:
 """Context manager using the protocol directly."""
 
 def __init__(self, name: str):
 self.name = name
 print(f"Creating {name}")
 
 def __enter__(self):
 """Called when entering 'with' block. Return value binds to 'as'."""
 print(f"Acquiring {self.name}")
 return self # The 'as' variable gets this value
 
 def __exit__(self, exc_type, exc_val, exc_tb):
 """Called when leaving 'with' block (even on exception).
 
 Args:
 exc_type: Exception class (or None if no exception)
 exc_val: Exception instance (or None)
 exc_tb: Traceback (or None)
 
 Returns:
 True to suppress the exception, False/None to propagate it.
 """
 print(f"Releasing {self.name}")
 if exc_type is not None:
 print(f"Exception occurred: {exc_val}")
 return False # Don't suppress exceptions

with ManagedResource("database") as db:
 print("Using database")
# Output:
# Creating database
# Acquiring database
# Using database
# Releasing database

contextlib — Decorator-Based Context Managers#

from contextlib import contextmanager, asynccontextmanager
import time

@contextmanager
def timer(label: str):
 """Measure execution time of a block."""
 start = time.perf_counter()
 try:
 yield # Control passes to the 'with' block
 finally:
 elapsed = time.perf_counter() - start
 print(f"{label}: {elapsed:.4f}s")

with timer("data processing"):
 process_large_dataset()

@contextmanager
def temporary_directory():
 """Create a temp dir, clean up after use."""
 import tempfile
 import shutil
 
 path = tempfile.mkdtemp()
 try:
 yield path
 finally:
 shutil.rmtree(path)

with temporary_directory() as tmpdir:
 # Work with tmpdir
 pass # Automatically cleaned up

Nested and Stacked Context Managers#

from contextlib import ExitStack

def process_files(file_paths: list[str]) -> list[str]:
 """Open multiple files safely — all closed even if one fails."""
 with ExitStack() as stack:
 files = [
 stack.enter_context(open(path, "r"))
 for path in file_paths
 ]
 return [f.read() for f in files]

# ExitStack also supports dynamic cleanup
@contextmanager
def managed_connection_pool(size: int):
 """Pool that cleans up all connections on exit."""
 with ExitStack() as stack:
 connections = []
 for _ in range(size):
 conn = create_connection()
 stack.callback(conn.close) # Register cleanup
 connections.append(conn)
 yield connections

Async Context Managers#

from contextlib import asynccontextmanager

@asynccontextmanager
async def async_db_session():
 """Async context manager for database sessions."""
 session = await create_session()
 try:
 yield session
 await session.commit()
 except Exception:
 await session.rollback()
 raise
 finally:
 await session.close()

async def get_user(user_id: int):
 async with async_db_session() as session:
 return await session.query(User).get(user_id)

Combining All Three#

Use Case: Instrumented Pipeline#

from functools import wraps
from contextlib import contextmanager
import time
import logging

logger = logging.getLogger(__name__)

# Decorator: adds timing and logging to any function
def instrumented(func):
 @wraps(func)
 def wrapper(*args, **kwargs):
 start = time.perf_counter()
 try:
 result = func(*args, **kwargs)
 elapsed = time.perf_counter() - start
 logger.info(f"{func.__name__} completed in {elapsed:.3f}s")
 return result
 except Exception as e:
 elapsed = time.perf_counter() - start
 logger.error(f"{func.__name__} failed after {elapsed:.3f}s: {e}")
 raise
 return wrapper

# Generator: lazy data pipeline
def read_records(source):
 """Generator: yields records one at a time."""
 for raw in source:
 yield parse_record(raw)

def filter_valid(records):
 """Generator: passes through only valid records."""
 for record in records:
 if record.is_valid():
 yield record

def transform(records):
 """Generator: applies business logic."""
 for record in records:
 yield record.transform()

# Context manager: manages the pipeline lifecycle
@contextmanager
def pipeline(source, destination):
 """Context manager: sets up and tears down the pipeline."""
 logger.info("Pipeline starting")
 dest = open_destination(destination)
 try:
 yield dest
 finally:
 dest.flush()
 dest.close()
 logger.info("Pipeline complete")

# Putting it all together
@instrumented
def run_etl(source_path: str, dest_path: str):
 with pipeline(source_path, dest_path) as dest:
 records = read_records(open(source_path))
 valid = filter_valid(records)
 transformed = transform(valid)
 
 for batch in batch(transformed, size=1000):
 dest.write_batch(batch)

Key Takeaways#

Always use @wraps — preserves function metadata for debugging and documentation
Decorator factories (decorators with arguments) add a third nesting level
Generators are lazy — use them for large datasets, infinite sequences, and pipelines
yield from delegates to sub-generators and forwards send/throw
Context managers guarantee cleanup — prefer them over try/finally
ExitStack handles dynamic numbers of context managers
Combine all three for clean, composable, resource-safe code

Python: Type Hints and Static Analysis

Mon, 01 Jan 0001 00:00:00 +0000

Why Type Hints#

# Without types — what does this accept? What does it return?
def process(data, config):
 ...

# With types — self-documenting, IDE-supported, statically checkable
def process(data: list[dict[str, Any]], config: ProcessConfig) -> ProcessResult:
 ...

Benefits:

IDE autocompletion and error detection
Static analysis catches bugs before runtime
Self-documenting code
Refactoring confidence

Important: Type hints are NOT enforced at runtime by default. They’re for tools (mypy, pyright, IDEs).

Basic Type Annotations#

from typing import Any

# Variables
name: str = "Alice"
age: int = 30
active: bool = True
score: float = 95.5

# Functions
def greet(name: str, excited: bool = False) -> str:
 if excited:
 return f"Hello, {name}!!!"
 return f"Hello, {name}"

# None return
def log(message: str) -> None:
 print(message)

# Collections (Python 3.9+ — use built-in types directly)
numbers: list[int] = [1, 2, 3]
mapping: dict[str, int] = {"a": 1, "b": 2}
coordinates: tuple[float, float] = (3.14, 2.71)
unique: set[str] = {"apple", "banana"}

# Variable-length tuple
values: tuple[int, ...] = (1, 2, 3, 4, 5)

Union Types and Optional#

# Union — accepts multiple types
def parse_id(value: int | str) -> int: # Python 3.10+ syntax
 if isinstance(value, str):
 return int(value)
 return value

# Optional — shorthand for X | None
def find_user(user_id: int) -> dict | None:
 """Returns user dict or None if not found."""
 ...

# Pre-3.10 syntax (still valid)
from typing import Union, Optional
def parse_id(value: Union[int, str]) -> int: ...
def find_user(user_id: int) -> Optional[dict]: ...

Generics#

from typing import TypeVar, Generic

T = TypeVar("T")
K = TypeVar("K")
V = TypeVar("V")

# Generic function
def first(items: list[T]) -> T | None:
 return items[0] if items else None

# The return type matches the input type
first([1, 2, 3]) # inferred as int | None
first(["a", "b"]) # inferred as str | None

# Bounded TypeVar — restricts to specific types
from typing import TypeVar
Numeric = TypeVar("Numeric", int, float)

def add(a: Numeric, b: Numeric) -> Numeric:
 return a + b

# Generic class
class Stack(Generic[T]):
 def __init__(self) -> None:
 self._items: list[T] = []
 
 def push(self, item: T) -> None:
 self._items.append(item)
 
 def pop(self) -> T:
 if not self._items:
 raise IndexError("Stack is empty")
 return self._items.pop()
 
 def peek(self) -> T:
 if not self._items:
 raise IndexError("Stack is empty")
 return self._items[-1]
 
 def __len__(self) -> int:
 return len(self._items)

# Usage — type checker knows the element type
int_stack: Stack[int] = Stack()
int_stack.push(42)
value: int = int_stack.pop() # Type checker knows this is int

Protocols — Structural Typing (Duck Typing with Types)#

from typing import Protocol, runtime_checkable

# Protocol defines what methods/attributes an object must have
# WITHOUT requiring inheritance

@runtime_checkable
class Renderable(Protocol):
 def render(self) -> str: ...

@runtime_checkable
class Sized(Protocol):
 def __len__(self) -> int: ...

class HTMLWidget:
 """No inheritance from Renderable — just implements the method."""
 def render(self) -> str:
 return "<div>Widget</div>"

class JSONResponse:
 def render(self) -> str:
 return '{"status": "ok"}'

# Both satisfy Renderable without inheriting from it
def display(item: Renderable) -> None:
 print(item.render())

display(HTMLWidget()) # OK
display(JSONResponse()) # OK

# Runtime check (because @runtime_checkable)
isinstance(HTMLWidget(), Renderable) # True

# Complex protocol
class Repository(Protocol[T]):
 def get(self, id: str) -> T | None: ...
 def save(self, entity: T) -> None: ...
 def delete(self, id: str) -> bool: ...
 def list(self, limit: int = 100) -> list[T]: ...

TypedDict — Typed Dictionaries#

from typing import TypedDict, Required, NotRequired

class UserProfile(TypedDict):
 name: str
 email: str
 age: int
 bio: NotRequired[str] # Optional key

class APIResponse(TypedDict):
 status: int
 data: dict[str, Any]
 errors: NotRequired[list[str]]

def create_user(profile: UserProfile) -> APIResponse:
 # Type checker knows exactly which keys exist and their types
 print(profile["name"]) # OK — str
 # print(profile["phone"]) # Error — key doesn't exist
 
 return {"status": 201, "data": {"id": "123", **profile}}

# TypedDict with inheritance
class BaseEvent(TypedDict):
 event_type: str
 timestamp: str

class ClickEvent(BaseEvent):
 x: int
 y: int
 element_id: str

Callable Types#

from typing import Callable, ParamSpec, Concatenate
from collections.abc import Callable as CallableABC

# Function that accepts a callback
def apply_operation(
 values: list[int],
 operation: Callable[[int], int] # Takes int, returns int
) -> list[int]:
 return [operation(v) for v in values]

apply_operation([1, 2, 3], lambda x: x * 2) # [2, 4, 6]

# ParamSpec — preserve parameter types through decorators
P = ParamSpec("P")
R = TypeVar("R")

def logged(func: Callable[P, R]) -> Callable[P, R]:
 """Decorator that preserves the exact signature of the wrapped function."""
 @wraps(func)
 def wrapper(*args: P.args, **kwargs: P.kwargs) -> R:
 print(f"Calling {func.__name__}")
 return func(*args, **kwargs)
 return wrapper

@logged
def add(a: int, b: int) -> int:
 return a + b

# Type checker knows add still takes (int, int) -> int

Literal, Final, and TypeGuard#

from typing import Literal, Final, TypeGuard

# Literal — restrict to specific values
def set_direction(direction: Literal["north", "south", "east", "west"]) -> None:
 ...

set_direction("north") # OK
set_direction("up") # Type error!

# Final — cannot be reassigned or overridden
MAX_RETRIES: Final = 3
MAX_RETRIES = 5 # Type error!

class Base:
 @final
 def critical_method(self) -> None:
 """Subclasses cannot override this."""
 ...

# TypeGuard — narrow types in conditional checks
def is_string_list(val: list[Any]) -> TypeGuard[list[str]]:
 """Returns True if all elements are strings."""
 return all(isinstance(x, str) for x in val)

def process(data: list[Any]) -> None:
 if is_string_list(data):
 # Type checker now knows data is list[str]
 for s in data:
 print(s.upper()) # OK — s is str

Type Aliases and NewType#

from typing import NewType, TypeAlias

# Type alias — just a shorthand (no runtime distinction)
JSON: TypeAlias = dict[str, "JSON"] | list["JSON"] | str | int | float | bool | None
Headers: TypeAlias = dict[str, str]

def fetch(url: str, headers: Headers) -> JSON:
 ...

# NewType — creates a distinct type (catches mixing up same-typed values)
UserId = NewType("UserId", int)
OrderId = NewType("OrderId", int)

def get_user(user_id: UserId) -> dict:
 ...

def get_order(order_id: OrderId) -> dict:
 ...

uid = UserId(42)
oid = OrderId(42)

get_user(uid) # OK
get_user(oid) # Type error! OrderId is not UserId
get_user(42) # Type error! int is not UserId

Overload — Multiple Signatures#

from typing import overload

@overload
def fetch(url: str, raw: Literal[True]) -> bytes: ...
@overload
def fetch(url: str, raw: Literal[False] = ...) -> str: ...

def fetch(url: str, raw: bool = False) -> bytes | str:
 response = requests.get(url)
 if raw:
 return response.content
 return response.text

# Type checker knows:
result1 = fetch("http://example.com", raw=True) # bytes
result2 = fetch("http://example.com", raw=False) # str
result3 = fetch("http://example.com") # str

Running Static Analysis#

mypy Configuration#

# pyproject.toml
[tool.mypy]
python_version = "3.12"
strict = true
warn_return_any = true
warn_unused_configs = true
disallow_untyped_defs = true
disallow_any_generics = true
check_untyped_defs = true

# Per-module overrides
[[tool.mypy.overrides]]
module = "tests.*"
disallow_untyped_defs = false

[[tool.mypy.overrides]]
module = "third_party_lib.*"
ignore_missing_imports = true

# Run mypy
mypy src/
mypy src/ --strict
mypy src/module.py --show-error-codes

Common mypy Errors and Fixes#

# Error: Incompatible return value type (got "Optional[str]", expected "str")
def get_name(user: dict) -> str:
 return user.get("name") # .get() returns Optional[str]

# Fix: handle the None case
def get_name(user: dict) -> str:
 name = user.get("name")
 if name is None:
 raise ValueError("User has no name")
 return name

# Error: Item "None" of "Optional[User]" has no attribute "email"
user = find_user(123)
print(user.email) # user might be None!

# Fix: narrow the type
user = find_user(123)
if user is not None:
 print(user.email) # OK — type narrowed to User

# Or use assert (for cases where None is a bug)
user = find_user(123)
assert user is not None, f"User 123 must exist"
print(user.email)

Hypothetical Use Case: Typed Event System#

from typing import TypeVar, Generic, Callable, Protocol
from dataclasses import dataclass

E = TypeVar("E", bound="Event")

@dataclass(frozen=True)
class Event:
 """Base event class."""
 pass

@dataclass(frozen=True)
class UserCreated(Event):
 user_id: str
 email: str

@dataclass(frozen=True)
class OrderPlaced(Event):
 order_id: str
 user_id: str
 total: float

class EventHandler(Protocol[E]):
 def __call__(self, event: E) -> None: ...

class EventBus:
 def __init__(self) -> None:
 self._handlers: dict[type[Event], list[Callable]] = {}
 
 def subscribe(self, event_type: type[E], handler: EventHandler[E]) -> None:
 self._handlers.setdefault(event_type, []).append(handler)
 
 def publish(self, event: Event) -> None:
 for handler in self._handlers.get(type(event), []):
 handler(event)

# Usage — fully type-safe
bus = EventBus()

def on_user_created(event: UserCreated) -> None:
 send_welcome_email(event.email)

def on_order_placed(event: OrderPlaced) -> None:
 process_payment(event.order_id, event.total)

bus.subscribe(UserCreated, on_user_created)
bus.subscribe(OrderPlaced, on_order_placed)

bus.publish(UserCreated(user_id="u1", email="alice@example.com"))

Key Takeaways#

Type hints don’t run — they’re for tools (mypy, pyright, IDEs)
Use X | None instead of Optional[X] (Python 3.10+)
Protocols enable duck typing with type safety — no inheritance required
Generics make reusable, type-safe containers and functions
NewType prevents mixing up same-typed values (UserId vs OrderId)
TypedDict types dictionary shapes — essential for JSON/API data
ParamSpec preserves function signatures through decorators
Start with mypy --strict on new projects — retrofitting is harder

Python: Concurrency and Parallelism

Mon, 01 Jan 0001 00:00:00 +0000

The GIL — Global Interpreter Lock#

flowchart LR
 subgraph CPython
 GIL["GIL (one lock)"]
 T1["Thread 1"] -.->|"holds GIL"| GIL
 T2["Thread 2"] -.->|"waiting"| GIL
 T3["Thread 3"] -.->|"waiting"| GIL
 end
 
 subgraph "What this means"
 A["CPU-bound: threads DON'T help"]
 B["I/O-bound: threads DO help"]
 end

The GIL ensures only one thread executes Python bytecode at a time.

Task Type	Use	Why
I/O-bound (network, disk, DB)	`asyncio` or `threading`	GIL is released during I/O waits
CPU-bound (math, parsing, ML)	`multiprocessing`	Separate processes, each with own GIL
Mixed	`asyncio` + `ProcessPoolExecutor`	Async for I/O, processes for CPU

Threading — Concurrent I/O#

import threading
from concurrent.futures import ThreadPoolExecutor, as_completed
import time

# Basic thread
def download(url: str) -> str:
 time.sleep(1) # Simulates network I/O
 return f"Downloaded {url}"

# ThreadPoolExecutor — the modern way
def download_all(urls: list[str]) -> list[str]:
 results = []
 with ThreadPoolExecutor(max_workers=10) as executor:
 # Submit all tasks
 futures = {executor.submit(download, url): url for url in urls}
 
 # Process results as they complete
 for future in as_completed(futures):
 url = futures[future]
 try:
 result = future.result(timeout=30)
 results.append(result)
 except Exception as e:
 print(f"Failed {url}: {e}")
 
 return results

# 10 downloads in ~1 second instead of ~10 seconds
urls = [f"http://example.com/page/{i}" for i in range(10)]
download_all(urls)

Thread Safety#

import threading

# Race condition example
counter = 0

def increment():
 global counter
 for _ in range(100_000):
 counter += 1 # NOT atomic! Read-modify-write

threads = [threading.Thread(target=increment) for _ in range(4)]
for t in threads: t.start()
for t in threads: t.join()
print(counter) # Less than 400,000 — race condition!

# Fix: use a Lock
lock = threading.Lock()
counter = 0

def safe_increment():
 global counter
 for _ in range(100_000):
 with lock: # Only one thread at a time
 counter += 1

# Other synchronization primitives
event = threading.Event() # Signal between threads
semaphore = threading.Semaphore(5) # Limit concurrent access
barrier = threading.Barrier(3) # Wait for N threads to arrive

Multiprocessing — True Parallelism#

from multiprocessing import Pool, Process, Queue
from concurrent.futures import ProcessPoolExecutor
import os

# CPU-bound task
def compute_heavy(n: int) -> int:
 """Simulate CPU-intensive work."""
 return sum(i * i for i in range(n))

# ProcessPoolExecutor — cleanest API
def parallel_compute(tasks: list[int]) -> list[int]:
 with ProcessPoolExecutor(max_workers=os.cpu_count()) as executor:
 results = list(executor.map(compute_heavy, tasks))
 return results

# Each process has its own Python interpreter and GIL
# Data is serialized (pickled) between processes

# Pool with imap for large datasets (lazy, ordered)
def process_large_dataset(items: list) -> list:
 with Pool(processes=4) as pool:
 # imap_unordered: results come as they finish (faster)
 results = []
 for result in pool.imap_unordered(transform, items, chunksize=100):
 results.append(result)
 return results

Shared State Between Processes#

from multiprocessing import Value, Array, Manager

# Shared memory (fast, limited types)
shared_counter = Value('i', 0) # 'i' = integer
shared_array = Array('d', [0.0] * 10) # 'd' = double

def worker(counter, array, index):
 with counter.get_lock():
 counter.value += 1
 array[index] = counter.value

# Manager (slower, supports complex types)
with Manager() as manager:
 shared_dict = manager.dict()
 shared_list = manager.list()
 
 # These can be passed to child processes
 # Changes are synchronized automatically

asyncio — Asynchronous I/O#

Core Concepts#

flowchart TD
 EL["Event Loop"] --> C1["Coroutine 1"]
 EL --> C2["Coroutine 2"]
 EL --> C3["Coroutine 3"]
 
 C1 -->|"await (I/O)"| EL
 C2 -->|"await (I/O)"| EL
 C3 -->|"running"| CPU["CPU"]
 
 Note["Single thread, cooperative multitasking.<br>Coroutines yield control at 'await' points."]

import asyncio
import aiohttp

# Coroutine — defined with async def
async def fetch_url(session: aiohttp.ClientSession, url: str) -> str:
 async with session.get(url) as response:
 return await response.text()

# Run multiple coroutines concurrently
async def fetch_all(urls: list[str]) -> list[str]:
 async with aiohttp.ClientSession() as session:
 tasks = [fetch_url(session, url) for url in urls]
 return await asyncio.gather(*tasks)

# Entry point
results = asyncio.run(fetch_all(urls))

asyncio Patterns#

import asyncio
from asyncio import Semaphore, Queue

# Rate limiting with semaphore
async def rate_limited_fetch(urls: list[str], max_concurrent: int = 5) -> list[str]:
 semaphore = Semaphore(max_concurrent)
 
 async def fetch_with_limit(url: str) -> str:
 async with semaphore: # At most max_concurrent at a time
 return await fetch_url(url)
 
 tasks = [fetch_with_limit(url) for url in urls]
 return await asyncio.gather(*tasks)

# Producer-consumer with asyncio.Queue
async def producer(queue: Queue, items: list):
 for item in items:
 await queue.put(item)
 await queue.put(None) # Sentinel to signal completion

async def consumer(queue: Queue, name: str):
 while True:
 item = await queue.get()
 if item is None:
 queue.task_done()
 break
 await process(item)
 queue.task_done()

async def pipeline(items: list, num_consumers: int = 3):
 queue: Queue = Queue(maxsize=100)
 
 producers = [asyncio.create_task(producer(queue, items))]
 consumers = [
 asyncio.create_task(consumer(queue, f"consumer-{i}"))
 for i in range(num_consumers)
 ]
 
 await asyncio.gather(*producers)
 await queue.join() # Wait for all items to be processed
 
 for c in consumers:
 c.cancel()

# Timeout handling
async def fetch_with_timeout(url: str, timeout: float = 10.0) -> str:
 try:
 return await asyncio.wait_for(fetch_url(url), timeout=timeout)
 except asyncio.TimeoutError:
 return f"Timeout fetching {url}"

# TaskGroup (Python 3.11+) — structured concurrency
async def fetch_all_structured(urls: list[str]) -> list[str]:
 results = []
 async with asyncio.TaskGroup() as tg:
 for url in urls:
 tg.create_task(fetch_and_append(url, results))
 # All tasks complete or all are cancelled if one raises
 return results

Async Generators and Iterators#

async def stream_events(url: str):
 """Async generator — yields events as they arrive."""
 async with aiohttp.ClientSession() as session:
 async with session.get(url) as response:
 async for line in response.content:
 if line.strip():
 yield json.loads(line)

# Consuming async generators
async def process_stream():
 async for event in stream_events("http://api.example.com/events"):
 handle_event(event)

# Async comprehension
async def get_all_users() -> list[dict]:
 return [user async for user in fetch_users_paginated()]

Combining Approaches#

import asyncio
from concurrent.futures import ProcessPoolExecutor

async def hybrid_pipeline(urls: list[str]) -> list[dict]:
 """Fetch data async, process CPU-heavy in parallel."""
 
 # Step 1: Fetch all data concurrently (I/O-bound → asyncio)
 raw_data = await fetch_all(urls)
 
 # Step 2: Process data in parallel (CPU-bound → multiprocessing)
 loop = asyncio.get_event_loop()
 with ProcessPoolExecutor() as pool:
 processed = await asyncio.gather(*[
 loop.run_in_executor(pool, cpu_heavy_transform, data)
 for data in raw_data
 ])
 
 return processed

Synchronization Primitives Comparison#

Primitive	threading	asyncio	Use Case
Lock	`threading.Lock()`	`asyncio.Lock()`	Mutual exclusion
Semaphore	`threading.Semaphore(n)`	`asyncio.Semaphore(n)`	Limit concurrency
Event	`threading.Event()`	`asyncio.Event()`	Signal between tasks
Queue	`queue.Queue()`	`asyncio.Queue()`	Producer-consumer
Barrier	`threading.Barrier(n)`	`asyncio.Barrier(n)`	Wait for N tasks

Hypothetical Use Cases#

Use Case: Web Scraper with Rate Limiting#

import asyncio
import aiohttp
from dataclasses import dataclass, field
from datetime import datetime

@dataclass
class ScrapeResult:
 url: str
 status: int
 content_length: int
 elapsed_ms: float
 error: str | None = None

async def scrape_with_rate_limit(
 urls: list[str],
 max_concurrent: int = 10,
 delay_between: float = 0.1,
) -> list[ScrapeResult]:
 """Scrape URLs with concurrency limit and politeness delay."""
 semaphore = asyncio.Semaphore(max_concurrent)
 results: list[ScrapeResult] = []
 
 async def fetch_one(session: aiohttp.ClientSession, url: str) -> ScrapeResult:
 async with semaphore:
 start = datetime.now()
 try:
 async with session.get(url, timeout=aiohttp.ClientTimeout(total=30)) as resp:
 content = await resp.read()
 elapsed = (datetime.now() - start).total_seconds() * 1000
 return ScrapeResult(
 url=url, status=resp.status,
 content_length=len(content), elapsed_ms=elapsed,
 )
 except Exception as e:
 elapsed = (datetime.now() - start).total_seconds() * 1000
 return ScrapeResult(
 url=url, status=0, content_length=0,
 elapsed_ms=elapsed, error=str(e),
 )
 finally:
 await asyncio.sleep(delay_between)
 
 async with aiohttp.ClientSession() as session:
 tasks = [fetch_one(session, url) for url in urls]
 results = await asyncio.gather(*tasks)
 
 return results

Use Case: Parallel Data Processing Pipeline#

from multiprocessing import Pool
from concurrent.futures import ProcessPoolExecutor
from pathlib import Path
import json

def process_file(file_path: str) -> dict:
 """CPU-intensive: parse, validate, transform a data file."""
 path = Path(file_path)
 data = json.loads(path.read_text())
 
 # Heavy computation
 transformed = {
 "source": path.name,
 "record_count": len(data),
 "summary": compute_statistics(data),
 "anomalies": detect_anomalies(data),
 }
 return transformed

def parallel_process_directory(data_dir: str) -> list[dict]:
 """Process all JSON files in a directory using all CPU cores."""
 files = list(Path(data_dir).glob("*.json"))
 
 with ProcessPoolExecutor() as executor:
 results = list(executor.map(
 process_file,
 [str(f) for f in files],
 chunksize=10, # Send 10 files per worker batch
 ))
 
 return results

Key Takeaways#

GIL means threads don’t help for CPU-bound work — use multiprocessing
asyncio is for I/O-bound concurrency — single thread, cooperative scheduling
ThreadPoolExecutor is the simplest way to parallelize I/O operations
ProcessPoolExecutor gives true parallelism for CPU-bound work
asyncio.gather runs coroutines concurrently; TaskGroup (3.11+) adds structured cancellation
Semaphores limit concurrency — essential for rate limiting
Never mix blocking I/O with asyncio — use run_in_executor for blocking calls
Data between processes is pickled — keep shared data minimal

Python: Packaging and Distribution

Mon, 01 Jan 0001 00:00:00 +0000

Project Structure#

my-project/
├── pyproject.toml # THE single source of truth for project metadata
├── src/
│ └── my_package/
│ ├── __init__.py
│ ├── core.py
│ ├── utils.py
│ └── py.typed # Marker file: "this package ships type stubs"
├── tests/
│ ├── __init__.py
│ ├── conftest.py
│ ├── test_core.py
│ └── test_utils.py
├── docs/
│ └── index.md
├── README.md
├── LICENSE
└── .gitignore

Why `src/` Layout?#

# Without src/ — accidental imports from project root
my_project/
├── my_package/ ← Python might import THIS instead of installed version
├── tests/
└── pyproject.toml

# With src/ — forces you to install the package to import it
my_project/
├── src/
│ └── my_package/ ← Only importable after `pip install -e .`
├── tests/
└── pyproject.toml

pyproject.toml — Modern Configuration#

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "my-package"
version = "1.2.0"
description = "A useful Python package"
readme = "README.md"
license = "MIT"
requires-python = ">=3.11"
authors = [
 { name = "Alice Developer", email = "alice@example.com" },
]
keywords = ["utility", "tools"]
classifiers = [
 "Development Status :: 4 - Beta",
 "Programming Language :: Python :: 3.11",
 "Programming Language :: Python :: 3.12",
 "Typing :: Typed",
]

dependencies = [
 "httpx>=0.25.0",
 "pydantic>=2.0,<3.0",
]

[project.optional-dependencies]
dev = [
 "pytest>=7.0",
 "pytest-cov>=4.0",
 "mypy>=1.5",
 "ruff>=0.1.0",
]
docs = [
 "mkdocs>=1.5",
 "mkdocs-material>=9.0",
]

[project.scripts]
my-cli = "my_package.cli:main"

[project.urls]
Homepage = "https://github.com/alice/my-package"
Documentation = "https://my-package.readthedocs.io"
Repository = "https://github.com/alice/my-package"

# Tool configuration in the same file
[tool.pytest.ini_options]
testpaths = ["tests"]
addopts = "-ra --strict-markers"

[tool.mypy]
python_version = "3.12"
strict = true

[tool.ruff]
target-version = "py312"
line-length = 100

[tool.ruff.lint]
select = ["E", "F", "I", "N", "UP", "B", "A", "SIM"]

Dependency Management#

Version Specifiers#

Specifier	Meaning	Example
`>=1.0`	At least 1.0	`httpx>=0.25.0`
`>=1.0,<2.0`	Compatible range	`pydantic>=2.0,<3.0`
`~=1.4`	Compatible release (>=1.4, <2.0)	`~=1.4.2` means >=1.4.2, <1.5
`==1.2.3`	Exact version	Pin for reproducibility

Virtual Environments#

# Create and activate
python -m venv .venv
source .venv/bin/activate # macOS/Linux

# Install in development mode (editable)
pip install -e ".[dev]"

# Lock dependencies for reproducibility
pip freeze > requirements.lock

# Or use pip-tools for better dependency resolution
pip install pip-tools
pip-compile pyproject.toml -o requirements.lock
pip-sync requirements.lock

uv — Modern Package Manager (Faster Alternative)#

# Install uv
curl -LsSf https://astral.sh/uv/install.sh | sh

# Create project
uv init my-project
cd my-project

# Add dependencies
uv add httpx pydantic
uv add --dev pytest mypy ruff

# Sync environment
uv sync

# Run commands in the environment
uv run pytest
uv run mypy src/

# Lock file is automatic (uv.lock)

Building and Publishing#

Build the Package#

# Install build tool
pip install build

# Build sdist and wheel
python -m build

# Output:
# dist/
# ├── my_package-1.2.0.tar.gz (source distribution)
# └── my_package-1.2.0-py3-none-any.whl (wheel — faster to install)

Publish to PyPI#

# Install twine
pip install twine

# Upload to Test PyPI first
twine upload --repository testpypi dist/*

# Upload to real PyPI
twine upload dist/*

# Or use trusted publishing (GitHub Actions → PyPI without tokens)

GitHub Actions CI/CD#

name: CI

on:
 push:
 branches: [main]
 pull_request:

jobs:
 test:
 runs-on: ubuntu-latest
 strategy:
 matrix:
 python-version: ["3.11", "3.12", "3.13"]
 
 steps:
 - uses: actions/checkout@v4
 - uses: actions/setup-python@v5
 with:
 python-version: ${{ matrix.python-version }}
 
 - run: pip install -e ".[dev]"
 - run: ruff check src/
 - run: mypy src/
 - run: pytest --cov=my_package --cov-report=xml
 
 publish:
 needs: test
 if: startsWith(github.ref, 'refs/tags/v')
 runs-on: ubuntu-latest
 permissions:
 id-token: write  # Trusted publishing
 
 steps:
 - uses: actions/checkout@v4
 - run: pip install build
 - run: python -m build
 - uses: pypa/gh-action-pypi-publish@release/v1

Entry Points and CLI Tools#

# src/my_package/cli.py
import argparse
import sys

def main():
 parser = argparse.ArgumentParser(description="My CLI tool")
 parser.add_argument("input", help="Input file path")
 parser.add_argument("-o", "--output", default="stdout", help="Output destination")
 parser.add_argument("-v", "--verbose", action="store_true")
 
 args = parser.parse_args()
 
 try:
 result = process(args.input)
 if args.output == "stdout":
 print(result)
 else:
 Path(args.output).write_text(result)
 except Exception as e:
 print(f"Error: {e}", file=sys.stderr)
 sys.exit(1)

if __name__ == "__main__":
 main()

# In pyproject.toml
[project.scripts]
my-cli = "my_package.cli:main"
# After `pip install`, `my-cli` is available as a command

Namespace Packages#

# Multiple packages sharing a namespace (no __init__.py at namespace level)
company-auth/
└── src/
 └── company/ ← NO __init__.py here
 └── auth/
 ├── __init__.py
 └── tokens.py

company-logging/
└── src/
 └── company/ ← NO __init__.py here
 └── logging/
 ├── __init__.py
 └── handlers.py

# Both installable, both importable:
from company.auth.tokens import create_token
from company.logging.handlers import JSONHandler

Hypothetical Use Case: Internal Library#

# pyproject.toml for an internal company library
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "acme-utils"
version = "2.1.0"
requires-python = ">=3.11"
dependencies = [
 "httpx>=0.25",
 "pydantic>=2.0,<3.0",
 "structlog>=23.0",
]

[project.optional-dependencies]
aws = ["boto3>=1.28"]
testing = ["pytest>=7.0", "pytest-asyncio>=0.21", "respx>=0.20"]

[tool.hatch.build.targets.wheel]
packages = ["src/acme"]

# Install from private registry
pip install acme-utils --index-url https://pypi.internal.company.com/simple/

# Or from git
pip install git+ssh://git@github.com/acme/acme-utils.git@v2.1.0

Key Takeaways#

pyproject.toml is the standard — replaces setup.py, setup.cfg, requirements.txt for metadata
Use src/ layout — prevents accidental imports from the project root
Pin ranges for libraries (>=1.0,<2.0), lock exact versions for applications
uv is the modern, fast alternative to pip + venv + pip-tools
Wheels are pre-built — faster to install than source distributions
Entry points ([project.scripts]) create CLI commands on install
Trusted publishing (OIDC) eliminates the need for PyPI API tokens in CI

Python: Testing

Mon, 01 Jan 0001 00:00:00 +0000

Testing Philosophy#

graph TD
 A["Testing Pyramid"] --> U["Unit Tests<br>(fast, isolated, many)"]
 A --> I["Integration Tests<br>(slower, fewer)"]
 A --> E["End-to-End Tests<br>(slowest, fewest)"]
 
 style U fill:#c8e6c9
 style I fill:#fff9c4
 style E fill:#ffcdd2

Level	Tests	Speed	Scope
Unit	Hundreds	Milliseconds	Single function/class
Integration	Dozens	Seconds	Multiple components together
E2E	Few	Minutes	Full system behavior

pytest — The Standard#

Basic Tests#

# tests/test_calculator.py
import pytest
from my_package.calculator import add, divide, Calculator

# Simple assertion
def test_add():
 assert add(2, 3) == 5

def test_add_negative():
 assert add(-1, 1) == 0

def test_add_floats():
 assert add(0.1, 0.2) == pytest.approx(0.3) # Float comparison

# Testing exceptions
def test_divide_by_zero():
 with pytest.raises(ZeroDivisionError, match="division by zero"):
 divide(10, 0)

def test_invalid_input():
 with pytest.raises(TypeError):
 add("a", 1)

Fixtures — Setup and Teardown#

import pytest
from pathlib import Path
import json

@pytest.fixture
def sample_user() -> dict:
 """Simple fixture — returns test data."""
 return {"name": "Alice", "email": "alice@example.com", "age": 30}

@pytest.fixture
def temp_config(tmp_path: Path) -> Path:
 """Fixture using built-in tmp_path — creates a temp config file."""
 config = {"database_url": "sqlite:///test.db", "debug": True}
 config_path = tmp_path / "config.json"
 config_path.write_text(json.dumps(config))
 return config_path

@pytest.fixture
def db_connection():
 """Fixture with setup AND teardown."""
 conn = create_test_database()
 yield conn # Test runs here
 conn.close()
 cleanup_test_database()

# Using fixtures
def test_user_creation(sample_user):
 user = User(**sample_user)
 assert user.name == "Alice"

def test_config_loading(temp_config):
 config = load_config(str(temp_config))
 assert config["debug"] is True

Fixture Scopes#

@pytest.fixture(scope="session")
def database():
 """Created once for the entire test session."""
 db = create_database()
 yield db
 db.drop()

@pytest.fixture(scope="module")
def api_client():
 """Created once per test module (file)."""
 return APIClient(base_url="http://test-api.local")

@pytest.fixture(scope="function") # Default
def clean_state(database):
 """Created fresh for each test function."""
 database.clear()
 yield database

Parametrize — Data-Driven Tests#

@pytest.mark.parametrize("input,expected", [
 ("hello", "HELLO"),
 ("world", "WORLD"),
 ("", ""),
 ("123", "123"),
 ("Hello World", "HELLO WORLD"),
])
def test_uppercase(input, expected):
 assert input.upper() == expected

@pytest.mark.parametrize("a,b,expected", [
 (1, 2, 3),
 (-1, 1, 0),
 (0, 0, 0),
 (100, 200, 300),
 (-5, -3, -8),
])
def test_add(a, b, expected):
 assert add(a, b) == expected

# Parametrize with IDs for readable output
@pytest.mark.parametrize("status_code,should_retry", [
 pytest.param(200, False, id="success"),
 pytest.param(429, True, id="rate-limited"),
 pytest.param(500, True, id="server-error"),
 pytest.param(404, False, id="not-found"),
])
def test_retry_logic(status_code, should_retry):
 assert should_retry_request(status_code) == should_retry

Markers — Categorizing Tests#

# conftest.py — register custom markers
def pytest_configure(config):
 config.addinivalue_line("markers", "slow: marks tests as slow")
 config.addinivalue_line("markers", "integration: integration tests")

# Usage
@pytest.mark.slow
def test_large_dataset_processing():
 ...

@pytest.mark.integration
def test_database_connection():
 ...

@pytest.mark.skipif(sys.platform == "win32", reason="Unix only")
def test_unix_permissions():
 ...

@pytest.mark.xfail(reason="Known bug #123")
def test_edge_case():
 ...

# Run only fast tests
pytest -m "not slow"

# Run only integration tests
pytest -m integration

# Run with verbose output
pytest -v --tb=short

Mocking — Isolating Units#

from unittest.mock import Mock, patch, MagicMock, AsyncMock
from my_package.service import UserService

# patch — replace an object during the test
@patch("my_package.service.database.get_user")
def test_get_user_found(mock_get_user):
 mock_get_user.return_value = {"id": 1, "name": "Alice"}
 
 service = UserService()
 user = service.get_user(1)
 
 assert user["name"] == "Alice"
 mock_get_user.assert_called_once_with(1)

# patch as context manager
def test_api_call():
 with patch("my_package.client.requests.get") as mock_get:
 mock_get.return_value.status_code = 200
 mock_get.return_value.json.return_value = {"data": "test"}
 
 result = fetch_data("http://api.example.com")
 
 assert result == {"data": "test"}
 mock_get.assert_called_once_with("http://api.example.com", timeout=30)

# Mock with side_effect — simulate different responses
@patch("my_package.service.external_api")
def test_retry_on_failure(mock_api):
 mock_api.side_effect = [
 ConnectionError("timeout"), # First call fails
 {"status": "ok"}, # Second call succeeds
 ]
 
 result = resilient_fetch()
 assert result == {"status": "ok"}
 assert mock_api.call_count == 2

# AsyncMock for async code
@patch("my_package.async_client.fetch", new_callable=AsyncMock)
async def test_async_fetch(mock_fetch):
 mock_fetch.return_value = {"users": []}
 result = await get_users()
 assert result == []

When NOT to Mock#

# DON'T mock what you own — test the real thing
# DON'T mock data structures (dicts, lists)
# DON'T mock simple pure functions

# DO mock:
# - External APIs (network calls)
# - Databases (in unit tests)
# - File system (when testing logic, not I/O)
# - Time (datetime.now, time.sleep)
# - Random (when testing deterministic behavior)

# Example: mock time
from freezegun import freeze_time

@freeze_time("2026-01-15 12:00:00")
def test_token_expiry():
 token = create_token(expires_in=3600)
 assert token.expires_at == datetime(2026, 1, 15, 13, 0, 0)

Testing Patterns#

Arrange-Act-Assert (AAA)#

def test_order_total_with_discount():
 # Arrange — set up test data
 order = Order(items=[
 OrderItem("Widget", price=10.00, quantity=3),
 OrderItem("Gadget", price=25.00, quantity=1),
 ])
 discount = PercentageDiscount(10) # 10% off
 
 # Act — perform the action
 total = order.calculate_total(discount=discount)
 
 # Assert — verify the result
 assert total == pytest.approx(49.50) # (30 + 25) * 0.9

Testing Error Paths#

def test_insufficient_funds():
 account = BankAccount(balance=100.00)
 
 with pytest.raises(InsufficientFundsError) as exc_info:
 account.withdraw(150.00)
 
 # Verify exception details
 assert exc_info.value.balance == 100.00
 assert exc_info.value.amount == 150.00
 assert "insufficient funds" in str(exc_info.value).lower()
 
 # Verify account state unchanged
 assert account.balance == 100.00

def test_validation_collects_all_errors():
 """Test that validation reports ALL errors, not just the first."""
 data = {"name": "", "email": "invalid", "age": -5}
 
 errors = validate_user(data)
 
 assert len(errors) == 3
 assert any(e.field == "name" for e in errors)
 assert any(e.field == "email" for e in errors)
 assert any(e.field == "age" for e in errors)

Property-Based Testing with Hypothesis#

from hypothesis import given, strategies as st, assume

@given(st.lists(st.integers()))
def test_sort_is_idempotent(lst):
 """Sorting twice gives the same result as sorting once."""
 assert sorted(sorted(lst)) == sorted(lst)

@given(st.lists(st.integers(), min_size=1))
def test_sort_preserves_length(lst):
 assert len(sorted(lst)) == len(lst)

@given(st.text(min_size=1), st.text(min_size=1))
def test_concat_length(a, b):
 assert len(a + b) == len(a) + len(b)

@given(st.integers(min_value=0, max_value=1000))
def test_encode_decode_roundtrip(n):
 """Encoding then decoding returns the original value."""
 encoded = encode(n)
 decoded = decode(encoded)
 assert decoded == n

Test Configuration#

# pyproject.toml
[tool.pytest.ini_options]
testpaths = ["tests"]
addopts = [
 "-ra", # Show summary of all non-passing tests
 "--strict-markers", # Error on unknown markers
 "--tb=short", # Shorter tracebacks
 "-q", # Quiet output
]
markers = [
 "slow: marks tests as slow (deselect with '-m \"not slow\"')",
 "integration: integration tests requiring external services",
]
filterwarnings = [
 "error", # Treat warnings as errors
 "ignore::DeprecationWarning:third_party_lib.*",
]

[tool.coverage.run]
source = ["src"]
branch = true

[tool.coverage.report]
fail_under = 80
show_missing = true
exclude_lines = [
 "pragma: no cover",
 "if TYPE_CHECKING:",
 "if __name__ == .__main__.",
 "@overload",
]

# Run with coverage
pytest --cov=my_package --cov-report=term-missing --cov-report=html

# Run specific test
pytest tests/test_core.py::test_add

# Run tests matching a pattern
pytest -k "test_user and not slow"

# Parallel execution
pytest -n auto # requires pytest-xdist

Hypothetical Use Case: Testing a REST API Client#

import pytest
import respx
import httpx
from my_package.api_client import APIClient, APIError

@pytest.fixture
def client():
 return APIClient(base_url="https://api.example.com", api_key="test-key")

class TestAPIClient:
 @respx.mock
 def test_get_user_success(self, client):
 respx.get("https://api.example.com/users/123").mock(
 return_value=httpx.Response(200, json={"id": 123, "name": "Alice"})
 )
 
 user = client.get_user(123)
 
 assert user["name"] == "Alice"
 
 @respx.mock
 def test_get_user_not_found(self, client):
 respx.get("https://api.example.com/users/999").mock(
 return_value=httpx.Response(404, json={"error": "Not found"})
 )
 
 with pytest.raises(APIError) as exc_info:
 client.get_user(999)
 
 assert exc_info.value.status_code == 404
 
 @respx.mock
 def test_retry_on_server_error(self, client):
 route = respx.get("https://api.example.com/users/1")
 route.side_effect = [
 httpx.Response(503),
 httpx.Response(503),
 httpx.Response(200, json={"id": 1, "name": "Bob"}),
 ]
 
 user = client.get_user(1)
 
 assert user["name"] == "Bob"
 assert route.call_count == 3
 
 @respx.mock
 def test_timeout_raises_api_error(self, client):
 respx.get("https://api.example.com/users/1").mock(
 side_effect=httpx.ReadTimeout("Connection timed out")
 )
 
 with pytest.raises(APIError, match="timed out"):
 client.get_user(1)

Key Takeaways#

pytest is the standard — use it over unittest for cleaner syntax
Fixtures manage test setup/teardown — use scopes to control lifecycle
Parametrize eliminates duplicate test functions — one test, many inputs
Mock external dependencies (APIs, DBs, time) — never mock what you own
AAA pattern (Arrange-Act-Assert) keeps tests readable
Property-based testing (Hypothesis) finds edge cases you’d never think of
Coverage is a guide, not a goal — 80% is a good target, 100% is often wasteful
Test behavior, not implementation — tests should survive refactoring

Web Frameworks

Mon, 01 Jan 0001 00:00:00 +0000

Web Frameworks#

Python powers some of the most popular web frameworks. This section covers the three main options — FastAPI for modern async APIs, Flask for lightweight applications, and Django for full-featured projects — and helps you choose between them.

The Python Web Ecosystem#

Framework	Type	Best For	Async
FastAPI	Micro	APIs, microservices	Native (ASGI)
Flask	Micro	Simple apps, prototypes	Via extensions
Django	Full-stack	Content sites, admin-heavy apps	Partial (Django 4.1+)
Starlette	Micro	FastAPI’s foundation	Native (ASGI)
Litestar	Micro	FastAPI alternative	Native (ASGI)

WSGI vs ASGI#

	WSGI	ASGI
Model	Synchronous	Asynchronous
Concurrency	Thread-per-request	Event loop + async
Servers	Gunicorn, uWSGI	Uvicorn, Hypercorn
Frameworks	Flask, Django (traditional)	FastAPI, Starlette, Django (channels)
WebSockets	Not supported	Supported

FastAPI#

Modern, fast, type-safe API framework built on Starlette and Pydantic: