Storage & Content Delivery Networks#

Choosing the right storage type and delivery mechanism directly impacts performance, cost, and reliability. This section covers cloud storage models, CDN architecture, and patterns for serving content at global scale.

Storage Types Compared#

When to Use Each#

graph TD
    Q1{Need shared filesystem?} -->|Yes| EFS[EFS / File Storage]
    Q1 -->|No| Q2{Need low-latency block device?}
    Q2 -->|Yes| EBS[EBS / Block Storage]
    Q2 -->|No| S3[S3 / Object Storage]

Object Storage (S3) Deep Dive#

Key Concepts#

• Bucket — container for objects (globally unique name)
• Object — file + metadata, identified by a key (path-like string)
• Prefix — simulated folder structure (images/2024/photo.jpg)
• Versioning — keep multiple versions of every object
• Lifecycle policies — automatically transition or delete objects based on age

Storage Classes and Lifecycle#

Lifecycle Policy Example#

{
  "Rules": [
    {
      "ID": "ArchiveOldLogs",
      "Status": "Enabled",
      "Prefix": "logs/",
      "Transitions": [
        { "Days": 30, "StorageClass": "STANDARD_IA" },
        { "Days": 90, "StorageClass": "GLACIER" }
      ],
      "Expiration": { "Days": 365 }
    }
  ]
}

CDN Architecture#

A Content Delivery Network (CDN) caches content at geographically distributed edge locations, reducing latency for end users by serving content from a nearby point of presence (PoP).

How a CDN Works#

sequenceDiagram
    participant User
    participant Edge[Edge Location (PoP)]
    participant Origin[Origin Server]
    
    User->>Edge: GET /image.jpg
    alt Cache HIT
        Edge->>User: 200 OK (cached copy)
    else Cache MISS
        Edge->>Origin: GET /image.jpg
        Origin->>Edge: 200 OK (original)
        Edge->>Edge: Store in cache
        Edge->>User: 200 OK
    end

Key Terminology#

CDN Benefits#

Cache-Control Headers#

Cache-Control headers tell the CDN (and browser) how to cache responses.

Best Practice by Content Type#

CDN Invalidation#

When content changes, you need to remove stale copies from edge caches.

Best practice: Use content-hashed filenames for static assets. Reserve invalidation for HTML and API responses where URL versioning isn’t practical.

Signed URLs and Signed Cookies#

Restrict access to CDN content without making the origin public.

Signed URLs#

Generate a time-limited URL that grants access to a specific object:

import boto3
from datetime import datetime, timedelta

s3 = boto3.client("s3")
url = s3.generate_presigned_url(
    "get_object",
    Params={"Bucket": "my-bucket", "Key": "private/report.pdf"},
    ExpiresIn=3600,  # 1 hour
)

Static Site Hosting#

Object storage + CDN is the standard pattern for static websites (HTML, CSS, JS, images).

graph LR
    User[User Browser] --> CDN[CloudFront / CDN]
    CDN --> S3[S3 Bucket - Origin]
    S3 -->|index.html, assets| CDN
    CDN -->|Cached response| User

Configuration#

1. Upload static files to S3 bucket
2. Enable static website hosting on the bucket
3. Create CloudFront distribution pointing to bucket
4. Configure custom domain + TLS certificate (ACM)
5. Set cache behaviours per path pattern

Advantages Over Traditional Hosting#

Blob Storage Patterns#

Write-Once, Read-Many (WORM)#

Objects written once and read frequently. Common for logs, media, compliance records. Enable object lock for regulatory requirements.

Multipart Upload#

For large files (> 100 MB), break into parts and upload in parallel:

1. Initiate multipart upload → get upload ID
2. Upload parts in parallel (each 5 MB - 5 GB)
3. Complete upload → S3 assembles the object

Event-Driven Processing#

graph LR
    Upload[File Upload] --> S3[S3 Bucket]
    S3 -->|S3 Event| Lambda[Lambda Function]
    Lambda --> Process[Thumbnail / Transcode / Index]
    Process --> Output[Output Bucket]

Data Lakes Introduction#

A data lake stores raw data in its native format (structured, semi-structured, unstructured) at any scale, enabling diverse analytics workloads.

Data Lake Architecture#

Data Lake vs Data Warehouse#

File Formats for Analytics#

Key Takeaways#

• Object storage is the default for unstructured data (files, images, backups) — it’s durable, cheap, and infinitely scalable
• Block storage is for databases and OS disks requiring low-latency I/O
• Storage lifecycle policies move data to cheaper tiers automatically — configure them from day one to control costs
• CDNs reduce latency by 10x by serving content from edge locations near users — use them for all static assets
• Content-hashed URLs (app.a3f2.js) eliminate the need for cache invalidation — the best invalidation is no invalidation
• Signed URLs provide time-limited secure access without making origins public
• Data lakes on object storage provide a cost-effective foundation for analytics — use columnar formats (Parquet) for query performance