Prompt: "The capital of France is"

Step 1: P("Paris" | "The capital of France is")     → "Paris"
Step 2: P("."     | "The capital of France is Paris") → "."
Step 3: P("<EOS>" | "The capital of France is Paris.") → STOP

Without KV Cache (naive):
  Token 1: compute attention over [prompt]
  Token 2: compute attention over [prompt + token1]        ← recomputes prompt
  Token 3: compute attention over [prompt + token1 + token2] ← recomputes everything
  Complexity: O(n²) per token

With KV Cache:
  Token 1: compute attention, STORE K,V for all positions
  Token 2: compute attention for new token using CACHED K,V + new K,V
  Token 3: compute attention for new token using CACHED K,V + new K,V
  Complexity: O(n) per token

KV Cache Size = 2 × num_layers × num_heads × head_dim × sequence_length × bytes_per_param

Example (Llama 3 70B, FP16, 4096 sequence length):
  = 2 × 80 × 64 × 128 × 4096 × 2 bytes
  = ~10.7 GB per sequence

Memory = num_parameters × bytes_per_parameter

FP32: 4 bytes per param  → 70B model = 280 GB
FP16: 2 bytes per param  → 70B model = 140 GB
INT8: 1 byte per param   → 70B model = 70 GB
INT4: 0.5 bytes per param → 70B model = 35 GB

Total GPU Memory = Model Weights + KV Cache + Activations + Overhead

Example: Serving Llama 3 70B in INT4 with 8 concurrent users (4K context):
  Model weights:  35 GB
  KV cache:       10.7 GB × 8 users = 85.6 GB
  Activations:    ~2 GB
  Overhead:       ~3 GB
  Total:          ~126 GB → needs 2× A100 80GB or 2× H100 80GB

from transformers import AutoModelForCausalLM, AutoTokenizer, GPTQConfig

quantization_config = GPTQConfig(
    bits=4,
    dataset="c4",  # Calibration dataset
    group_size=128,  # Quantize in groups of 128 weights
    desc_act=True,  # Order by activation magnitude
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-70B",
    quantization_config=quantization_config,
    device_map="auto",
)

from awq import AutoAWQForCausalLM

model = AutoAWQForCausalLM.from_pretrained("meta-llama/Llama-3.1-70B")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-70B")

quant_config = {"zero_point": True, "q_group_size": 128, "w_bit": 4}
model.quantize(tokenizer, quant_config=quant_config)
model.save_quantized("llama-3.1-70b-awq")

Common GGUF quantization levels:
  Q2_K  — 2-bit (very aggressive, noticeable quality loss)
  Q4_0  — 4-bit (basic, fast)
  Q4_K_M — 4-bit (k-quant medium, good balance)
  Q5_K_M — 5-bit (higher quality)
  Q6_K  — 6-bit (near-FP16 quality)
  Q8_0  — 8-bit (minimal quality loss)

from vllm import LLM, SamplingParams

# Load model
llm = LLM(
    model="meta-llama/Llama-3.1-70B-Instruct",
    tensor_parallel_size=2,  # Split across 2 GPUs
    quantization="awq",
    max_model_len=8192,
    gpu_memory_utilization=0.9,
)

# Generate
sampling_params = SamplingParams(temperature=0.7, top_p=0.9, max_tokens=512)
outputs = llm.generate(["Explain quantum computing in simple terms"], sampling_params)

$ vllm serve meta-llama/Llama-3.1-70B-Instruct \
    --tensor-parallel-size 2 \
    --quantization awq \
    --max-model-len 8192 \
    --port 8000

$ docker run --gpus all -p 8080:80 \
    ghcr.io/huggingface/text-generation-inference:latest \
    --model-id meta-llama/Llama-3.1-70B-Instruct \
    --quantize awq \
    --num-shard 2 \
    --max-input-length 4096 \
    --max-total-tokens 8192

$ ollama pull llama3.1:70b-instruct-q4_K_M
$ ollama run llama3.1:70b-instruct-q4_K_M "Explain transformers"

# Serve as API
$ ollama serve  # Exposes OpenAI-compatible API on :11434

import ollama

response = ollama.chat(
    model="llama3.1:70b-instruct-q4_K_M",
    messages=[{"role": "user", "content": "Explain transformers"}],
)

$ ./llama-server \
    -m models/llama-3.1-70b-instruct-Q4_K_M.gguf \
    -ngl 80 \       # Offload all layers to GPU
    -c 8192 \       # Context length
    --host 0.0.0.0 \
    --port 8080

Request 1: "Hi"           → generates 3 tokens   (done at step 3, waits until step 50)
Request 2: "Write essay"  → generates 50 tokens  (done at step 50)
Request 3: "Hello"        → generates 2 tokens   (done at step 2, waits until step 50)

GPU utilization: poor after short requests finish

Step 1:  [Req1, Req2, Req3] — all generating
Step 3:  [Req1✓, Req2, Req3] — Req1 done, Req4 enters
Step 4:  [Req4, Req2, Req3✓] — Req3 done, Req5 enters
Step 5:  [Req4, Req2, Req5] — continuous flow
...

Standard decoding (70B model):
  Step 1: generate token 1 (slow)
  Step 2: generate token 2 (slow)
  Step 3: generate token 3 (slow)
  ...
  Total: N slow steps

Speculative decoding:
  Draft model (7B): quickly generate 5 candidate tokens
  Target model (70B): verify all 5 in ONE forward pass (parallel)
  Accept: first 3 match → keep them (3 tokens for cost of ~1 step)
  Reject: token 4 differs → regenerate from there

┌─────────────────────────────────────────────────────────┐
│ Draft model generates: "The quick brown fox jumps"      │
│ Target model verifies:  ✓    ✓     ✓     ✓    ✗        │
│ Accepted tokens:       "The quick brown fox"            │
│ Speedup: 4 tokens verified in 1 target forward pass    │
└─────────────────────────────────────────────────────────┘

When self-hosting makes sense:
  ✓ High volume (>$5K/month API spend)
  ✓ Data privacy requirements (no external API calls)
  ✓ Predictable, sustained load
  ✓ Need for customization (fine-tuned models)
  ✓ Latency-sensitive (co-located with application)

When API providers win:
  ✓ Variable/bursty load
  ✓ Need latest frontier models
  ✓ Small team (no ML ops capacity)
  ✓ Rapid prototyping
  ✓ Low volume

def route_request(query: str) -> str:
    """Route to appropriate model based on complexity."""
    # Use a cheap classifier to determine complexity
    complexity = classify_complexity(query)  # Uses small model or heuristics

    if complexity == "simple":
        return call_model("gpt-4o-mini", query)  # $0.15/1M input
    elif complexity == "medium":
        return call_model("claude-haiku", query)  # $0.80/1M input
    else:
        return call_model("gpt-4o", query)  # $2.50/1M input

First request:  [System prompt: 2000 tokens] + [User: 100 tokens] → full price
Second request: [System prompt: 2000 tokens (CACHED)] + [User: 50 tokens] → cache discount

Anthropic: 90% discount on cached input tokens
OpenAI: 50% discount on cached input tokens

class SemanticCache:
    def __init__(self, vector_store, similarity_threshold: float = 0.95):
        self.store = vector_store
        self.threshold = similarity_threshold

    def get(self, query: str) -> str | None:
        embedding = embed(query)
        results = self.store.search(embedding, limit=1)
        if results and results[0].score > self.threshold:
            return results[0].payload["response"]
        return None

    def set(self, query: str, response: str):
        embedding = embed(query)
        self.store.upsert(vector=embedding, payload={"query": query, "response": response})

Larger batch size → Higher throughput, Higher latency
Smaller batch size → Lower latency, Lower throughput

                Throughput
                    ▲
                    │         ╭──────────
                    │       ╭─╯
                    │     ╭─╯
                    │   ╭─╯
                    │ ╭─╯
                    │╭╯
                    ├──────────────────────▶ Batch Size
                    │
    Latency         │
                    ▲
                    │              ╭────────
                    │           ╭──╯
                    │        ╭──╯
                    │     ╭──╯
                    │  ───╯
                    ├──────────────────────▶ Batch Size

import mlx.core as mx
from mlx_lm import load, generate

model, tokenizer = load("mlx-community/Llama-3.1-8B-Instruct-4bit")
response = generate(model, tokenizer, prompt="Explain edge AI", max_tokens=200)

Prompt: "Return a JSON object with name and age"

Unconstrained output (may fail):
  {"name": "Alice", "age": 30}       ← valid (lucky)
  Sure! Here's the JSON:             ← invalid (preamble)
  {"name": "Alice", age: 30}         ← invalid (missing quotes)
  {"name": "Alice", "age": "thirty"} ← valid JSON, wrong type

from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4o",
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "user_info",
            "strict": True,
            "schema": {
                "type": "object",
                "properties": {
                    "name": {"type": "string"},
                    "age": {"type": "integer", "minimum": 0},
                    "email": {"type": "string", "format": "email"},
                },
                "required": ["name", "age", "email"],
                "additionalProperties": False,
            },
        },
    },
    messages=[{"role": "user", "content": "Generate info for a test user"}],
)

# GBNF grammar for a JSON object with specific fields
root   ::= "{" ws "\"action\":" ws action "," ws "\"reason\":" ws string "}"
action ::= "\"approve\"" | "\"reject\"" | "\"escalate\""
string ::= "\"" [^"\\]* "\""
ws     ::= [ \t\n]*

How grammar-based decoding works:

Normal decoding:     P(token | context) → sample from full vocabulary
Constrained:         P(token | context) → mask invalid tokens → sample from valid set

Token vocabulary: ["{", "}", "\"", "name", "age", ":", ...]
Grammar state: expecting value after "age":
  Valid next tokens:   [0-9, "-"]
  Invalid (masked):    ["}", "\"", "name", ...]

import outlines

model = outlines.models.vllm("meta-llama/Llama-3.1-8B-Instruct")

# Generate matching a regex
generator = outlines.generate.regex(model, r"\d{3}-\d{2}-\d{4}")
result = generator("Generate a fake SSN: ")

# Generate matching a JSON schema
from pydantic import BaseModel

class Review(BaseModel):
    sentiment: str  # "positive" | "negative" | "neutral"
    score: float
    summary: str

generator = outlines.generate.json(model, Review)
review = generator("Review this product: Great quality, fast shipping")

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {"type": "string"},
                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
                },
                "required": ["location"],
            },
        },
    }
]

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Weather in Paris?"}],
    tools=tools,
    tool_choice="auto",
)
# Guaranteed valid: {"name": "get_weather", "arguments": {"location": "Paris"}}

Constrained decoding overhead sources:
  1. Token masking — compute valid token set at each step
  2. Reduced sampling space — may force suboptimal tokens
  3. Grammar state tracking — maintain parser state

Typical overhead:
  Simple JSON schema:  5-10% slower generation
  Complex nested schema: 15-25% slower
  Regex patterns:      10-15% slower
  Free-form grammar:   15-30% slower

Mitigation:
  - Pre-compile grammars/schemas (one-time cost)
  - Use provider-native JSON mode when possible (optimized internally)
  - Keep schemas simple — deep nesting increases overhead

// Server-side: Express + OpenAI streaming
import express from "express";
import OpenAI from "openai";

const app = express();
const openai = new OpenAI();

app.post("/chat/stream", async (req, res) => {
  res.setHeader("Content-Type", "text/event-stream");
  res.setHeader("Cache-Control", "no-cache");
  res.setHeader("Connection", "keep-alive");

  const stream = await openai.chat.completions.create({
    model: "gpt-4o",
    messages: req.body.messages,
    stream: true,
  });

  for await (const chunk of stream) {
    const content = chunk.choices[0]?.delta?.content;
    if (content) {
      res.write(`data: ${JSON.stringify({ content })}\n\n`);
    }
  }

  res.write("data: [DONE]\n\n");
  res.end();
});

// Client-side: consuming SSE stream
async function streamChat(messages: Message[]): AsyncGenerator<string> {
  const response = await fetch("/chat/stream", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ messages }),
  });

  const reader = response.body!.getReader();
  const decoder = new TextDecoder();
  let buffer = "";

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;

    buffer += decoder.decode(value, { stream: true });
    const lines = buffer.split("\n\n");
    buffer = lines.pop() || "";

    for (const line of lines) {
      if (line.startsWith("data: ") && line !== "data: [DONE]") {
        const { content } = JSON.parse(line.slice(6));
        yield content;
      }
    }
  }
}

async def stream_with_tools(messages: list, tools: list):
    """Handle streaming where tool calls may interrupt generation."""
    stream = await client.chat.completions.create(
        model="gpt-4o",
        messages=messages,
        tools=tools,
        stream=True,
    )

    tool_calls_buffer = {}
    async for chunk in stream:
        delta = chunk.choices[0].delta

        # Regular content — stream immediately
        if delta.content:
            yield {"type": "content", "text": delta.content}

        # Tool call accumulation — buffer until complete
        if delta.tool_calls:
            for tc in delta.tool_calls:
                idx = tc.index
                if idx not in tool_calls_buffer:
                    tool_calls_buffer[idx] = {"name": "", "arguments": ""}
                if tc.function.name:
                    tool_calls_buffer[idx]["name"] = tc.function.name
                if tc.function.arguments:
                    tool_calls_buffer[idx]["arguments"] += tc.function.arguments

        # On finish_reason="tool_calls", execute and continue
        if chunk.choices[0].finish_reason == "tool_calls":
            for tc in tool_calls_buffer.values():
                result = await execute_tool(tc["name"], tc["arguments"])
                yield {"type": "tool_result", "name": tc["name"], "result": result}

Pattern 1: Append-only (simple)
  Each token appends to a text buffer → re-render

Pattern 2: Markdown streaming
  Accumulate tokens → parse partial markdown → render HTML
  Challenge: incomplete markdown (e.g., "**bold" without closing)

Pattern 3: Structured streaming
  Buffer until valid JSON fragment → parse and render component
  Used for: charts, tables, code blocks with syntax highlighting

Pattern 4: Optimistic rendering
  Show tokens immediately + reformat on sentence boundaries
  Reduces visual jitter from partial words

from enum import Enum

class Complexity(Enum):
    SIMPLE = "simple"      # Factual Q&A, formatting, classification
    MEDIUM = "medium"      # Summarization, translation, simple reasoning
    COMPLEX = "complex"    # Multi-step reasoning, code generation, analysis

class ModelRouter:
    def __init__(self):
        self.classifier = load_classifier()  # Small fine-tuned model or rules
        self.models = {
            Complexity.SIMPLE: "gpt-4o-mini",
            Complexity.MEDIUM: "claude-3-5-haiku-latest",
            Complexity.COMPLEX: "gpt-4o",
        }

    def route(self, query: str, context: dict) -> str:
        # Heuristic signals for routing
        signals = {
            "token_count": count_tokens(query),
            "has_code": bool(re.search(r"```", query)),
            "requires_reasoning": any(w in query.lower()
                for w in ["analyze", "compare", "design", "why"]),
            "is_classification": context.get("task_type") == "classify",
        }

        complexity = self.classifier.predict(signals)
        return self.models[complexity]

async def cascade_generate(query: str, threshold: float = 0.85) -> str:
    """Try cheap model first, escalate if uncertain."""
    # Level 1: Fast and cheap
    response = await call_model("gpt-4o-mini", query, logprobs=True)
    confidence = compute_confidence(response.logprobs)

    if confidence >= threshold:
        return response.text

    # Level 2: More capable
    response = await call_model("gpt-4o", query, logprobs=True)
    confidence = compute_confidence(response.logprobs)

    if confidence >= threshold:
        return response.text

    # Level 3: Most capable (always accept)
    response = await call_model("claude-sonnet-4", query)
    return response.text

Cascade economics (1000 requests):
  Without cascade: 1000 × $0.003 (GPT-4o) = $3.00
  With cascade:    700 × $0.00015 (mini) + 200 × $0.003 (4o) + 100 × $0.003 (sonnet)
                 = $0.105 + $0.60 + $0.30 = $1.005 (66% savings)

import random
import hashlib

class ModelABTest:
    def __init__(self, experiments: list[dict]):
        self.experiments = experiments
        # Example: [{"model": "gpt-4o", "weight": 0.5},
        #           {"model": "claude-sonnet-4", "weight": 0.5}]

    def select_model(self, user_id: str, experiment_id: str) -> str:
        """Deterministic assignment based on user_id for consistency."""
        hash_input = f"{user_id}:{experiment_id}"
        hash_val = int(hashlib.sha256(hash_input.encode()).hexdigest(), 16)
        bucket = (hash_val % 1000) / 1000.0

        cumulative = 0.0
        for exp in self.experiments:
            cumulative += exp["weight"]
            if bucket < cumulative:
                return exp["model"]
        return self.experiments[-1]["model"]

    def log_result(self, user_id: str, model: str, metrics: dict):
        """Log for analysis: latency, quality score, user satisfaction."""
        emit_metric("ab_test", {
            "user_id": user_id,
            "model": model,
            "ttft_ms": metrics["ttft_ms"],
            "total_tokens": metrics["total_tokens"],
            "quality_score": metrics.get("quality_score"),
            "thumbs_up": metrics.get("thumbs_up"),
        })

import time
from dataclasses import dataclass, field

@dataclass
class InferenceMetrics:
    request_id: str
    model: str
    start_time: float = field(default_factory=time.time)
    first_token_time: float | None = None
    end_time: float | None = None
    input_tokens: int = 0
    output_tokens: int = 0
    error: str | None = None

    @property
    def ttft_ms(self) -> float | None:
        if self.first_token_time:
            return (self.first_token_time - self.start_time) * 1000
        return None

    @property
    def total_latency_ms(self) -> float | None:
        if self.end_time:
            return (self.end_time - self.start_time) * 1000
        return None

    @property
    def tokens_per_second(self) -> float | None:
        if self.end_time and self.first_token_time and self.output_tokens > 1:
            decode_time = self.end_time - self.first_token_time
            return (self.output_tokens - 1) / decode_time if decode_time > 0 else None
        return None

    def emit(self):
        """Send to monitoring system (Datadog, Prometheus, etc.)."""
        metrics = {
            "llm.ttft_ms": self.ttft_ms,
            "llm.total_latency_ms": self.total_latency_ms,
            "llm.tokens_per_second": self.tokens_per_second,
            "llm.input_tokens": self.input_tokens,
            "llm.output_tokens": self.output_tokens,
            "llm.model": self.model,
        }
        send_to_monitoring(metrics)

Safety rules for LLM logging:

  ✓ Log: model name, token counts, latency, finish reason, request ID
  ✓ Log: prompt template name (not content), tool calls made
  ⚠ Conditional: full prompts/completions (only with PII redaction)
  ✗ Never log: API keys, user credentials, raw PII

Redaction strategy:
  1. Hash user identifiers before logging
  2. Strip emails, phone numbers, SSNs via regex
  3. Use a separate, access-controlled store for full prompt logs
  4. Set retention policies (7-30 days for debug, aggregates longer)

@dataclass
class CostTracker:
    """Track per-request and aggregate LLM costs."""

    pricing: dict[str, dict[str, float]]  # model -> {input: $/1M, output: $/1M}

    def request_cost(self, model: str, input_tokens: int, output_tokens: int) -> float:
        rates = self.pricing[model]
        return (input_tokens * rates["input"] + output_tokens * rates["output"]) / 1_000_000

    def daily_report(self, requests: list[InferenceMetrics]) -> dict:
        by_model = {}
        for req in requests:
            cost = self.request_cost(req.model, req.input_tokens, req.output_tokens)
            by_model.setdefault(req.model, {"count": 0, "cost": 0.0, "tokens": 0})
            by_model[req.model]["count"] += 1
            by_model[req.model]["cost"] += cost
            by_model[req.model]["tokens"] += req.input_tokens + req.output_tokens
        return by_model

Quality signals to monitor:

  1. Refusal rate spike — model refusing valid requests
  2. Empty/truncated responses — max_tokens hit or errors
  3. Tool call failure rate — invalid JSON, wrong function names
  4. User feedback (thumbs down) — lagging indicator
  5. Response length anomalies — suddenly shorter/longer than baseline
  6. Latency percentile drift — p95 creeping up over days

Alert thresholds (example):
  - TTFT p95 > 2s for 5 minutes → warning
  - Error rate > 1% for 5 minutes → critical
  - Cost per hour > 2× daily average → warning
  - Refusal rate > 5% → investigate

# Distributed tracing for LLM pipelines
from opentelemetry import trace

tracer = trace.get_tracer("llm-service")

async def handle_request(query: str):
    with tracer.start_as_current_span("llm_request") as span:
        span.set_attribute("llm.query_length", len(query))

        with tracer.start_as_current_span("retrieval"):
            context = await retrieve_documents(query)
            span.set_attribute("retrieval.doc_count", len(context))

        with tracer.start_as_current_span("inference") as llm_span:
            response = await generate(query, context)
            llm_span.set_attribute("llm.model", response.model)
            llm_span.set_attribute("llm.input_tokens", response.input_tokens)
            llm_span.set_attribute("llm.output_tokens", response.output_tokens)
            llm_span.set_attribute("llm.ttft_ms", response.ttft_ms)

        return response.text

┌─────────────────────────────────────────────────────────────┐
│                     Load Balancer                            │
│              (least-connections or queue-aware)              │
└─────────┬──────────────┬──────────────┬────────────────────┘
          │              │              │
    ┌─────▼─────┐  ┌────▼──────┐  ┌───▼───────┐
    │ GPU Node 1│  │ GPU Node 2│  │ GPU Node 3│
    │ 2× H100   │  │ 2× H100   │  │ 2× H100   │
    │ vLLM      │  │ vLLM      │  │ vLLM      │
    │ Queue: 3  │  │ Queue: 7  │  │ Queue: 1  │
    └───────────┘  └───────────┘  └───────────┘

# Auto-scaling policy for GPU inference nodes
scaling_config = {
    "metric": "queue_depth_per_node",
    "scale_up": {
        "threshold": 10,          # Queue > 10 requests per node
        "cooldown_seconds": 300,  # Wait 5 min between scale-ups
        "increment": 1,           # Add 1 node at a time
    },
    "scale_down": {
        "threshold": 2,           # Queue < 2 requests per node
        "cooldown_seconds": 900,  # Wait 15 min (GPU nodes are expensive to start)
        "decrement": 1,
    },
    "limits": {
        "min_nodes": 2,           # Always keep 2 for redundancy
        "max_nodes": 8,           # Budget cap
    },
}

Why queue depth > CPU utilization for LLM scaling:

  CPU/GPU utilization is always high during inference (80-95%).
  It doesn't tell you if users are WAITING.

  Queue depth directly measures user-facing impact:
    Queue = 0-3:  healthy, low latency
    Queue = 5-10: degrading, scale up soon
    Queue = 15+:  users waiting, scale up NOW

Strategy: Tiered degradation

Normal load:     All requests → full model (GPT-4o equivalent)
High load:       Simple requests → smaller model, complex → full model
Overload:        Queue > threshold → reject with retry-after header
                 OR degrade to smaller model for all requests

Implementation:
  1. Request queue with priority levels
  2. Admission control (reject early rather than timeout)
  3. Model downgrade under pressure (70B → 8B)
  4. Pre-computed responses for common queries (cache)
  5. Rate limiting per user/tenant

class AdmissionController:
    """Protect GPU nodes from overload."""

    def __init__(self, max_queue: int = 50, max_pending_tokens: int = 100_000):
        self.max_queue = max_queue
        self.max_pending_tokens = max_pending_tokens

    async def admit(self, request: InferenceRequest) -> AdmissionResult:
        current_queue = await self.get_queue_depth()
        pending_tokens = await self.get_pending_token_count()

        if current_queue >= self.max_queue:
            return AdmissionResult.REJECT  # 429 with Retry-After

        if pending_tokens + request.estimated_tokens > self.max_pending_tokens:
            if request.priority == "low":
                return AdmissionResult.REJECT
            return AdmissionResult.DOWNGRADE  # Use smaller model

        return AdmissionResult.ADMIT

Capacity formula:

  Required GPU nodes = Peak QPS × Avg tokens per request / Tokens per second per node

  Example:
    Peak QPS: 50 requests/second
    Avg output tokens: 200
    Tokens/sec per node (vLLM, 2×H100, 70B-AWQ): ~2000 tok/s

    Required: 50 × 200 / 2000 = 5 nodes

  Add headroom:
    +1 node for redundancy
    +1 node for traffic spikes (20% buffer)
    Total: 7 nodes

  Cost estimate (on-demand H100 instances):
    7 nodes × 2 GPUs × ~$3/hr/GPU = ~$42/hour = ~$30K/month