---
title: "Distributed Systems"
description: "Distributed systems patterns for locking, resilience, idempotency, and rate limiting. Use when implementing distributed locks, circuit breakers, retry policies, idempotency keys, token bucket rate limiters, or fault tolerance patterns."
canonical: "https://orchestkit.yonyon.ai/docs/reference/skills/distributed-systems"
---

# Distributed Systems

Distributed systems patterns for locking, resilience, idempotency, and rate limiting. Use when implementing distributed locks, circuit breakers, retry policies, idempotency keys, token bucket rate limiters, or fault tolerance patterns.

<span className="badge badge-gray">Reference</span> <span className="badge badge-yellow">medium</span>

> **Not directly invocable** — no slash command and no model auto-selection. An agent loads it explicitly via `Read()`.

<ContextualSkillSidebar slug="distributed-systems" />

> **Distributed Systems** Distributed systems patterns for locking, resilience, idempotency, and rate limiting. Use when implementing distributed locks, circuit breakers, retry policies, idempotency keys, token bucket rate limiters, or fault tolerance patterns.


# Distributed Systems Patterns

Comprehensive patterns for building reliable distributed systems. Each category has individual rule files in `rules/` loaded on-demand.

## Quick Reference

| Category | Rules | Impact | When to Use |
|----------|-------|--------|-------------|
| [Distributed Locks](#distributed-locks) | 1 | CRITICAL | Fencing tokens, owner validation; Redis/Redlock and Postgres advisory via upstream docs |
| [Resilience](#resilience) | 3 | CRITICAL | Circuit breakers, retry with backoff, bulkhead isolation |
| [Idempotency](#idempotency) | 1 | HIGH | Idempotency keys; dedup and database-backed storage via upstream docs |
| [Rate Limiting](#rate-limiting) | 2 | HIGH | Token bucket, sliding window; SlowAPI integration via upstream docs |
| [Edge Computing](#edge-computing) | 2 | HIGH | Edge workers, V8 isolates, CDN caching, geo-routing |
| [Event-Driven](#event-driven) | 2 | HIGH | Event sourcing, CQRS, transactional outbox, sagas |

**Total: 11 rules across 6 categories.** Removed topics point at first-party sources in [Upstream coverage](#upstream-coverage-do-not-restate); ork-specific scars live in `$\{CLAUDE_PLUGIN_ROOT\}/skills/distributed-systems/references/ork-delta.md`.

## Quick Start

```python
# Redis distributed lock with Lua scripts
async with RedisLock(redis_client, "payment:order-123"):
    await process_payment(order_id)

# Circuit breaker for external APIs
@circuit_breaker(failure_threshold=5, recovery_timeout=30)
@retry(max_attempts=3, base_delay=1.0)
async def call_external_api():
    ...

# Idempotent API endpoint
@router.post("/payments")
async def create_payment(
    data: PaymentCreate,
    idempotency_key: str = Header(..., alias="Idempotency-Key"),
):
    return await idempotent_execute(db, idempotency_key, "/payments", process)

# Token bucket rate limiting
limiter = TokenBucketLimiter(redis_client, capacity=100, refill_rate=10)
if await limiter.is_allowed(f"user:{user_id}"):
    await handle_request()
```

## Distributed Locks

Coordinate exclusive access to resources across multiple service instances.

| Rule | File | Key Pattern |
|------|------|-------------|
| Fencing Tokens | `$\{CLAUDE_PLUGIN_ROOT\}/skills/distributed-systems/rules/locks-fencing-tokens.md` | Owner validation, TTL, heartbeat extension |

Redis single-node locks, Redlock quorum, and PostgreSQL advisory locks are first-party documented; see [Upstream coverage](#upstream-coverage-do-not-restate).

## Resilience

Production-grade fault tolerance for distributed systems.

| Rule | File | Key Pattern |
|------|------|-------------|
| Circuit Breaker | `$\{CLAUDE_PLUGIN_ROOT\}/skills/distributed-systems/rules/resilience-circuit-breaker.md` | CLOSED/OPEN/HALF_OPEN states, sliding window |
| Retry & Backoff | `$\{CLAUDE_PLUGIN_ROOT\}/skills/distributed-systems/rules/resilience-retry-backoff.md` | Exponential backoff, jitter, error classification |
| Bulkhead Isolation | `$\{CLAUDE_PLUGIN_ROOT\}/skills/distributed-systems/rules/resilience-bulkhead.md` | Semaphore tiers, rejection policies, queue depth |

## Idempotency

Ensure operations can be safely retried without unintended side effects.

| Rule | File | Key Pattern |
|------|------|-------------|
| Idempotency Keys | `$\{CLAUDE_PLUGIN_ROOT\}/skills/distributed-systems/rules/idempotency-keys.md` | Deterministic hashing, Stripe-style headers |

Event-consumer dedup and database-backed idempotency storage follow the Stripe pattern; see [Upstream coverage](#upstream-coverage-do-not-restate).

## Rate Limiting

Protect APIs with distributed rate limiting using Redis.

| Rule | File | Key Pattern |
|------|------|-------------|
| Token Bucket | `$\{CLAUDE_PLUGIN_ROOT\}/skills/distributed-systems/rules/ratelimit-token-bucket.md` | Redis Lua scripts, burst capacity, refill rate |
| Sliding Window | `$\{CLAUDE_PLUGIN_ROOT\}/skills/distributed-systems/rules/ratelimit-sliding-window.md` | Sorted sets, precise counting, no boundary spikes |

SlowAPI + Redis wiring and tiered limits are first-party documented; see [Upstream coverage](#upstream-coverage-do-not-restate).

## Edge Computing

Edge runtime patterns for Cloudflare Workers, Vercel Edge, and Deno Deploy.

| Rule | File | Key Pattern |
|------|------|-------------|
| Edge Workers | `$\{CLAUDE_PLUGIN_ROOT\}/skills/distributed-systems/rules/edge-workers.md` | V8 isolate constraints, Web APIs, geo-routing, auth at edge |
| Edge Caching | `$\{CLAUDE_PLUGIN_ROOT\}/skills/distributed-systems/rules/edge-caching.md` | Cache-aside at edge, CDN headers, KV storage, stale-while-revalidate |

## Event-Driven

Event sourcing, CQRS, saga orchestration, and reliable messaging patterns.

| Rule | File | Key Pattern |
|------|------|-------------|
| Event Sourcing | `$\{CLAUDE_PLUGIN_ROOT\}/skills/distributed-systems/rules/event-sourcing.md` | Event-sourced aggregates, CQRS read models, optimistic concurrency |
| Event Messaging | `$\{CLAUDE_PLUGIN_ROOT\}/skills/distributed-systems/rules/event-messaging.md` | Transactional outbox, saga compensation, idempotent consumers |

## Upstream coverage (do not restate)

These topics were removed from this skill on 2026-07-31 (wrap-plus-delta thinning) because a first-party source maintains them. Consult the source; do not re-add tutorials here. Ork-specific scars for these topics live in `$\{CLAUDE_PLUGIN_ROOT\}/skills/distributed-systems/references/ork-delta.md`.

| Topic | First-party source |
|-------|--------------------|
| Redis single-node locks, Redlock algorithm and quorum | https://redis.io/docs/latest/develop/use/patterns/distributed-locks/ |
| PostgreSQL advisory locks (session and transaction level) | https://www.postgresql.org/docs/current/explicit-locking.html#ADVISORY-LOCKS |
| Circuit breaker pattern, thresholds, setup and rollout guides | https://learn.microsoft.com/azure/architecture/patterns/circuit-breaker |
| Bulkhead pattern deep dive (thread pool, semaphore, tiers) | https://learn.microsoft.com/azure/architecture/patterns/bulkhead |
| Retry strategies, exponential backoff, jitter, retry budgets | https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/ and https://tenacity.readthedocs.io |
| HTTP and LLM provider error classification (retryable vs not) | https://docs.claude.com/en/api/errors and https://platform.openai.com/docs/guides/error-codes |
| Idempotency keys, request dedup, database-backed idempotency | https://docs.stripe.com/api/idempotent_requests |
| Token bucket algorithm and Redis rate-limiting patterns | https://redis.io/glossary/rate-limiting/ |
| FastAPI distributed rate limiting (SlowAPI middleware, tiers) | https://slowapi.readthedocs.io/ |
| LLM fallback chains, provider failover, cost tracking | https://vercel.com/docs/ai-gateway; model ids and pricing at https://docs.claude.com/en/docs/about-claude/pricing |

## Key Decisions

| Decision | Recommendation |
|----------|----------------|
| Lock backend | Redis for speed, PostgreSQL if already using it, Redlock for HA |
| Lock TTL | 2-3x expected operation time |
| Circuit breaker recovery | Half-open probe with sliding window |
| Retry algorithm | Exponential backoff + full jitter |
| Bulkhead isolation | Semaphore-based tiers (Critical/Standard/Optional) |
| Idempotency storage | Redis (speed) + DB (durability), 24-72h TTL |
| Rate limit algorithm | Token bucket for most APIs, sliding window for strict quotas |
| Rate limit storage | Redis (distributed, atomic Lua scripts) |

## When NOT to Use

No separate event-sourcing/saga/CQRS skills exist; they are rules within distributed-systems. But most projects never need them.

| Pattern | Interview | Hackathon | MVP | Growth | Enterprise | Simpler Alternative |
|---------|-----------|-----------|-----|--------|------------|---------------------|
| Event sourcing | OVERKILL | OVERKILL | OVERKILL | OVERKILL | WHEN JUSTIFIED | Append-only table with status column |
| Saga orchestration | OVERKILL | OVERKILL | OVERKILL | SELECTIVE | APPROPRIATE | Sequential service calls with manual rollback |
| Circuit breaker | OVERKILL | OVERKILL | BORDERLINE | APPROPRIATE | REQUIRED | Try/except with timeout |
| Distributed locks | OVERKILL | OVERKILL | BORDERLINE | APPROPRIATE | REQUIRED | Database row-level lock (SELECT FOR UPDATE) |
| CQRS | OVERKILL | OVERKILL | OVERKILL | OVERKILL | WHEN JUSTIFIED | Single model for read/write |
| Transactional outbox | OVERKILL | OVERKILL | OVERKILL | SELECTIVE | APPROPRIATE | Direct publish after commit |
| Rate limiting | OVERKILL | OVERKILL | SIMPLE ONLY | APPROPRIATE | REQUIRED | Nginx rate limit or cloud WAF |

**Rule of thumb:** If you have a single server process, you do not need distributed systems patterns. Use in-process alternatives. Add distribution only when you actually have multiple instances.

## Anti-Patterns (FORBIDDEN)

```python
# LOCKS: Never forget TTL (causes deadlocks)
await redis.set(f"lock:{name}", "1")  # WRONG - no expiry!

# LOCKS: Never release without owner check
await redis.delete(f"lock:{name}")  # WRONG - might release others' lock

# RESILIENCE: Never retry non-retryable errors
@retry(max_attempts=5, retryable_exceptions={Exception})  # Retries 401!

# RESILIENCE: Never put retry outside circuit breaker
@retry  # Would retry when circuit is open!
@circuit_breaker
async def call(): ...

# IDEMPOTENCY: Never use non-deterministic keys
key = str(uuid.uuid4())  # Different every time!

# IDEMPOTENCY: Never cache error responses
if response.status_code >= 400:
    await cache_response(key, response)  # Errors should retry!

# RATE LIMITING: Never use in-memory counters in distributed systems
request_counts = {}  # Lost on restart, not shared across instances
```

## Detailed Documentation

| Resource | Description |
|----------|-------------|
| `$\{CLAUDE_PLUGIN_ROOT\}/skills/distributed-systems/scripts` | Templates: lock implementations, circuit breaker, rate limiter |
| `$\{CLAUDE_PLUGIN_ROOT\}/skills/distributed-systems/references/ork-delta.md` | Ork-specific scars and house decisions kept after the wrap-plus-delta thinning |

## Related Skills

- `caching` - Redis caching patterns, cache as fallback
- `background-jobs` - Job deduplication, async processing with retry
- `observability-monitoring` - Metrics and alerting for circuit breaker state changes
- `error-handling-rfc9457` - Structured error responses for resilience failures
- `auth-patterns` - API key management, authentication integration


---

## Rules (11)

### Configure edge caching with CDN invalidation, TTL, and stale-while-revalidate strategies — HIGH


## Edge Caching & CDN Patterns

Cache responses at the edge using Cache API, KV storage, and CDN headers for sub-millisecond response times.

**Incorrect — caching without TTL or invalidation:**
```typescript
// WRONG: Cache forever, no way to update
export default {
  async fetch(request: Request) {
    const cache = caches.default;
    const cached = await cache.match(request);
    if (cached) return cached; // Stale forever!

    const response = await fetch(request);
    await cache.put(request, response.clone()); // No expiry!
    return response;
  }
};
```

**Correct — cache-aside with TTL and stale-while-revalidate:**
```typescript
export default {
  async fetch(request: Request, env: Env): Promise<Response> {
    const url = new URL(request.url);
    const cacheKey = new Request(url.toString(), request);
    const cache = caches.default;

    // Check edge cache
    let response = await cache.match(cacheKey);
    if (response) return response;

    // Check KV (global, eventually consistent)
    const kvData = await env.CACHE_KV.get(url.pathname, 'text');
    if (kvData) {
      response = new Response(kvData, {
        headers: {
          'Content-Type': 'application/json',
          'Cache-Control': 'public, max-age=60, stale-while-revalidate=300',
          'CDN-Cache-Control': 'max-age=3600', // CDN caches longer
        },
      });
      // Populate edge cache
      await cache.put(cacheKey, response.clone());
      return response;
    }

    // Fetch from origin
    response = await fetch(request);
    const body = await response.text();

    // Store in KV with TTL
    await env.CACHE_KV.put(url.pathname, body, { expirationTtl: 3600 });

    const cachedResponse = new Response(body, {
      headers: {
        'Content-Type': 'application/json',
        'Cache-Control': 'public, max-age=60, stale-while-revalidate=300',
      },
    });
    await cache.put(cacheKey, cachedResponse.clone());
    return cachedResponse;
  }
};
```

**Cache header strategy:**
```
# Static assets (hashed filenames)
Cache-Control: public, max-age=31536000, immutable

# API responses (frequently changing)
Cache-Control: public, max-age=60, stale-while-revalidate=300

# Personalized content
Cache-Control: private, no-cache
Vary: Cookie, Authorization
```

**Key rules:**
- Always set `Cache-Control` with `max-age` and `stale-while-revalidate`
- Use `CDN-Cache-Control` to set different TTLs for CDN vs browser
- Use `Vary` header for content that changes by user/locale
- Cloudflare KV is eventually consistent (read-after-write may be stale)
- Use purge APIs for immediate invalidation of critical content
- Never cache authenticated/personalized responses without `Vary` or `private`


### Deploy edge workers with V8 isolate runtime constraints and fallback handling — HIGH


## Edge Workers & Runtime

Deploy code to Cloudflare Workers, Vercel Edge, or Deno Deploy with correct runtime constraints and platform-specific patterns.

**Incorrect — using Node.js APIs at edge:**
```typescript
// WRONG: Node.js APIs not available in edge runtime
import fs from 'fs';
import { createHash } from 'crypto';

export default async function handler(req: Request) {
  const data = fs.readFileSync('./config.json'); // FAILS at edge
  const hash = createHash('sha256').update(data); // FAILS at edge
  return new Response(hash.digest('hex'));
}
```

**Correct — using Web APIs at edge:**
```typescript
// Cloudflare Worker with Web Crypto API
export default {
  async fetch(request: Request): Promise<Response> {
    const url = new URL(request.url);
    const country = (request as any).cf?.country || 'US';

    // Web Crypto API (available at edge)
    const token = request.headers.get('Authorization')?.replace('Bearer ', '');
    if (token) {
      const key = await crypto.subtle.importKey(
        'raw',
        new TextEncoder().encode(SECRET),
        { name: 'HMAC', hash: 'SHA-256' },
        false,
        ['verify']
      );
    }

    // Geo-based routing
    if (country === 'EU') {
      return Response.redirect(`https://eu.example.com${url.pathname}`);
    }

    return fetch(request);
  }
};
```

**Vercel Edge Middleware pattern:**
```typescript
// middleware.ts (Next.js Edge Middleware)
import { NextResponse } from 'next/server';
import type { NextRequest } from 'next/server';

export const config = { matcher: ['/dashboard/:path*'] };

export function middleware(request: NextRequest) {
  const token = request.cookies.get('session');
  if (!token) {
    return NextResponse.redirect(new URL('/login', request.url));
  }
  // A/B testing via cookie
  const bucket = request.cookies.get('ab-bucket')?.value || 'control';
  return NextResponse.rewrite(new URL(`/${bucket}${request.nextUrl.pathname}`, request.url));
}
```

**Key rules:**
- Edge runtimes use V8 isolates — no `fs`, `path`, `child_process`, native modules
- Available: `fetch`, `Request`, `Response`, `crypto.subtle`, `TextEncoder`, streams
- Keep bundles &lt; 1MB compressed for fast cold starts (&lt; 1ms on Cloudflare)
- Use KV (Cloudflare) or Edge Config (Vercel) for distributed state
- Use Durable Objects for strong consistency when needed


### Guarantee at-least-once delivery using transactional outbox and event messaging patterns — HIGH


## Event Messaging & Outbox

Reliable event publishing with transactional outbox, saga orchestration, and message queue patterns.

**Incorrect — dual-write without outbox (data loss on failure):**
```python
# WRONG: If publish fails, DB has data but event is lost
async def create_order(order: Order):
    await db.insert(order)              # Step 1: succeeds
    await message_broker.publish(        # Step 2: might fail!
        "order.created", order.model_dump()
    )
    # If publish fails: order exists but no event was sent
    # Downstream services never know about the order
```

**Correct — transactional outbox pattern:**
```python
# Outbox table: events written atomically with business data
async def create_order(order: Order, db: AsyncSession):
    async with db.begin():
        # Both in same transaction — atomic!
        db.add(order)
        db.add(OutboxEvent(
            aggregate_id=order.id,
            event_type="order.created",
            payload=order.model_dump(),
            created_at=datetime.now(datetime.UTC),
        ))

# Outbox publisher (separate process, polls for unsent events)
async def publish_outbox_events(db: AsyncSession, broker: MessageBroker):
    while True:
        async with db.begin():
            events = await db.execute(
                select(OutboxEvent)
                .where(OutboxEvent.published_at.is_(None))
                .order_by(OutboxEvent.created_at)
                .limit(100)
                .with_for_update(skip_locked=True)  # Concurrent workers safe
            )
            for event in events.scalars():
                await broker.publish(event.event_type, event.payload)
                event.published_at = datetime.now(datetime.UTC)
        await asyncio.sleep(1)
```

**Saga orchestration pattern:**
```python
class OrderSaga:
    steps = [
        SagaStep("reserve_inventory", compensate="release_inventory"),
        SagaStep("charge_payment", compensate="refund_payment"),
        SagaStep("ship_order", compensate="cancel_shipment"),
    ]

    async def execute(self, order_id: str):
        completed = []
        for step in self.steps:
            try:
                await step.execute(order_id)
                completed.append(step)
            except Exception:
                # Compensate in reverse order
                for s in reversed(completed):
                    await s.compensate(order_id)
                raise SagaFailed(f"Failed at {step.name}")
```

**Idempotent consumer (prevents duplicate processing):**
```python
async def handle_event(event: Event, db: AsyncSession):
    async with db.begin():
        # Check if already processed
        exists = await db.execute(
            select(ProcessedEvent).where(ProcessedEvent.event_id == event.id)
        )
        if exists.scalar():
            return  # Already processed, skip

        # Process the event
        await process_order(event.payload)

        # Mark as processed
        db.add(ProcessedEvent(event_id=event.id, processed_at=datetime.now(datetime.UTC)))
```

**Key rules:**
- Use transactional outbox to atomically save data and events
- All saga steps must have compensating actions
- Every message consumer must be idempotent (use event ID deduplication)
- Use `SKIP LOCKED` for concurrent outbox workers
- Kafka for high-throughput streaming, RabbitMQ for routing, Redis Streams for simplicity
- Use dead letter queues (DLQ) for messages that fail after max retries


### Implement event sourcing with CQRS for full audit trails and temporal queries — HIGH


## Event Sourcing & CQRS

Store state as immutable events and separate read/write models for scalable, auditable systems.

**Incorrect — mutable state without event history:**
```python
# WRONG: Direct mutation loses history
class Account:
    def __init__(self, balance: float = 0):
        self.balance = balance

    def deposit(self, amount: float):
        self.balance += amount  # No record of what happened!

    def withdraw(self, amount: float):
        self.balance -= amount  # Can't audit, can't replay
```

**Correct — event-sourced aggregate with CQRS:**
```python
from dataclasses import dataclass, field
from typing import Any

@dataclass
class DomainEvent:
    aggregate_id: str
    version: int
    data: dict[str, Any]

class Account:
    def __init__(self):
        self._changes: list[DomainEvent] = []
        self._version = 0
        self.balance = 0.0

    def deposit(self, amount: float):
        if amount <= 0:
            raise ValueError("Amount must be positive")
        self._raise_event("MoneyDeposited", {"amount": amount})

    def withdraw(self, amount: float):
        if amount > self.balance:
            raise ValueError("Insufficient funds")
        self._raise_event("MoneyWithdrawn", {"amount": amount})

    def _raise_event(self, event_type: str, data: dict):
        event = DomainEvent(
            aggregate_id=self.id,
            version=self._version + 1,
            data={"type": event_type, **data},
        )
        self._apply(event)
        self._changes.append(event)

    def _apply(self, event: DomainEvent):
        match event.data["type"]:
            case "MoneyDeposited":
                self.balance += event.data["amount"]
            case "MoneyWithdrawn":
                self.balance -= event.data["amount"]
        self._version = event.version

# Event store with optimistic concurrency
class EventStore:
    async def save(self, aggregate_id: str, events: list[DomainEvent], expected_version: int):
        async with self.db.transaction():
            current = await self.db.fetchval(
                "SELECT MAX(version) FROM events WHERE aggregate_id = $1",
                aggregate_id,
            )
            if current != expected_version:
                raise ConcurrencyError(f"Expected {expected_version}, got {current}")
            for event in events:
                await self.db.execute(
                    "INSERT INTO events (aggregate_id, version, data) VALUES ($1, $2, $3)",
                    aggregate_id, event.version, event.data,
                )

# CQRS: Separate read model projection
class BalanceProjection:
    async def handle(self, event: DomainEvent):
        match event.data["type"]:
            case "MoneyDeposited":
                await self.db.execute(
                    "UPDATE account_balances SET balance = balance + $1 WHERE id = $2",
                    event.data["amount"], event.aggregate_id,
                )
            case "MoneyWithdrawn":
                await self.db.execute(
                    "UPDATE account_balances SET balance = balance - $1 WHERE id = $2",
                    event.data["amount"], event.aggregate_id,
                )
```

**Key rules:**
- Events are immutable and named in past tense (`OrderPlaced`, not `PlaceOrder`)
- Use optimistic concurrency with version checks to prevent conflicts
- CQRS read models are eventually consistent — design UX accordingly
- Snapshot every N events (e.g., 100) to avoid replaying long event streams
- Never delete events — use compensating events instead


### Generate deterministic idempotency keys using Stripe-style headers for safe retries — HIGH


# Idempotency Key Generation & Stripe-Style Header

## Deterministic Key Generation

```python
import hashlib
import json
from typing import Any

def generate_idempotency_key(
    *,
    entity_id: str,
    action: str,
    params: dict[str, Any] | None = None,
) -> str:
    """Generate deterministic idempotency key.

    Same input always produces the same key.
    """
    content = f"{entity_id}:{action}"
    if params:
        content += f":{json.dumps(params, sort_keys=True)}"
    return hashlib.sha256(content.encode()).hexdigest()[:32]

# Examples
key = generate_idempotency_key(
    entity_id="order-123",
    action="create",
    params={"amount": 100, "currency": "USD"},
)
```

## Stripe-Style Idempotency Header

Clients send `Idempotency-Key` header with POST requests. Server caches successful responses and replays them on duplicate requests.

```
Client Request (Idempotency-Key: abc-123)
     |
     v
Check cache (Redis) --> Exists? --> Return cached response
     |                                (Idempotent-Replayed: true)
     NO
     |
     v
Acquire lock --> Process request --> Cache response --> Return
```

## FastAPI Middleware

```python
from fastapi import Request, Response
from starlette.middleware.base import BaseHTTPMiddleware
import redis.asyncio as redis
import json

class IdempotencyMiddleware(BaseHTTPMiddleware):
    """Handle Idempotency-Key header for POST/PUT/PATCH."""

    def __init__(self, app, redis_client: redis.Redis, ttl: int = 86400):
        super().__init__(app)
        self.redis = redis_client
        self.ttl = ttl

    async def dispatch(self, request: Request, call_next):
        if request.method not in ("POST", "PUT", "PATCH"):
            return await call_next(request)

        idempotency_key = request.headers.get("Idempotency-Key")
        if not idempotency_key:
            return await call_next(request)

        cache_key = f"idem:{request.url.path}:{idempotency_key}"

        # Check for cached response
        cached = await self.redis.get(cache_key)
        if cached:
            data = json.loads(cached)
            return Response(
                content=data["body"],
                status_code=data["status"],
                media_type="application/json",
                headers={"X-Idempotent-Replayed": "true"},
            )

        # Process request
        response = await call_next(request)

        # Cache successful responses only
        if 200 <= response.status_code < 300:
            body = b"".join([chunk async for chunk in response.body_iterator])
            await self.redis.setex(
                cache_key, self.ttl,
                json.dumps({"body": body.decode(), "status": response.status_code}),
            )
            return Response(content=body, status_code=response.status_code,
                          media_type=response.media_type)

        return response
```

## Key Principles

1. **Keys are deterministic** -- same input = same key (never use uuid4)
2. **Keys are scoped to endpoint** -- same key on different endpoints = different operations
3. **24-hour window** -- keys expire after 24 hours
4. **Only cache success** -- errors (4xx/5xx) allow retry
5. **Lock during processing** -- prevents concurrent duplicates

## Common Mistakes

```python
# NEVER use non-deterministic keys
key = str(uuid.uuid4())  # Different every time!

# NEVER include timestamps in keys
key = f"{event.id}:{datetime.now()}"  # Timestamp varies!

# NEVER skip idempotency for financial operations
@router.post("/payments")
async def create_payment(data):
    return await process_payment(data)  # No idempotency!
```

**Incorrect — Random UUID keys make retry detection impossible:**
```python
# Client generates new key on each retry
key = str(uuid.uuid4())  # New key every time!
await post("/api/orders", headers={"Idempotency-Key": key}, data=order)
# Retry creates duplicate order
```

**Correct — Deterministic keys ensure retries are detected and deduplicated:**
```python
# Client generates same key for same operation
key = hashlib.sha256(f"{order_id}:create:{json.dumps(order_data, sort_keys=True)}".encode()).hexdigest()
await post("/api/orders", headers={"Idempotency-Key": key}, data=order)
# Retry returns cached response
```


### Validate lock ownership with fencing tokens and TTL to prevent data corruption — CRITICAL


# Lock Safety: Fencing Tokens, TTL & Heartbeat

## Owner Validation (Fencing)

Every lock operation MUST validate the owner before acting. Without this, a slow process whose lock expired can corrupt data when a new owner holds the lock.

```python
# WRONG: No owner check
await redis.delete(f"lock:{name}")  # Might release someone else's lock!

# CORRECT: Atomic owner check via Lua
RELEASE = """
if redis.call('get', KEYS[1]) == ARGV[1] then
    return redis.call('del', KEYS[1])
end
return 0
"""
```

## TTL Management

Lock TTL must be set to prevent deadlocks from crashed processes. Rule of thumb: TTL = 2-3x expected operation duration.

| Operation Duration | Recommended TTL | Rationale |
|-------------------|-----------------|-----------|
| &lt; 1 second | 5 seconds | Fast operations with margin |
| 1-10 seconds | 30 seconds | Standard processing |
| 10-60 seconds | 3 minutes | Long operations, use heartbeat |
| > 60 seconds | 5 minutes + heartbeat | Must extend during processing |

## Heartbeat Extension

For long-running tasks, extend the lock TTL periodically to prevent expiry mid-operation.

```python
import asyncio
from datetime import timedelta

async def long_running_task(task_id: str, redis_client):
    lock = RedisLock(redis_client, f"task:{task_id}", ttl=timedelta(seconds=30))

    async with lock:
        # Background heartbeat extends lock every 10s
        async def heartbeat():
            while lock.is_acquired:
                await lock.extend(timedelta(seconds=30))
                await asyncio.sleep(10)

        heartbeat_task = asyncio.create_task(heartbeat())
        try:
            await do_long_work()
        finally:
            heartbeat_task.cancel()
```

## Lock Retry with Exponential Backoff

```python
from functools import wraps

def with_lock(lock_name: str, ttl_s: int = 30, retries: int = 3):
    """Decorator to acquire lock before function execution."""
    def decorator(func):
        @wraps(func)
        async def wrapper(*args, redis_client=None, **kwargs):
            for attempt in range(retries):
                lock = RedisLock(redis_client, lock_name, ttl_ms=ttl_s * 1000)
                if await lock.acquire():
                    try:
                        return await func(*args, **kwargs)
                    finally:
                        await lock.release()
                await asyncio.sleep(0.1 * (2 ** attempt))  # Backoff
            raise LockAcquisitionError(f"Failed after {retries} attempts")
        return wrapper
    return decorator
```

## Lock Ordering (Deadlock Prevention)

When acquiring multiple locks, always acquire in a consistent order.

```python
async def transfer_funds(session, from_account: int, to_account: int, amount):
    # Always lock in sorted order to prevent deadlocks
    accounts = sorted([from_account, to_account])
    for account_id in accounts:
        await session.execute(
            text("SELECT pg_advisory_xact_lock(:ns, :id)"),
            {"ns": NAMESPACE_ACCOUNT, "id": account_id},
        )
    await debit_account(session, from_account, amount)
    await credit_account(session, to_account, amount)
    await session.commit()
```

## Checklist

- Owner ID stored with lock (UUIDv7 recommended)
- Atomic release validates owner via Lua script
- TTL always set (prevents permanent deadlocks)
- Heartbeat for operations > 30 seconds
- Lock ordering for multiple locks
- Retry with exponential backoff + jitter
- Metrics: acquisition time, hold duration, failures

**Incorrect — Releasing lock without owner validation can delete another process's lock:**
```python
await redis.delete(f"lock:{resource_id}")
# If our lock expired and another process acquired it,
# we just deleted their lock!
```

**Correct — Atomic owner validation via Lua ensures only owner can release:**
```python
RELEASE_SCRIPT = """
if redis.call('get', KEYS[1]) == ARGV[1] then
  return redis.call('del', KEYS[1])
end
return 0
"""
result = await redis.eval(RELEASE_SCRIPT, keys=[f"lock:{resource_id}"], args=[owner_id])
# Only deletes if owner_id matches
```


### Implement sliding window rate limiting with Redis sorted sets to prevent boundary spikes — HIGH


# Sliding Window Rate Limiting

Precise rate limiting that avoids fixed window boundary spikes.

## Problem with Fixed Windows

A user can hit 100 at 0:59 and 100 at 1:01 = 200 requests in 2 seconds with a "100/minute" limit.

Sliding window solves this by tracking individual request timestamps.

## Redis Implementation

```python
import redis.asyncio as redis
from datetime import datetime, timezone

class SlidingWindowLimiter:
    def __init__(self, redis_client: redis.Redis, window_seconds: int = 60):
        self.redis = redis_client
        self.window = window_seconds

    async def is_allowed(self, key: str, limit: int) -> tuple[bool, int]:
        """Returns (allowed, remaining)."""
        now = datetime.now(timezone.utc).timestamp()
        window_start = now - self.window
        bucket_key = f"ratelimit:sliding:{key}"

        pipe = self.redis.pipeline()
        pipe.zremrangebyscore(bucket_key, 0, window_start)  # Remove old
        pipe.zcard(bucket_key)                               # Count current
        pipe.zadd(bucket_key, {str(now): now})               # Add this request
        pipe.expire(bucket_key, self.window * 2)             # Set expiry

        results = await pipe.execute()
        current_count = results[1]

        if current_count < limit:
            return True, limit - current_count - 1
        return False, 0
```

## Atomic Lua Script (Better for Production)

```lua
local key = KEYS[1]
local limit = tonumber(ARGV[1])
local window = tonumber(ARGV[2])
local now = tonumber(ARGV[3])

-- Remove old entries
redis.call('ZREMRANGEBYSCORE', key, 0, now - window * 1000)

-- Count current entries
local count = redis.call('ZCARD', key)

if count < limit then
    redis.call('ZADD', key, now, now .. ':' .. math.random())
    redis.call('EXPIRE', key, window)
    return {1, limit - count - 1, 0}
else
    local oldest = redis.call('ZRANGE', key, 0, 0, 'WITHSCORES')
    local retry_after = 0
    if oldest[2] then
        retry_after = math.ceil((tonumber(oldest[2]) + window * 1000 - now) / 1000)
    end
    return {0, 0, retry_after}
end
```

## When to Use

| Sliding Window | Token Bucket |
|---------------|-------------|
| Strict quotas (billing) | General API limits |
| No burst tolerance | Burst-friendly |
| Higher memory (O(n) timestamps) | O(1) memory |
| Exact counting | Approximate counting |

## Best Practices

1. Use Redis sorted sets with timestamps as scores
2. Clean expired entries on every check (ZREMRANGEBYSCORE)
3. Set EXPIRE on the key to auto-cleanup inactive users
4. Use pipeline or Lua script for atomicity
5. Consider memory: each request stores a member in the sorted set

**Incorrect — Fixed window allows 200 requests in 2 seconds with "100/min" limit:**
```python
# Fixed window resets at minute boundaries
window_start = int(time.time() / 60) * 60
if get_count(f"{user}:{window_start}") < 100:
    allow()
# User can send 100 at 0:59 and 100 at 1:01 = 200 in 2 seconds!
```

**Correct — Sliding window tracks individual timestamps for precise limiting:**
```python
now = time.time()
window_start = now - 60  # Last 60 seconds
await redis.zremrangebyscore(key, 0, window_start)  # Remove old
count = await redis.zcard(key)  # Count current
if count < 100:
    await redis.zadd(key, {str(now): now})  # Add request
# Exactly 100 requests per rolling 60-second window
```


### Implement token bucket rate limiting with atomic Redis operations and burst capacity — HIGH


# Token Bucket Rate Limiting

Allows bursts up to bucket capacity while maintaining a steady average rate.

## How It Works

```
Bucket: capacity=10, refill_rate=5/sec

t=0s: 10 tokens | 10 requests -> 0 tokens (burst allowed)
t=1s: +5 tokens | 5 tokens available
t=2s: +5 tokens | 10 tokens (capped at capacity)
```

## Redis Implementation (Atomic Lua Script)

```python
import redis.asyncio as redis
from datetime import datetime, timezone

class TokenBucketLimiter:
    SCRIPT = """
    local key = KEYS[1]
    local capacity = tonumber(ARGV[1])
    local refill_rate = tonumber(ARGV[2])
    local tokens_requested = tonumber(ARGV[3])
    local now = tonumber(ARGV[4])

    local bucket = redis.call('HMGET', key, 'tokens', 'last_update')
    local current_tokens = tonumber(bucket[1]) or capacity
    local last_update = tonumber(bucket[2]) or now

    -- Calculate refill
    local elapsed = (now - last_update) / 1000
    current_tokens = math.min(capacity, current_tokens + elapsed * refill_rate)

    -- Check and consume
    local allowed = 0
    local remaining = math.floor(current_tokens)
    local retry_after = 0

    if current_tokens >= tokens_requested then
        allowed = 1
        remaining = math.floor(current_tokens - tokens_requested)
        current_tokens = current_tokens - tokens_requested
    else
        local needed = tokens_requested - current_tokens
        retry_after = math.ceil(needed / refill_rate)
    end

    redis.call('HMSET', key, 'tokens', current_tokens, 'last_update', now)
    redis.call('EXPIRE', key, math.ceil(capacity / refill_rate) * 2)
    return {allowed, remaining, retry_after}
    """

    def __init__(self, redis_client: redis.Redis,
                 capacity: int = 100, refill_rate: float = 10.0):
        self.redis = redis_client
        self.capacity = capacity
        self.refill_rate = refill_rate

    async def is_allowed(self, key: str, tokens: int = 1) -> bool:
        now = datetime.now(timezone.utc).timestamp() * 1000
        result = await self.redis.eval(
            self.SCRIPT, 1, f"ratelimit:token:{key}",
            self.capacity, self.refill_rate, tokens, now,
        )
        return result[0] == 1
```

## Properties

| Property | Description |
|----------|-------------|
| **Burst Capacity** | Allows short bursts up to bucket size |
| **Smooth Limiting** | Tokens refill continuously |
| **O(1) Memory** | Only stores tokens + timestamp per key |
| **Distributed** | Atomic via Redis Lua script |

## When to Use

**Good for:** API rate limiting (natural bursts), user actions (login), resource protection

**Not ideal for:** Strict per-second quotas (use sliding window), billing limits, fair queuing (use leaky bucket)

## vs Sliding Window

| Aspect | Token Bucket | Sliding Window |
|--------|-------------|----------------|
| Burst Handling | Allows up to capacity | Spreads evenly |
| Memory | O(1) per key | O(n) timestamps |
| Precision | Approximate | Exact |
| Redis Operations | 1 HMSET | 1 ZADD + 1 ZREMRANGEBYSCORE |

**Incorrect — Check-then-act pattern has race condition in token refill:**
```typescript
const bucket = await redis.get(`bucket:${user}`);
const tokens = JSON.parse(bucket).tokens + elapsed * refillRate;
if (tokens >= 1) {
  // Race! Another request can pass this check too
  await redis.set(`bucket:${user}`, JSON.stringify({tokens: tokens - 1}));
}
```

**Correct — Atomic Lua script ensures thread-safe token bucket operations:**
```typescript
const result = await redis.eval(TOKEN_BUCKET_SCRIPT,
  keys: [`bucket:${user}`],
  args: [capacity, refillRate, tokensRequested, Date.now()]
);
// Single atomic operation, no race conditions
```


### Isolate failures with bulkhead partitioning and tier-based resource capacity limits — CRITICAL


# Bulkhead Pattern

Isolates failures by partitioning resources into independent pools. One failing component does not bring down the entire system.

## Tier-Based Configuration

| Tier | Workers | Queue | Timeout | Use Case |
|------|---------|-------|---------|----------|
| 1 (Critical) | 5 | 10 | 180-300s | Synthesis, quality gate, user-facing |
| 2 (Standard) | 8 | 12 | 120s | Analysis agents, data processing |
| 3 (Optional) | 4 | 6 | 60s | Enrichment, caching, analytics |

## Implementation

```python
from asyncio import Semaphore, wait_for, TimeoutError
from enum import Enum

class Tier(Enum):
    CRITICAL = 1
    STANDARD = 2
    OPTIONAL = 3

class Bulkhead:
    def __init__(self, tier: Tier, max_concurrent: int, queue_size: int, timeout: float):
        self.tier = tier
        self.semaphore = Semaphore(max_concurrent)
        self.queue_size = queue_size
        self.timeout = timeout
        self.waiting = 0
        self.active = 0

    async def execute(self, fn):
        if self.waiting >= self.queue_size:
            raise BulkheadFullError(f"Tier {self.tier.name} queue full")

        self.waiting += 1
        try:
            await wait_for(self.semaphore.acquire(), timeout=self.timeout)
            self.waiting -= 1
            self.active += 1
            try:
                return await wait_for(fn(), timeout=self.timeout)
            finally:
                self.active -= 1
                self.semaphore.release()
        except TimeoutError:
            self.waiting -= 1
            raise BulkheadTimeoutError(f"Tier {self.tier.name} timeout")
```

## Rejection Policies

```python
class RejectionPolicy(Enum):
    ABORT = "abort"        # Return error immediately
    CALLER_RUNS = "caller" # Execute in caller's context (blocking)
    DISCARD = "discard"    # Silently drop (for optional ops)
    QUEUE = "queue"        # Wait in bounded queue

TIER_POLICIES = {
    Tier.CRITICAL: RejectionPolicy.QUEUE,      # Wait for slot
    Tier.STANDARD: RejectionPolicy.CALLER_RUNS, # Degrade caller
    Tier.OPTIONAL: RejectionPolicy.DISCARD,     # Skip if busy
}
```

## Graceful Degradation by Tier

```python
async def run_analysis(content: str):
    results = {}

    # Tier 1: Must succeed
    results["core"] = await tier1_bulkhead.execute(
        lambda: analyze_core(content)
    )

    # Tier 2: Best effort
    try:
        results["enriched"] = await tier2_bulkhead.execute(
            lambda: enrich_analysis(content)
        )
    except BulkheadFullError:
        results["enriched"] = None  # Skip enrichment

    # Tier 3: Optional (silent failure)
    try:
        await tier3_bulkhead.execute(lambda: warm_cache(results))
    except (BulkheadFullError, BulkheadTimeoutError):
        pass

    return results
```

## Best Practices

1. **Size based on downstream capacity** -- if API allows 60 RPM, don't set 100 concurrent
2. **Monitor queue depth** -- alert when consistently > 80% full
3. **Combine with circuit breaker** -- slow calls trigger circuit, clearing bulkhead slots
4. **Use per-dependency bulkheads** -- not per-endpoint (too granular)
5. **Return 503 with Retry-After** -- when rejecting, don't return 500

## Common Mistakes

- Too many bulkheads (per-endpoint instead of per-dependency)
- Ignoring rejection handling (BulkheadFullError becomes 500)
- No correlation with circuit breaker (slots stay blocked on slow service)

**Incorrect — No isolation means slow downstream service blocks all operations:**
```python
async def fetch_data():
    await slow_external_api()  # Takes 30s when degraded
# All requests wait, entire system becomes slow
```

**Correct — Bulkhead isolates slow service, protecting critical operations:**
```python
@bulkhead(tier=Tier.OPTIONAL, max_concurrent=3, timeout=5)
async def fetch_data():
    await slow_external_api()
# Only 3 concurrent + waiting queue, rest get fast rejection
```


### Prevent cascade failures with circuit breaker thresholds and recovery probe patterns — CRITICAL


# Circuit Breaker Pattern

Prevents cascade failures by "tripping" when a downstream service exceeds failure thresholds.

## State Machine

```
              failures >= threshold
    CLOSED --------------------------------> OPEN
       ^                                      |
       |                                      |
       | probe succeeds              timeout  |
       |                              expires |
       |         +-------------+              |
       +---------+  HALF_OPEN  |<-------------+
                 +-------------+
                       |
                       | probe fails
                       v
                     OPEN
```

- **CLOSED**: All requests pass through, failures counted in sliding window
- **OPEN**: All requests rejected immediately, return fallback
- **HALF_OPEN**: Limited probe requests test recovery

## Configuration

| Parameter | Recommended | Description |
|-----------|-------------|-------------|
| `failure_threshold` | 5 | Failures before opening |
| `success_threshold` | 2 | Successes in half-open to close |
| `recovery_timeout` | 30s | Time before half-open transition |
| `sliding_window_size` | 10 | Requests to consider for failure rate |
| `slow_call_threshold` | 5-30s | Calls slower than this count as failures |

## Implementation

```python
from collections import deque
from enum import Enum
from time import time

class CircuitState(Enum):
    CLOSED = "closed"
    OPEN = "open"
    HALF_OPEN = "half_open"

class CircuitBreaker:
    def __init__(self, name: str, failure_threshold: int = 5,
                 recovery_timeout: float = 30.0):
        self.name = name
        self._state = CircuitState.CLOSED
        self._failure_count = 0
        self._success_count = 0
        self._last_failure_time = None
        self._threshold = failure_threshold
        self._recovery_timeout = recovery_timeout

    async def call(self, fn, *args, **kwargs):
        if self._state == CircuitState.OPEN:
            if self._should_attempt_recovery():
                self._state = CircuitState.HALF_OPEN
            else:
                raise CircuitOpenError(self.name)

        try:
            result = await fn(*args, **kwargs)
            self._on_success()
            return result
        except Exception as e:
            self._on_failure()
            raise

    def _on_success(self):
        if self._state == CircuitState.HALF_OPEN:
            self._success_count += 1
            if self._success_count >= 2:
                self._state = CircuitState.CLOSED
                self._failure_count = 0

    def _on_failure(self):
        self._last_failure_time = time()
        if self._state == CircuitState.HALF_OPEN:
            self._state = CircuitState.OPEN
        else:
            self._failure_count += 1
            if self._failure_count >= self._threshold:
                self._state = CircuitState.OPEN
```

## Best Practices

1. **Use sliding windows, not fixed counters** -- one success should not reset everything
2. **Per-service breakers** -- never use a single global breaker
3. **Always provide fallbacks** -- cached data, default response, or partial results
4. **Separate health from circuit state** -- `/health` always returns 200
5. **Include observability** -- every state change = metric + log + alert on OPEN

## Pattern Composition

```python
# Retry INSIDE circuit breaker
@circuit_breaker(failure_threshold=5)
@retry(max_attempts=3, backoff=exponential)
async def call_service():
    ...

# Bulkhead + Circuit breaker
@circuit_breaker(service="analysis")
@bulkhead(tier=Tier.STANDARD, max_concurrent=3)
async def analyze():
    ...
```

## Presets by Service Type

| Service | Threshold | Recovery | Slow Call |
|---------|-----------|----------|-----------|
| LLM API | 3 | 60s | 30s |
| External API | 5 | 30s | 10s |
| Database | 2-3 | 15s | 5s |

**Incorrect — No circuit breaker causes cascade failure when downstream is down:**
```python
async def call_payment_api():
    return await http.post("https://api.payment.com/charge")
    # Keeps trying even when API is down, causing 30s timeouts on every request
# Entire service becomes unresponsive
```

**Correct — Circuit breaker trips on failures, returning fast failures and allowing recovery:**
```python
@circuit_breaker(failure_threshold=5, recovery_timeout=30)
async def call_payment_api():
    return await http.post("https://api.payment.com/charge")
# After 5 failures, circuit opens and returns fallback immediately
```


### Implement exponential backoff with jitter to prevent thundering herd on retries — CRITICAL


# Retry with Exponential Backoff

## Backoff Formula

```
delay = min(base * 2^attempt, max_delay)
jitter = random(0, delay)   # Full jitter (recommended)
sleep(jitter)
```

| Attempt | Base Delay | With Full Jitter |
|---------|-----------|------------------|
| 1 | 1s | 0.0s - 1.0s |
| 2 | 2s | 0.0s - 2.0s |
| 3 | 4s | 0.0s - 4.0s |
| 4 | 8s | 0.0s - 8.0s |

Full jitter prevents thundering herd when many clients retry simultaneously.

## Error Classification

```python
RETRYABLE_ERRORS = {
    # HTTP Status Codes
    408, 429, 500, 502, 503, 504,
    # Python Exceptions
    ConnectionError, TimeoutError, ConnectionResetError,
    # LLM API Errors
    "rate_limit_exceeded", "model_overloaded", "server_error",
}

NON_RETRYABLE_ERRORS = {
    400, 401, 403, 404, 422,
    "invalid_api_key", "content_policy_violation",
    "invalid_request_error", "model_not_found",
}
```

## Retry Decorator

```python
import asyncio
import random
from functools import wraps

def retry(max_attempts=3, base_delay=1.0, max_delay=60.0, jitter=True):
    """Async retry with exponential backoff."""
    def decorator(fn):
        @wraps(fn)
        async def wrapper(*args, **kwargs):
            for attempt in range(1, max_attempts + 1):
                try:
                    return await fn(*args, **kwargs)
                except Exception as e:
                    if not is_retryable(e) or attempt == max_attempts:
                        raise
                    delay = min(base_delay * (2 ** (attempt - 1)), max_delay)
                    if jitter:
                        delay = random.uniform(0, delay)
                    await asyncio.sleep(delay)
        return wrapper
    return decorator
```

## Retry with Content Truncation (LLM)

```python
async def retry_with_truncation(fn, content: str, max_attempts: int = 3):
    """Retry LLM call, truncating on context_length_exceeded."""
    for attempt in range(1, max_attempts + 1):
        try:
            return await fn(content)
        except ContextLengthExceededError:
            if attempt == max_attempts:
                raise
            content = content[:int(len(content) * 0.75)]
```

## Retry Budget

Prevents retry storms by limiting total retries per time window.

```python
class RetryBudget:
    def __init__(self, budget_per_second: float = 10.0):
        self.budget = budget_per_second
        self.last_update = time.time()

    def can_retry(self) -> bool:
        self._replenish()
        return self.budget >= 1.0

    def use_retry(self):
        if self.budget >= 1.0:
            self.budget -= 1.0
```

## Presets

| Use Case | Max Attempts | Base Delay | Max Delay |
|----------|-------------|-----------|-----------|
| User-facing API | 2 | 0.5s | 2s |
| Background job | 5 | 2.0s | 60s |
| LLM API call | 3 | 1.0s | 60s |
| Rate-limited API | 3 | 2.0s | 120s |

## Critical Rules

1. **Always use jitter** -- prevents thundering herd
2. **Classify errors** -- never retry 401/403/404
3. **Bound retries** -- max 3-5 attempts, never infinite
4. **Retry inside circuit breaker** -- circuit only sees final result
5. **Use Retry-After header** -- respect server's backoff request
6. **Log all retries** -- include trace ID for correlation

**Incorrect — Fixed delay without jitter causes thundering herd when service recovers:**
```python
for attempt in range(3):
    try:
        return await api_call()
    except Exception:
        await asyncio.sleep(1)  # All clients retry at same time!
# 1000 clients = 1000 simultaneous retries
```

**Correct — Exponential backoff with jitter spreads retries over time:**
```python
for attempt in range(3):
    try:
        return await api_call()
    except Exception:
        delay = min(2 ** attempt, 60)  # 1s, 2s, 4s...
        await asyncio.sleep(random.uniform(0, delay))  # Jitter spreads load
```



---

## References (1)

### Ork Delta

# Distributed Systems: OrchestKit Delta

House rules and scars kept after the 2026-07-31 wrap-plus-delta thinning of this skill.
Vendor pattern tutorials were removed; SKILL.md section "Upstream coverage (do not restate)"
maps every removed topic to its first-party source. Maintained for src/skills/distributed-systems.

## Never hardcode model names or pricing tables in resilience prose

Why: The deleted references/llm-resilience.md shipped a PRICING dict with a duplicated "claude-sonnet-5" key and December 2025 per-token prices that rotted silently for months; .claude/rules/skill-authoring.md ("Version and API Claims Must Be Machine-Checkable") exists because of exactly this failure class.
Upstream: https://docs.claude.com/en/docs/about-claude/pricing (current model ids and pricing)

## Do not document integrations OrchestKit does not ship

Why: The deleted examples/orchestkit-workflow-resilience.md (507 lines) described a backend/app analysis pipeline with supervisor agents, bulkhead registries, and Langfuse wiring that has never existed in this repo (the failure class named by the 2026-06 theater-vs-reality audit); removed 2026-07-31.
Upstream: src/skills/CONTRIBUTING-SKILLS.md (house authoring standard; no vendor surface owns this rule)

## Single-source bulkhead tier sizing in rules/resilience-bulkhead.md

Why: The deleted references/bulkhead-pattern.md and the deleted pipeline example carried a second tier table (Standard tier: 3 concurrent, 5 queue) contradicting the surviving rule's table (Standard tier: 8 workers, 12 queue); the two sources of truth had already drifted when found on 2026-07-31. Change tier numbers only in the rule file.
Upstream: https://learn.microsoft.com/azure/architecture/patterns/bulkhead for the pattern itself

## Keep LLM fallback chains quality-first with a semantic-cache floor

Why: House default from the February 2026 v2.0.0 consolidation of four resilience skills into distributed-systems: primary frontier model, then a mini-class fallback, then semantic cache at 0.85 similarity, then a degraded default response (never a bare error). After the thinning this configuration is recorded only here; the file that carried it (references/llm-resilience.md) was upstream restatement around it.
Upstream: https://vercel.com/docs/ai-gateway (provider failover and model routing); provider limits at https://docs.claude.com/en/api/rate-limits