Resilience Patterns
The problem
In a system of many interdependent services, one slow or failing service can cascade — callers pile up waiting on it, exhaust their own resources, and go down too, turning one component's failure into a system-wide outage unless something deliberately stops the cascade.
Why now
Distributed system basics already established that any service can fail or slow down unpredictably; error handling already gave the discipline for responding to a single failure cleanly. Resilience patterns are that same discipline scaled up to prevent one service's failure from propagating through an entire dependency chain.
Mental model
A circuit breaker is exactly what it sounds like: after enough failures, it 'trips' and stops sending requests to a struggling service entirely for a while, failing fast instead of piling up waiting calls — giving the struggling service room to recover instead of being buried under retries. A bulkhead limits how much of a shared resource (threads, connections) one dependency can consume, so one slow dependency can't starve every other request of resources too.
Requires
- Distributed System BasicsDistributed Systems
Resilience patterns exist specifically to survive the partial-failure reality distributed-system-basics establishes as unavoidable; you need that reality already accepted before patterns for containing its consequences make sense.
- Error HandlingProgramming Fundamentals
A circuit breaker or retry policy is a structured, architecture-level generalization of the same failure-handling discipline error-handling already covered at the level of a single function call.
Unlocks
Used in
- Circuit breakers (Hystrix, resilience4j) protecting callers from a struggling downstream service
- Retry with exponential backoff, avoiding hammering an already-struggling service with immediate retries
- Bulkheads, isolating resource pools so one failing dependency can't exhaust resources needed by others
- Timeouts as the most basic resilience pattern — never wait forever for a response
Projects
- Implement a basic circuit breaker wrapping calls to a simulated flaky service, and demonstrate it trips after repeated failures and recovers after a cooldown
- Simulate a cascading failure (one slow dependency exhausting a shared thread pool) and fix it using a bulkhead that isolates that dependency's resource usage
Examples
- A circuit breaker that trips after 5 consecutive failures stops sending new requests for 30 seconds, then allows a single test request through to check if the service has recovered
- Without a bulkhead, a slow third-party API call can exhaust an entire connection pool, starving unrelated requests that don't even depend on that API
Resources
Mastery checklist
- I can implement a basic circuit breaker with trip and recovery behavior
- I can explain what a bulkhead protects against and how
- I can explain why immediate, unlimited retries can make an outage worse instead of better
- I can identify where a cascading failure risk exists in a described system and propose a resilience pattern to address it