Implementing Circuit Breakers in Production

When you're building systems that depend on external services, failures are inevitable. APIs go down, networks partition, and services become slow. Circuit breakers are one of the most effective patterns for handling these scenarios gracefully.

The Problem

Working with multiple LLM providers (OpenAI, Anthropic, Google) presents several challenges. Each provider has different failure modes, rate limits, and latency characteristics. Without proper handling, a slow or failing provider can bring down an entire pipeline.

The naive approach is to add timeouts and retries. But this creates new problems:

• Retries during an outage amplify the problem (retry storms)
• Long timeouts block resources and cascade failures
• Users wait for failures we already know will happen

What is a Circuit Breaker?

The circuit breaker pattern, popularized by Michael Nygard in "Release It!", works like an electrical circuit breaker. When failures exceed a threshold, the circuit "opens" and immediately fails requests without attempting them.

Three states:

1. Closed - Normal operation. Requests pass through.
2. Open - Failure threshold exceeded. Requests fail immediately.
3. Half-Open - Testing if the service recovered. Let some requests through.

Implementation

Here's a simplified version of a production-ready circuit breaker:

from datetime import datetime, timedelta
from enum import Enum
from threading import Lock

class CircuitState(Enum):
    CLOSED = "closed"
    OPEN = "open"
    HALF_OPEN = "half_open"

class CircuitBreaker:
    def __init__(
        self,
        failure_threshold: int = 5,
        recovery_timeout: timedelta = timedelta(seconds=30),
        half_open_max_calls: int = 3
    ):
        self.failure_threshold = failure_threshold
        self.recovery_timeout = recovery_timeout
        self.half_open_max_calls = half_open_max_calls
        
        self.state = CircuitState.CLOSED
        self.failure_count = 0
        self.last_failure_time = None
        self.half_open_calls = 0
        self.lock = Lock()
    
    def can_execute(self) -> bool:
        with self.lock:
            if self.state == CircuitState.CLOSED:
                return True
            
            if self.state == CircuitState.OPEN:
                if datetime.now() - self.last_failure_time > self.recovery_timeout:
                    self.state = CircuitState.HALF_OPEN
                    self.half_open_calls = 0
                    return True
                return False
            
            # Half-open: allow limited calls
            if self.half_open_calls < self.half_open_max_calls:
                self.half_open_calls += 1
                return True
            return False
    
    def record_success(self):
        with self.lock:
            if self.state == CircuitState.HALF_OPEN:
                self.state = CircuitState.CLOSED
            self.failure_count = 0
    
    def record_failure(self):
        with self.lock:
            self.failure_count += 1
            self.last_failure_time = datetime.now()
            
            if self.failure_count >= self.failure_threshold:
                self.state = CircuitState.OPEN
            
            if self.state == CircuitState.HALF_OPEN:
                self.state = CircuitState.OPEN

Key Decisions

Failure threshold: We use 5 failures in a sliding window. Too low and you'll open on transient errors. Too high and you'll let too many requests fail.

Recovery timeout: 30 seconds works for most APIs. For rate-limited services, we use longer timeouts based on the rate limit reset time.

What counts as failure: Not all errors should trip the breaker. We only count:

• Connection timeouts
• 5xx errors
• Rate limit responses

Client errors (4xx) are the caller's problem, not a service failure.

Fallback Strategies

An open circuit isn't useful if you just return an error. We use several fallback approaches:

1. Provider failover: If OpenAI's circuit opens, route to Anthropic
2. Cached responses: Return stale data if appropriate
3. Graceful degradation: Offer reduced functionality
4. Queue for retry: Persist the request and process later

Monitoring

Circuit state changes are important signals. We track:

• Circuit open events (alerts)
• Time spent in each state
• Fallback success rates
• False positive rate (opened when service was healthy)

Lessons Learned

Start conservative: It's easier to relax thresholds than to explain cascading failures.

Test the open state: Your fallback logic needs as much testing as the happy path.

Different breakers for different operations: Read and write operations to the same service might need different thresholds.

Document the behavior: Teams need to know what happens when a circuit opens.

Circuit breakers are one piece of a resilience strategy. Combined with timeouts, bulkheads, and proper retry policies, they help build systems that degrade gracefully instead of failing catastrophically.

Sources and further reading

The primary material behind the pattern, and what to read once the basics land.

• Circuit breaker by Martin Fowler. The canonical description of closed, open and half-open, and what each state is for. The idea comes from Michael Nygard's Release It!, which is worth reading whole.
• Addressing cascading failures from the Google SRE book. What actually happens when a dependency degrades and every caller retries into it. This is the failure a breaker exists to stop.
• Handling overload, same book, for load shedding and graceful degradation, which are the tools a breaker hands off to once it opens.
• Timeouts, retries and backoff with jitter in the Amazon Builders' Library. A breaker without timeouts underneath it never trips, because nothing ever fails fast enough to count.
• Exponential backoff and jitter from the AWS Architecture Blog, for what the half-open probe should look like when many clients recover at once.

Continue learning

If the three states now make sense, the next thing to understand is timeouts, because a breaker can only count failures that the caller actually gives up on. After that, bulkheads, which stop one failing dependency from consuming the thread pool everything else needs. Then load shedding, which is what you do when the problem is your own service rather than a dependency.

FAQ

Where should breaker state live? Wherever the resource is shared. Per-process state means every worker discovers the outage independently, which on a large fleet defeats most of the point. Shared state in a database or cache is the usual answer.

What failure rate should open the breaker? Start from what a healthy dependency looks like, not from a round number. If it normally errors on 1% of calls, a 50% threshold over a meaningful window is a real signal, while 5% is noise that will trip during a routine blip.

Should a breaker cover all errors? No. Count the ones that mean the dependency is unhealthy: timeouts, connection failures, 5xx. A 400 means your request was wrong and will fail identically on every retry, and a 429 is a rate limit that backoff handles better.