Circuit breaker

How it works

State machine. The breaker is a per-upstream in-process state machine with three states:

Closed  --[ failures >= failureThresholdCount, after window fills ]-->  Open
Open    --[ time.Since(openedAt) >= halfOpenAfter, on TryAcquirePermit ]-->  HalfOpen
HalfOpen --[ halfOpenSuccess >= successThresholdCount, after trial fills ]--> Closed
HalfOpen --[ any failure OR trial fills but successCount < threshold ]--> Open (openedAt reset)

Closed state — ring-buffer window. The breaker maintains a circular ring buffer of the last failureThresholdCapacity outcomes (success or failure). The buffer is pre-allocated at construction with capacity max(failureThresholdCapacity, failureThresholdCount, 1). failsafe/breaker.go:L107-125 The open condition is evaluated only once the buffer is full: if failures >= failureThresholdCount, the breaker trips to Open and records openedAt = time.Now(). failsafe/breaker.go:L280-300

Open state. While Open, every call to TryAcquirePermit() returns false immediately — no transport dial occurs, no timeout ticks. The upstream returns ErrFailsafeCircuitBreakerOpen instantly. After halfOpenAfter has elapsed, the next incoming request transitions the breaker to HalfOpen and is allowed through as a probe.

HalfOpen state. Up to max(successThresholdCapacity, successThresholdCount, 1) concurrent trial permits are issued (tracked via halfOpenInflight). When halfOpenSuccess + halfOpenFailure >= successThresholdCapacity (trial window full), the breaker closes if successThresholdCount successes were recorded; otherwise it re-opens and openedAt resets. A single failure in HalfOpen re-opens immediately without waiting for the window to fill. failsafe/breaker.go:L232-237

TryAcquirePermit() per-state logic. failsafe/breaker.go:L139-176

• StateClosed: always returns true immediately (atomic load, no lock needed).
• StateHalfOpen: acquires mu, re-checks state (race guard), computes cap = max(successThresholdCapacity, successThresholdCount, 1), returns halfOpenInflight < cap; if permitted, increments halfOpenInflight.
• StateOpen: acquires mu, re-checks state (race guard with one level of recursive re-evaluation if another goroutine races to HalfOpen first), checks time.Since(openedAt) >= halfOpenAfter; if elapsed → transitions to HalfOpen, sets halfOpenInflight = 1, returns true; otherwise returns false.

Record(outcome) per-state logic. failsafe/breaker.go:L187-247

• OutcomeIgnore: no-op — returns immediately without touching counters.
• StateClosed: calls pushLocked(isFailure), increments lifetime counters, calls checkOpenLocked() which opens if b.count >= failCap && b.failures >= failCount.
• StateHalfOpen: decrements halfOpenInflight, increments trial success/failure counter. Two paths: (a) if halfOpenSuccess + halfOpenFailure >= successCap (trial window full), sufficient successes → resetWindowLocked(); transitionLocked(Closed), insufficient → re-opens; (b) if OutcomeFailure and halfOpenFailure > 0 → immediately re-opens even if the window is not yet full.
• StateOpen: only increments lifetime counters (should not normally be reached because the caller should not have bypassed TryAcquirePermit).

What counts as a failure. The upstreamBreakerOutcome function classifies each (response, error) pair: upstream/upstream_executor.go:L391-421

• OutcomeFailure: ErrCodeEndpointServerSideException (5xx), ErrCodeEndpointTransportFailure, ErrCodeEndpointUnauthorized, ErrCodeEndpointBillingIssue, and a syncing upstream that returns an empty/null response (EvmSyncingStateSyncing && resp.IsResultEmptyish()).
• OutcomeIgnore: request-canceled, request-skipped, rate-limit errors, timeout errors, client-side errors, hedge cancellations — these do not affect the window.
• OutcomeSuccess: all non-error, non-syncing-empty responses.

What bypasses the breaker entirely. Hedge attempts (isHedge == true), internal state-poller probes (req.IsInternal()), and composite batch fan-out sub-requests (req.IsCompositeRequest()) all skip TryAcquirePermit and Record.

Interaction with routing. When the breaker refuses a permit, it returns ErrFailsafeCircuitBreakerOpen. This error is not retryable towards the same upstream (IsRetryableTowardsUpstream = false) but is retryable at the network level — the routing loop tries the next upstream transparently. common/errors.go:L2458-2459

Relationship with selection policy. The circuit breaker and the selection policy are complementary, not redundant. The selection policy excludes upstreams based on rolling health metrics computed across all requests. The circuit breaker acts at the individual-request level: it blocks transport dials when it already knows the upstream is failing, based on its own ring buffer. An upstream can be simultaneously excluded by the selection policy AND have an open circuit breaker; both independently prevent traffic.

Cordoning is a third orthogonal mechanism: an operator-driven sticky flag that keeps an upstream out of rotation regardless of metrics or breaker state.

Cache connector breaker. Cache connectors have their own circuit breaker via FailsafeConnector. The outcome classifier is different: only transport errors count as OutcomeFailure; cache misses (RecordNotFound) and context cancellations are OutcomeIgnore. The error produced uses scope "connector" rather than "upstream".

The isTransportError helper (used by the cache breaker outcome classifier) matches: ErrCodeEndpointTransportFailure; net.Error with Timeout(); io.EOF, io.ErrUnexpectedEOF, syscall.ECONNREFUSED, syscall.ECONNRESET, syscall.EPIPE, syscall.ETIMEDOUT; gRPC codes Unavailable, DeadlineExceeded, Aborted; and string substrings: "connection refused", "connection reset", "broken pipe", "no such host", "network is unreachable", "tls handshake", "i/o timeout", "operation timed out", "use of closed network connection", "client is closed", "unexpectedly closed", "goaway", "clusterdown", "masterdown", "tryagain", "redis is loading".

Window reset. Both Closed→Open tripping and HalfOpen→Closed recovery call resetWindowLocked() — every cycle starts with a fresh ring buffer. failsafe/breaker.go:L302-310

Scope restriction. Setting circuitBreaker inside a network-level failsafe block causes NewNetworkExecutor to return ErrFailsafeConfiguration and eRPC will not start. The circuit breaker is an upstream-only construct. erpc/network_executor.go:L68-73

Config schema

All fields live under upstreams[*].failsafe[*].circuitBreaker. Config struct: common/config.go:L1381-1387. Defaults: common/defaults.go:L2342-2387.

Worked examples

All patterns below are distilled from real production fleets; comments explain the non-obvious choices.

1. Fast-trip breaker for a chain with hot-baseline upstreams (production pattern). On chains where upstreams sit at high baseline load and even 50% error rate for 30 requests signals something is genuinely broken, trip fast and probe quickly. This is the pattern deployed for Cronos upstreams after a 5.77M upstream call spiral caused by a single user burst — cut broken upstreams out of rotation in under a second of traffic, probe after 30s:

failsafe:  - matchMethod: "*"    circuitBreaker:      # trip at 50% error rate: 15 failures in a 30-request window      # window is small so a stuck upstream pegs within ~1s of traffic      failureThresholdCount: 15      failureThresholdCapacity: 30      # 30s: long enough to avoid thrashing, short enough to restore      # the upstream quickly during a transient vendor outage      halfOpenAfter: 30s      # require 60% success in half-open before re-closing (3 of 5)      successThresholdCount: 3      # keep trial window small so re-admission decision is fast      successThresholdCapacity: 5

2. Fan out one circuit breaker config across all method-specific upstream failsafe blocks (TypeScript pattern). When an upstream has per-method failsafe entries (different timeouts per method), each entry needs the circuit breaker attached independently — the upstream-level SetDefaults rule only preserves blocks explicitly listed. Spreading a shared breaker object across all entries is the clean way to ensure coverage without repeating the block manually:

# YAML: repeat the circuitBreaker block on every method-specific entryfailsafe:  - matchMethod: "eth_getLogs|eth_getBlockReceipts"    timeout: { duration: 15s }    circuitBreaker:      failureThresholdCount: 15      failureThresholdCapacity: 30      halfOpenAfter: 30s      successThresholdCount: 3      successThresholdCapacity: 5  - matchMethod: "*"    circuitBreaker:      failureThresholdCount: 15      failureThresholdCapacity: 30      halfOpenAfter: 30s      successThresholdCount: 3      successThresholdCapacity: 5

3. Conservative production config — tolerates transient blips. For stable chains where upstreams have rare failures, a larger window prevents false trips while still protecting against hard outages. Use this as the starting point for any chain that does not show the hot-baseline pattern:

failsafe:  - matchMethod: "*"    circuitBreaker:      # window must fill to 80 before the trip condition is evaluated at all      failureThresholdCount: 20      failureThresholdCapacity: 80      # 5 min: upstream needs time to restart cleanly      halfOpenAfter: 5m      # 8 successes out of 200 trial slots — loose, but default of 200 is      # intentional (the successThresholdCapacity=10 block in SetDefaults is      # unreachable dead code; effective default is 200)      successThresholdCount: 8      successThresholdCapacity: 200

4. Method-specific breaker for heavy archive queries. eth_getLogs and eth_getBlockReceipts fail sooner on overloaded nodes than lightweight calls. A tighter breaker on these methods cuts them off quickly without affecting eth_call or block getters, which may still be healthy on the same node:

failsafe:  - matchMethod: "eth_getLogs|eth_getBlockReceipts"    circuitBreaker:      # small window: heavy queries fail fast when the node is overloaded      failureThresholdCount: 5      failureThresholdCapacity: 20      halfOpenAfter: 2m      successThresholdCount: 3      successThresholdCapacity: 10  # catch-all: conservative for lightweight methods that are usually healthy  - matchMethod: "*"    circuitBreaker:      failureThresholdCount: 20      failureThresholdCapacity: 80      halfOpenAfter: 5m      successThresholdCount: 8      successThresholdCapacity: 200

5. Tight HalfOpen trial — recover quickly or re-open fast. When you have many upstream options and want to minimize time spent probing a still-sick upstream, shrink the trial window and require a high success ratio. This pairs well with the fast-trip breaker above:

failsafe:  - matchMethod: "*"    circuitBreaker:      failureThresholdCount: 10      failureThresholdCapacity: 40      halfOpenAfter: 1m      # 5 of 6 = 83% success required: re-open immediately if not healthy      successThresholdCount: 5      successThresholdCapacity: 6

Request/response behavior

• While Open, TryAcquirePermit() returns false immediately — the upstream returns ErrFailsafeCircuitBreakerOpen with scope "upstream" without dialing the transport. common/errors.go:L1665-1692
• The error code ErrCodeFailsafeCircuitBreakerOpen is non-retryable towards the same upstream (IsRetryableTowardsUpstream = false) but IS retryable at the network level (IsRetryableTowardNetwork = true — the default, since the error does not set the retryableTowardNetwork=false detail field). The network routing loop therefore tries the next upstream; if all upstreams are open the aggregated error includes the count "N upstream circuit breaker open". common/errors.go:L1054-1055
• erpc_upstream_attempt_outcome_total records outcome="breaker_open" for each refused permit. The client sees the upstream failure only if ALL upstreams are exhausted.
• The CircuitBreakerState() accessor on Upstream surfaces only the catch-all (matchMethod: "*", no finality filter) breaker for the admin UI and simulator panel. upstream/upstream.go:L320-333

Best practices

• Let the window fill before worrying. The breaker cannot trip until failureThresholdCapacity outcomes are recorded. With the default capacity of 80, a freshly restarted upstream takes at least 80 requests before any trip can occur. Reduce capacity if you need faster detection.
• Set halfOpenAfter longer than the upstream's actual restart time. If the cooldown is shorter than the upstream's recovery, the HalfOpen probe will fail immediately, re-open the breaker, and start another 5-minute wait — flapping. Monitor erpc_upstream_breaker_state_change_total{transition="half_open_to_open"} to detect this.
• Use method-specific breakers for heavy queries. Archive calls like eth_getLogs fail sooner on overloaded nodes than lightweight calls. Separate breakers prevent a log-heavy spike from taking your whole upstream out of rotation for everything else.
• The circuit breaker does not replace the selection policy. The selection policy removes upstreams from the ordered list proactively (before they fail); the circuit breaker blocks in-progress requests reactively. Use both: selection policy removes degraded upstreams before they accumulate failures; circuit breaker handles sudden hard failures.
• Do not put circuitBreaker at network scope. It will cause a startup error. The breaker is upstream-scoped by design — tripping it at the network level would block all upstreams simultaneously.
• Hedge attempts are invisible to the breaker. A hedge that gets a 5xx does not increment the failure window. Only primary (non-hedge) attempts count. This is intentional to avoid penalizing upstreams for aggressive hedging behavior.
• Watch the successThresholdCapacity default (200, not 10). Due to a known dead-code bug in SetDefaults, the effective default is 200 — the HalfOpen trial window accepts up to 200 concurrent probes before evaluating. If you need faster re-close decisions, set it explicitly.

Edge cases & gotchas

1. Window must be full before tripping. failureThresholdCapacity: 80, failureThresholdCount: 1 will NOT trip until 80 outcomes are in the buffer — even if the first 79 were failures. checkOpenLocked short-circuits with if b.count < failCap { return }. failsafe/breaker.go:L292-294
2. successThresholdCapacity effective default is 200, not 10 — the second defaults block is unreachable dead code. SetDefaults at common/defaults.go:L2357-L2384 (opens in a new tab) contains two if c.SuccessThresholdCapacity == 0 guards. The first (L2357-L2363) sets 200; the second (L2378-L2384, which would set 10) is never reached. Effective default is 200.
3. Single HalfOpen failure re-opens immediately. Even before the trial window is full, any OutcomeFailure in HalfOpen re-opens and resets openedAt. Ensure halfOpenAfter is long enough for the upstream to actually recover or flapping will occur. failsafe/breaker.go:L232-237
4. Hedge attempts are invisible to the breaker. A hedge attempt that gets a 500 from the upstream does not count as a failure. Only the primary (non-hedge) attempt counts. This is intentional to avoid penalizing upstreams for aggressive hedging.
5. Internal probe calls bypass the breaker in both directions. State-poller probes (req.IsInternal()) neither block on an open breaker nor record outcomes. An Open upstream will receive internal probes but no user traffic.
6. circuitBreaker at network scope causes startup failure. Even circuitBreaker: {} in a network failsafe block causes NewNetworkExecutor to return an error and eRPC will not start. erpc/network_executor.go:L68-73
7. Per-method breakers are independent. CircuitBreakerState() on Upstream only exposes the matchMethod: "*" catch-all breaker. An eth_getLogs breaker tripping does not affect the * catch-all breaker.
8. EVM syncing state triggers a failure. A syncing upstream that returns an empty/null response is classified as OutcomeFailure — a syncing node can trip the breaker even without an error code. upstream/upstream_executor.go:L412-419
9. Window reset on every open/close. Both tripping Closed→Open and closing HalfOpen→Closed call resetWindowLocked() — a flapping upstream starts each cycle with a clean slate.
10. Cordon and circuit breaker are orthogonal. A cordoned upstream also has its circuit breaker checked (if configured). An uncordon does not reset the breaker.
11. OnTransition fires in a goroutine. The metric increment is async. Under extremely high test parallelism this can appear as a missing metric increment immediately after a transition. failsafe/breaker.go:L312-334
12. Cache connector breakers do not emit the state-change metric. The OnTransition hook is not wired for cache executors — erpc_upstream_breaker_state_change_total only fires for upstream breakers.
13. Race safety in Open→HalfOpen. When two goroutines race to call TryAcquirePermit while the breaker is Open and halfOpenAfter has just elapsed, the second goroutine re-checks state after acquiring mu and, seeing HalfOpen, recursively re-evaluates — bounded to one extra call. No double-transition is possible. failsafe/breaker.go:L160-166
14. halfOpenAfter is required at validation, but SetDefaults fills it. Validate() errors with "failsafe.circuitBreaker.halfOpenAfter is required" if the field is zero. In practice eRPC's config loading calls SetDefaults (which fills 5m) before Validate, so omitting the field from YAML is safe. If you call Validate before SetDefaults in custom tooling, the zero value will be rejected. common/validation.go:L1038-L1040 (opens in a new tab)

Observability

Programmatic access. The breaker exposes a Metrics() method that returns (failures, successes, executions uint64) as lifetime atomic counters. failsafe/breaker.go:L345-350 These are not surfaced as Prometheus gauges — use the counters above for dashboards.

Log messages. On every state transition, transitionLocked emits a WARN-level log:

{"level":"warn","from":"closed","to":"open","reason":"failure_threshold","executions":N,"successes":N,"failures":N,"message":"circuit breaker state changed"}

Reason strings: "failure_threshold" (Closed→Open), "half_open_delay_elapsed" (Open→HalfOpen), "half_open_success_threshold" (HalfOpen→Closed), "half_open_failure" (HalfOpen→Open).

Source code entry points

• failsafe/breaker.go (opens in a new tab) — self-contained circuit breaker state machine: ring buffer, TryAcquirePermit, Record, transitionLocked, OnTransition hook, State/Metrics accessors
• upstream/upstream_executor.go:L368-L421 (opens in a new tab) — upstreamBreakerEligible and upstreamBreakerOutcome classifiers; callBreakerWithTimeout wrapper
• upstream/upstream.go:L87-L99 (opens in a new tab) — makeBreakerTransitionHook wires OnTransition to the Prometheus counter; classifyUpstreamOutcome maps ErrCodeFailsafeCircuitBreakerOpen to UpstreamOutcomeBreakerOpen
• data/cache_executor.go:L220-L234 (opens in a new tab) — cacheBreakerOutcome classifier for cache connector breakers
• data/failsafe.go:L21-L133 (opens in a new tab) — FailsafeConnector wrapper for cache connectors; isTransportError helper
• common/config.go:L1381-L1396 (opens in a new tab) — CircuitBreakerPolicyConfig struct and Copy()
• common/defaults.go:L2342-L2387 (opens in a new tab) — CircuitBreakerPolicyConfig.SetDefaults() including the double-assignment dead-code bug at L2357-L2384
• common/validation.go:L1037-L1059 (opens in a new tab) — CircuitBreakerPolicyConfig.Validate() with all field constraints
• common/errors.go:L1665-L1703 (opens in a new tab) — ErrFailsafeCircuitBreakerOpen type, constructor, and DeepestMessage
• common/errors.go:L2455-L2459 (opens in a new tab) — IsRetryableTowardsUpstream: ErrCodeFailsafeCircuitBreakerOpen → not upstream-retryable
• telemetry/metrics.go:L482-L489 (opens in a new tab) — MetricUpstreamBreakerStateChange counter definition
• erpc/network_executor.go:L68-L73 (opens in a new tab) — enforces network-scope restriction
• erpc/networks_skip_test.go:L174-L262 (opens in a new tab) — integration test confirming breaker-open causes rotation to next upstream without dialing the failing one
• erpc/networks_test.go:L6088-L6521 (opens in a new tab) — integration tests for threshold tripping, half-open recovery, and full open/close cycle

Related pages

• Retry — retry attempts are not blocked by an open breaker; the routing loop tries the next upstream instead.
• Timeout — bounds individual request attempts from outside the breaker.
• Hedge — hedge attempts bypass the breaker and do not record outcomes.
• Selection policies — proactively removes degraded upstreams before they accumulate breaker failures.
• Survive provider outages — the end-to-end outcome this feature serves.