How eRPC works

How it works

eRPC routes each JSON-RPC call through a deterministic, layered pipeline. Understanding the pipeline order matters for configuration — earlier stages short-circuit later ones.

1. HTTP ingress (erpc/http_server.go)

The handler parses the URL path for {projectId}/{architecture}/{chainId}, applies aliasing rules if configured, reads and decompresses the body (gzip supported), and dispatches single or batch JSON-RPC payloads. Batch items fan out as concurrent goroutines; results reassemble before the response is written. The real client IP is resolved from trusted forwarder headers before any auth or rate-limit checks.

2. RequestProcessor (erpc/request_processor.go:35–67)

ProcessUnary is the entry point for both HTTP and gRPC. It resolves the project, wraps raw JSON bytes in NormalizedRequest, calls Validate() (missing method → immediate error), authenticates the consumer via project.AuthenticateConsumer, resolves networkID = "{architecture}:{chainId}", applies directive defaults from network.Config().DirectiveDefaults, then delegates to project.Forward.

3. Project layer (erpc/projects.go)

PreparedProject.Forward acquires a project-level rate-limit permit (no-op when rateLimitBudget is unset), fires HandleProjectPreForward — the first EVM hook, which handles eth_blockNumber (early cache return), eth_call (static call optimisation), eth_chainId (served from config), and proactive eth_getLogs / trace_filter range splitting. If the hook handles the request, HandleNetworkPostForward is called on the result before returning. Otherwise the call proceeds to network.Forward.

4. Network layer (erpc/networks.go:931–1603)

Network.Forward is the core dispatch function, in execution order:

1. Static responses — matched against network.cfg.StaticResponses before any upstream is contacted.
2. Multiplexing — identical in-flight requests share one leader goroutine; followers wait and receive a copied response with their original request ID patched in.
3. Cache read — cacheDal.Get fans out across all matching policies; first hit short-circuits the rest of the pipeline.
4. Upstream ordering — policyEngine.GetOrdered returns upstreams ranked by selection policy scores; falls back to registry order on cold start.
5. EVM network pre-forward hook (HandleNetworkPreForward) — upstream-aware short-circuits for eth_getLogs range enforcement, eth_chainId, trace_filter.
6. Future-block short-circuit — requests for a concrete block beyond every eligible upstream head return null immediately without touching any upstream.
7. Method guard — eth_accounts and eth_sign are hardcoded unsupported; stateful methods require exactly one targeted upstream.
8. Network rate limiting — network-level budget checked.
9. Request preparation — evm.NormalizeHttpJsonRpc normalizes params (block tag → concrete hex interpolation, EVM-specific transformations), caches block number for downstream availability checks.
10. Failsafe executor selection — getFailsafeExecutors iterates failsafe[] in config order, returning the first whose matchMethod (wildcard) and matchFinality match.

5. Failsafe executor chain (erpc/network_executor.go:69–200)

The executor nests policies in a fixed order:

timeout (wraps entire invocation)
  ↳ if consensus enabled && !SkipConsensus:
      consensus(
        retry(
          hedge(tryOneUpstream)   ← one upstream per consensus slot
        )
      )
    else:
      retry(
        hedge(runUpstreamSweep)   ← all upstreams per sweep
      )

• Timeout wraps the context with context.WithTimeoutCause; the cause is ErrDynamicTimeoutExceeded. Can be a static duration or an adaptive quantile function over observed per-method latency.
• Consensus is delegated to the consensusRunner interface. The executor does not import the consensus package directly.
• Retry (runRetry) iterates up to maxAttempts. For data-unavailable retries (block_unavailable, empty_result, missing_data) it uses an EMA block-time-relative delay (falling back to emptyResultDelay). For genuine errors it uses exponential backoff via failsafe.ComputeBackoff. firstInformativeErr is tracked so a bare ErrNoUpstreamsLeftToSelect on a later attempt does not mask the root cause.
• Hedge (runHedge) fires up to maxCount parallel copies after an adaptive delay. Write methods are never hedged. A hedge leg returning an emptyish result does not win the race — siblings continue until a non-null response arrives.

6. Upstream sweep loop (erpc/networks.go:1226–1407)

For each iteration inside sweepFn:

1. Calls req.NextUpstream() — atomic round-robin with skip logic for consumed and permanently errored upstreams; safe for concurrent hedge goroutines.
2. Block availability gating — compares the request's block number against the upstream's EvmEffectiveLatestBlock. Retryable if within maxRetryableBlockDistance (default 128) of the upstream head; fail-open on poller errors.
3. Calls HandleUpstreamPreForward then u.Forward.
4. HandleUpstreamPostForward validates the response (integrity checks, mark-empty-as-error methods, eth_sendRawTransaction idempotency).
5. normalizeResponse rewrites the JSON-RPC id to match the client's original byte-for-byte (preserving large 64-bit integers via IDRawBytes()).
6. MarkUpstreamCompleted releases retryable-errored upstreams back into rotation for the next failsafe retry round. Deterministic client errors return immediately.

7. Response write and async cache

Back in http_server.go, setResponseHeaders emits the full X-ERPC-* diagnostic surface (attempts, retries, hedges, cache status, winning upstream, duration). The JSON-RPC response body is streamed. An async goroutine fires cacheDal.Set with a 10-second deadline under appCtx — a client disconnect never aborts the cache write. Panics in the write goroutine are recovered and reported via erpc_unexpected_panic_total{scope="cache-set"}.

gRPC query-stream path

ProcessQueryStream handles eth_queryBlocks, eth_queryTransactions, eth_queryLogs, eth_queryTraces, and eth_queryTransfers. Auth and rate limiting are identical to the unary path. EvmQueryExecutor.Execute attempts native pipe-through to an upstream gRPC BDS client; if no upstream supports structured queries, it shims via sequential JSON-RPC subrequests (eth_getBlockByNumber, eth_getLogs, trace_block, etc.) with block-boundary- aware pagination. Composite subrequests set IsCompositeRequest=true to skip network-scope retry and hedge, preventing exponential amplification.

Config schema

Config fields governing the lifecycle. Network-level failsafe and directive defaults are the primary control surface.

EVM network config — all under networks[].evm. in YAML; defaults set by EvmNetworkConfig.SetDefaults() (common/defaults.go):

Worked examples

1. Baseline resilient config — timeout + retry + hedge for any method. Good starting point for a production network with multiple upstreams. The timeout bounds the whole race; retry recovers transient upstream errors; hedge cuts tail latency on slow providers:

failsafe:  - matchMethod: "*"    timeout:      duration: 10s    retry:      maxAttempts: 3      backoffMaxDelay: 1s    hedge:      delay:        quantile: 0.7        min: 100ms        max: 2s      maxCount: 1

2. Slow-data retry for pending transactions. eth_getTransactionReceipt and eth_getTransactionByHash return null until the transaction is mined. Combine emptyResultMaxAttempts with the directive default retryPending: true to poll until confirmed, using block-time-relative delays:

directiveDefaults:  retryPending: truefailsafe:  - matchMethod: "eth_getTransactionReceipt|eth_getTransactionByHash"    retry:      maxAttempts: 10      emptyResultMaxAttempts: 10      emptyResultDelay: 500ms

3. Consensus for critical state reads. Use consensus when multiple upstreams must agree on a result before it is returned. skipConsensus directive lets callers opt out per-request:

failsafe:  - matchMethod: "eth_getBalance|eth_call"    consensus:      minParticipants: 2      method: "majority"

4. Multiplexing for high-traffic hot paths. When many clients request the same block simultaneously, multiplexing deduplicates the upstream request — one upstream call serves all waiting followers with a response ID patch per follower:

multiplexing:  enabled: true

Request/response behavior

• Directive precedence: HTTP headers → query params (query params override headers for use-upstream, retry-empty, retry-pending, skip-cache-read, skip-interpolation, skip-consensus). Config directiveDefaults are applied only when directives are not already set — HTTP-set directives are never overwritten.

  common/request.go:L563-565

• Response headers emitted by setResponseHeaders (erpc/http_server.go:1105) under server.executionHeaders:

  Header	When present	Value
  X-ERPC-Version	always	eRPC version string
  X-ERPC-Commit	always	git commit SHA
  X-ERPC-Attempts	always	total physical ops (upstream + cache)
  X-ERPC-Upstream-Attempts	always	upstream-scope attempt count
  X-ERPC-Upstream-Retries	always	upstream-scope retry count
  X-ERPC-Upstream-Hedges	always	upstream-scope hedge count
  X-ERPC-Network-Attempts	always	network-scope rotation count
  X-ERPC-Network-Retries	always	network-scope retry count
  X-ERPC-Network-Hedges	always	network-scope hedge count
  X-ERPC-Cache-Attempts	only if cache exercised	cache-scope attempt count
  X-ERPC-Consensus-Slots	only if consensus used	number of consensus slots
  X-ERPC-Consensus-Disputes	only if disputes	number of disputes
  X-ERPC-Cache	if response available	HIT or MISS
  X-ERPC-Upstream	if upstream response	upstream ID that served it
  X-ERPC-Duration	if response	total request duration in milliseconds
  X-ERPC-Upstreams	executionHeaders=all	per-attempt trace: id=role:outcome:durationMs:won|lost

• Request directives — settable via HTTP header or query param (query overrides header for the non-validation directives). Header-only validation directives (X-ERPC-Validate-*, X-ERPC-Enforce-*) are not settable via query params. common/request.go:L38-114

  Header	Query param	Behavior
  X-ERPC-Retry-Empty	retry-empty	Retry emptyish upstream responses
  X-ERPC-Retry-Pending	retry-pending	Retry pending tx lookups
  X-ERPC-Skip-Cache-Read	skip-cache-read	"true" = skip all; connector-id pattern = skip matching
  X-ERPC-Use-Upstream	use-upstream	Pin request to matching upstream(s) by id, glob, or tag
  X-ERPC-Skip-Interpolation	skip-interpolation	Don't replace block tags with hex in outbound params
  X-ERPC-Skip-Consensus	skip-consensus	Bypass consensus; retry+hedge still apply
  X-ERPC-Enforce-Highest-Block	enforce-highest-block	Block integrity validation
  X-ERPC-Enforce-GetLogs-Range	enforce-getlogs-range	Enforce eth_getLogs block range limit
  X-ERPC-Enforce-Non-Null-Tagged-Blocks	enforce-non-null-tagged-blocks	Reject null on tagged block lookups
  X-ERPC-Enforce-Log-Index-Strict-Increments	enforce-log-index-strict-increments	Validate log index ordering
  X-ERPC-Validate-Logs-Bloom-Emptiness	validate-logs-bloom-emptiness	Bloom↔logs consistency check
  X-ERPC-Validate-Logs-Bloom-Match	validate-logs-bloom-match	Recalculate bloom from logs
  X-ERPC-Validate-Tx-Hash-Uniqueness	validate-tx-hash-uniqueness	No duplicate tx hashes in block
  X-ERPC-Validate-Transaction-Index	validate-transaction-index	Validate tx positions
  X-ERPC-Receipts-Count-Exact	receipts-count-exact	Expected exact receipt count
  X-ERPC-Receipts-Count-At-Least	receipts-count-at-least	Minimum receipt count
  X-ERPC-Validation-Expected-Block-Hash	validation-expected-block-hash	Expected block hash ground truth
  X-ERPC-Validation-Expected-Block-Number	validation-expected-block-number	Expected block number ground truth
  X-ERPC-Validate-Transactions-Root	validate-transactions-root	Check transactionsRoot consistency
  X-ERPC-Validate-Header-Field-Lengths	validate-header-field-lengths	Validate EVM header field byte lengths
  X-ERPC-Validate-Transaction-Fields	validate-transaction-fields	Validate transaction fields
  X-ERPC-Validate-Transaction-Block-Info	validate-transaction-block-info	Validate tx block info
  X-ERPC-Validate-Log-Fields	validate-log-fields	Validate log fields

• Retry reasons — shouldRetryWithReason returns one of: execution_exception_retryable (EVM execution exception with retryableTowardNetwork=true), block_unavailable (ErrCodeUpstreamBlockUnavailable, subject to emptyResultMaxAttempts cap), missing_data (ErrCodeEndpointMissingData unless RetryEmpty=false), retryable_error (generic IsRetryableTowardNetwork), empty_result (emptyish response with RetryEmpty=true), pending_tx (tx-lookup methods with RetryPending=true). Composite requests are never retried at network scope. erpc/network_executor.go:L374-463

• Batch requests fan out to per-item goroutines; each item is processed independently through the full pipeline and reassembled before writing.

• Composite subrequests (IsCompositeRequest=true) skip network-scope retry and hedge to prevent exponential amplification of range-split sub-requests.

  erpc/network_executor.go:L378-380

• Response ID fidelity: normalizeResponse uses IDRawBytes() for byte-perfect round-trip of the JSON-RPC id field, preserving large 64-bit integer IDs that cannot be represented as float64. erpc/networks.go:L2234-2266

Best practices

• Set a timeout for every failsafe block in production; without it, a hung upstream holds the goroutine indefinitely and can exhaust the connection pool under load.
• Use adaptive hedge delay (quantile: 0.7, min: 100ms, max: 2s) over static delays — static delays go stale as provider latencies shift; quantile mode self-adjusts per method.
• Enable multiplexing for networks that see bursts of identical calls (e.g. eth_blockNumber, eth_getBlockByNumber?latest); it eliminates duplicate upstream cost at zero config complexity.
• Set emptyResultDelay as a bootstrap fallback for data-unavailable retries — the EMA block-time delay takes a few requests to warm up; without a fallback the first retries fire immediately.
• Do not set directiveDefaults.retryEmpty: true globally unless you also set emptyResultMaxAttempts; without a cap, methods that legitimately return null (e.g. unindexed blocks on archive nodes) will exhaust maxAttempts retries on every call.
• Prefer staticResponses for methods your upstreams do not support and will never support (e.g. a custom method returning a fixed value); this avoids wasting upstream budget on guaranteed errors.
• Internal requests (state pollers, chainId probes) set IsInternal=true, bypassing retry, hedge, and circuit breaker — only per-attempt timeout applies. Never forward internal requests to a public consumer-facing failsafe path.

Edge cases & gotchas

1. ApplyDirectiveDefaults is called twice — once in RequestProcessor and again defensively in Network.Forward for gRPC callers. The guard ensures only the first call has effect. common/request.go:L563-565
2. Multiplexer follower race on close: a follower arriving during leader cleanup retries LoadOrStore and becomes the new leader or joins a new follower group. erpc/networks.go:L2017-2025
3. ErrNoUpstreamsLeftToSelect degeneration on retries: after the first retry round, all upstreams may be marked consumed, hiding the original error. firstInformativeErr is tracked and surfaced in the final ErrFailsafeRetryExceeded wrapping. erpc/network_executor.go:L265-284
4. Hedge emptyish rejection: a fast null from a lagging upstream does not win the hedge race — in-flight siblings continue until a non-null result arrives. Methods in emptyResultAccept (e.g. eth_call) are exempt. erpc/network_executor.go:L607-624
5. SkipConsensus falls through to retry+hedge: when skipConsensus=true, the executor skips consensus entirely but all other failsafe policies still apply. erpc/network_executor.go:L178-191
6. Cache write panic recovery: panics in the async write goroutine are recovered and reported via erpc_unexpected_panic_total{scope="cache-set"}. erpc/networks.go:L1496-1508
7. Stateful method enforcement: methods marked Stateful require exactly one targeted upstream. Without a UseUpstream selector, multiple upstreams return ErrNotImplemented. eth_accounts and eth_sign are always unsupported. erpc/networks.go:L2112-2143
8. Block availability check is fail-open: if EvmAssertBlockAvailability errors (poller issues, partial state), the request proceeds to the upstream rather than being gated. erpc/networks.go:L1949-1959
9. eth_sendRawTransaction is never wrapped in ErrFailsafeRetryExceeded: execution- reverted responses are surfaced directly — the revert IS the answer, not a retry condition. erpc/network_executor.go:L278-283
10. Async cache write does not block the response: the write goroutine uses appCtx, not the request context, so a client disconnect never aborts the write. The 10-second deadline is on the write itself, not on waiting for it. erpc/networks.go:L1494-1518
11. Query shim retry safety: a gRPC query stream error is only retried on the next upstream if no page has been emitted (StreamError.PageEmitted gate). Partial-result retries are forbidden. erpc/query_executor.go:L247-253
12. TranslateToJsonRpcException dominant-code selection: when all upstreams return skipped or unsupported errors, the wire JSON-RPC code depends on the first child's error type — either -32601 (method not found) or -32603 (server error). Clients cannot reliably distinguish "globally unsupported" from "internal error" by code alone; the data field carries the full eRPC error chain for programmatic detection.
13. trace_block → debug_traceBlockByNumber fallback in shim: shimQueryTraces first tries trace_block; if the upstream returns ErrCodeEndpointUnsupported, it retries with debug_traceBlockByNumber?callTracer. If that is also unsupported, returns gRPC Unimplemented. erpc/query_shim.go:L388-420
14. paginateLogsByBlock never splits a block across pages: when adding a block's logs would exceed the page limit and the page already has items, the function stops before that block (producing a cursor). If the first block alone exceeds the limit, all its logs are included — the limit is a soft cap at block boundaries, not an absolute log count. erpc/query_shim.go:L617-649
15. Hash-based projection preserves block hash: ProjectBlockFields never zeros the Hash field even when sel.Hash=false, because the hash is required for cursor semantics in paginated query responses. erpc/query_field_projection.go:L12-15
16. MarkEmptyAsErrorMethods only fires when RetryEmpty directive is true: even if a method is in the list, conversion to ErrEndpointMissingData is gated on req.Directives().RetryEmpty. Operators must set retryEmpty: true (directive default or per-request header) for this feature to activate. architecture/evm/common.go:L23-25
17. Custom markEmptyAsErrorMethods completely replaces the default 11-method set — there is no merge. Setting markEmptyAsErrorMethods: ["custom_method"] disables all default methods. An explicitly empty markEmptyAsErrorMethods: [] disables the feature entirely. common/defaults.go:L2110-2112
18. 200-OK EVM revert detection uses dt[1:11]: in ExtractJsonRpcError, the revert-payload check is dt[1:11] == "0x08c379a0" because jr.GetResultString() returns the JSON-encoded value — dt[0] is the JSON double-quote character, so the actual hex string starts at index 1. A check starting at index 0 always fails on valid JSON strings. architecture/evm/error_normalizer.go:L609-624
19. FullySyncedThreshold = 4: an upstream must return eth_syncing: false four consecutive times before eRPC stops sending syncing-check probes. One syncing=true resets the counter. architecture/evm/evm_state_poller.go:L21

Observability

Key OpenTelemetry spans:

• Network.Forward — top-level; attributes: cache.hit, multiplexed, failsafe.matched_method
• PolicyEngine.GetOrdered — upstream ordering time
• Network.forwardAttempt — per-attempt; attributes: execution.attempt, execution.retry, execution.hedge
• Network.UpstreamLoop — per-upstream iteration; attributes: upstream.id, upstream.latest_block, skipped, skip_reason
• Network.TryForward — call to doForward per upstream
• Network.NormalizeResponse — response ID rewrite
• Network.EnrichStatePoller — state poller suggestion after block response
• Upstream.PreForwardHook / Upstream.PostForwardHook — EVM method hooks per upstream
• Project.Forward → Project.PreForwardHook → Network.PostForwardHook
• QueryStream.Handle — gRPC query stream top-level span
• Query.Execute, Query.ResolveQueryBounds, Query.ShimBlocks/Logs/Transactions/Traces/Transfers, Query.ForwardSubrequest
• Cache.Get / Cache.Set — cache fan-out spans with network.id / upstream.id attributes
• Evm.ExtractBlockReferenceFromRequest / Evm.ExtractBlockReferenceFromResponse — block-ref resolution detail spans

Source code entry points

• erpc/request_processor.go (opens in a new tab) — RequestProcessor.ProcessUnary / ProcessQueryStream: project resolution, auth, network selection, unified entry point for HTTP and gRPC.
• erpc/networks.go (opens in a new tab) — Network.Forward: multiplexing, cache read/write, upstream ordering, block-availability gating, all EVM hooks, normalizeResponse, enrichStatePoller.
• erpc/network_executor.go (opens in a new tab) — networkExecutor.Run: failsafe policy composition (timeout → consensus → retry → hedge); retry reason classification and backoff computation.
• erpc/projects.go (opens in a new tab) — PreparedProject.Forward: project-level rate limiting, metrics, shadow upstreams, EVM hook wrapper.
• erpc/http_server.go (opens in a new tab) — HTTP ingress, batch dispatch, directive enrichment, setResponseHeaders.
• architecture/evm/hooks.go (opens in a new tab) — Four EVM lifecycle hooks: HandleProjectPreForward, HandleNetworkPreForward, HandleNetworkPostForward, HandleUpstreamPostForward.
• common/request.go (opens in a new tab) — NormalizedRequest: directive system, NextUpstream round-robin, MarkUpstreamCompleted, composite type, finality caching, ExecState.
• erpc/query_executor.go (opens in a new tab) — EvmQueryExecutor.Execute: structured query dispatch; native pipe-through vs. shim routing.

Related pages

• Survive provider outages — retry, hedge, and circuit breaker in depth.
• Cut costs & latency — cache policies, TTLs, and storage drivers.
• Scale chains & providers — selection policies, scoring, and shadow upstreams.
• Trust the data — consensus, integrity checks, and getLogs splitting.
• Lock it down — auth strategies, rate limiters, CORS.
• See everything — Prometheus metrics, OTel spans, dashboards.
• Retry — retry policy config reference.
• Hedge — hedge policy config reference.
• Rate limiters — budget and rule configuration.