Upstreams

How it works

Config-time pipeline. When eRPC loads a project, each upstream in upstreams[] first runs ApplyDefaults against upstreamDefaults — only-if-unset for scalar fields; all-or-nothing for tags, failsafe, and routing; per-field merge for EVM sub-fields; wholesale-then-per-field for SVM sub-fields — then SetDefaults fills remaining gaps: deriving an id if empty, defaulting type to evm only when it was omitted (so an SVM upstream must say type: svm), injecting ignoreMethods: ["*"] when allowMethods is set without an explicit ignoreMethods. Endpoints using a vendor shorthand (alchemy://KEY, drpc://KEY, and 20+ other prefixes) are converted into provider configs at this stage; the rest of the upstream block travels as a "*" override. (common/defaults.go:L1152-1241)

Runtime bootstrap. Every upstream registers as a named background BootstrapTask (upstream/<id>) so service startup never blocks. The initializer retries with exponential backoff (3 s → 130 s, factor 1.5) unless the task returns a TaskFatal error. Bootstrap calls eth_chainId, stores the detected value, derives networkId = evm:<chainId>, and starts the EVM state poller. A chain ID mismatch against a configured evm.chainId is fatal (no retry). A state-poller startup failure is not fatal — the upstream registers and availability checks fail-open until the poller recovers. (upstream/registry.go:L95-104)

Request path. Before forwarding, shouldSkip runs in order: shadow upstreams skip real traffic → evm.skipWhenSyncing with a syncing poller → ShouldHandleMethod (ignore → allow with wildcard patterns, result cached per method forever) → use-upstream directive matching (upstream ID first, then tags for purely-positive patterns). Block-availability gating is deliberately deferred to the network layer so a "block slightly ahead" is classified retryable rather than short-circuited. (upstream/upstream.go:L1505-1548)

Block availability. An upstream can declare the block window it serves via evm.blockAvailability. Bounds can be relative to the chain tip (latestBlockMinus), relative to the detected earliest block (earliestBlockPlus), or fixed (exactBlock). When a request targets a block outside the window, eRPC skips the upstream and tries the next rather than failing the request. The network layer turns an unavailable-block skip into ErrUpstreamBlockUnavailable, retryable when the block is within 128 blocks of the tip (configurable via networks[].evm.maxRetryableBlockDistance). (erpc/networks.go:L1919-1987)

Set it once for a whole project under upstreamDefaults instead of repeating it on every upstream — useful when a pool is uniformly pruned (e.g. nodes started with a retention window). An upstream that declares its own blockAvailability or the deprecated maxAvailableRecentBlocks keeps its own value; the project default only fills upstreams that declare neither, and it takes precedence over the 128-block window otherwise derived from nodeType: full. (common/defaults.go:L2029-2042)

projects:
  - id: main
    upstreamDefaults:
      evm:
        blockAvailability:
          lower:
            latestBlockMinus: 604800 # serve only the most recent ~604800 blocks

When every upstream skips the block, the client receives a JSON-RPC error with code -32014 ("block not available") rather than a generic internal error, so a pruned range is distinguishable from eRPC itself failing. (common/json_rpc.go:L1715-1740)

Every upstream must agree. If one of them instead timed out or returned a server error, that upstream never answered, so the request falls back to a generic retryable error rather than claiming the block is definitively absent. (common/json_rpc.go:L1588-1606)

Cordoning. Cordon state lives on the health tracker as an explicit flag — it is NOT a rolling-window metric and survives window rotation and idle eviction. Admin RPCs via the Admin API: erpc_cordonUpstream, erpc_uncordonUpstream, erpc_listCordoned. The selection-policy engine exposes cordonedReason to the JS policy; the default policy's removeCordoned() step drops cordoned upstreams from routing. (health/tracker.go:L793-849)

HTTP client and proxy pools. Each HTTP upstream gets a pre-warmed http.Transport with up to 256 idle connections per host (unlimited active), TCP keepalive at 15-second intervals, and a 60-second end-to-end call timeout. Proxy pools let you route outbound traffic through SOCKS5 or HTTP proxies, round-robined by an atomic counter.

Fixed transport parameters (not user-configurable; same values for direct and proxy transports):

(clients/http_json_rpc_client.go:L109-128; proxy transports: clients/proxy_pool_registry.go:L82-98)

Config schema

All paths relative to projects[*]. "Default" = value after SetDefaults/ApplyDefaults.

upstreamDefaults inheritance semantics — scalar vs composite fields differ critically:

• Scalar fields (endpoint, type, vendorName, rateLimitBudget, autoIgnoreUnsupportedMethods): inherited individually when the upstream's own value is zero.
• tags: all-or-nothing — if the upstream declares even one tag, no defaults tags are inherited.
• failsafe: all-or-nothing — any failsafe entry on the upstream means none from defaults apply.
• routing: all-or-nothing — same rule.
• EVM sub-fields (statePollerInterval, statePollerDebounce, maxAvailableRecentBlocks, auto-splitting thresholds, integrity): per-field merge when both sides are non-nil.
• SVM sub-fields (chain, cluster, checkGenesisHash): an upstream with no svm block inherits upstreamDefaults.svm wholesale (copied, not pointer-shared); one with a partial block fills only its empty fields. This is the field set that lets a homogeneous pool declare chain/cluster once instead of repeating it per upstream.
• jsonRpc: shallow-copied from defaults only when the upstream has no jsonRpc block at all; even jsonRpc: {} blocks inheritance.

Block-availability bound kinds and probe types (for earliestBlockPlus bounds):

Worked examples

All patterns below are distilled from real production fleets; comments explain the non-obvious choices.

1. Full node + archive node pool with explicit block windows. Route recent blocks to a cheap full node and archive queries to a dedicated archive node. In production, internal nodes run at a 5-second state-poller cadence for tight head tracking, and latestBlockMinus: 0 on the upper bound prevents the cache upstream from serving stale-head responses:

projects:  - id: main    upstreams:      - id: full-node        endpoint: https://rpc.example.com        evm:          chainId: 1          statePollerInterval: 5000ms          blockAvailability:            # eRPC only sends this upstream blocks it can actually serve            lower:              latestBlockMinus: 128            upper:              # latestBlockMinus: 0 = "no blocks beyond current head" —              # prevents sending future/reorg blocks to a lagging node              latestBlockMinus: 0              probe: blockHeader      - id: archive-node        endpoint: https://archive.rpc.example.com        evm:          chainId: 1          # no lower bound = unbounded archive (serves any historical block)          statePollerInterval: 10s

2. gRPC cache upstream scoped to read methods only. Internal deployments wire a gRPC cache service as a high-priority upstream that only handles cacheable read methods. The cache gets a high scoreMultipliers.overall (3.5×) so the selection engine prefers it; earliestBlockPlus + updateRate keeps the lower bound re-evaluated hourly as the cache fills:

projects:  - id: main    upstreams:      - id: grpc-cache        # gRPC cache connector — high score so selection engine prefers it        endpoint: grpc://cache.internal.svc.cluster.local:8033        vendorName: internal-cache        routing:          scoreMultipliers:            - network: ""              method: ""              # 3.5× score: cache hits should almost always be tried first              overall: 3.5        evm:          chainId: 42161          statePollerInterval: 1000ms          blockAvailability:            lower:              # probe re-runs every 5s so bounds reflect actual cache fill              earliestBlockPlus: 0              probe: blockHeader              updateRate: 5s            upper:              latestBlockMinus: 0              probe: blockHeader        # only serve the methods the cache actually holds        ignoreMethods:          - "*"        allowMethods:          - eth_getLogs          - eth_getBlockByHash          - eth_getBlockReceipts          - eth_getBlockByNumber          - eth_getTransactionByHash          - eth_getTransactionReceipt        failsafe:          - matchMethod: "*"            timeout:              # tight cap: cache reads must be fast or it's not worth it              duration: 1s              quantile: 0.8              minDuration: 200ms              maxDuration: 1s            hedge:              # race a second cache read quickly — latency is stable enough              # for static-style floor here (50ms)              quantile: 0.9              maxCount: 1              minDelay: 50ms              maxDelay: 100ms

3. Vendor provider as deprioritized fallback with rate-limit auto-tune. Paid vendors like Alchemy are expensive at volume. Production configs set scoreMultipliers.overall: 0.2 to deprioritize them behind cheaper nodes, and pair a rateLimitBudget with rateLimitAutoTune so eRPC backs off automatically when the vendor starts throttling (decreaseFactor 0.7 cuts budget aggressively on errors; increaseFactor 1.1 recovers slowly):

projects:  - id: main    upstreams:      - id: alchemy-mainnet        endpoint: https://eth-mainnet.g.alchemy.com/v2/${ALCHEMY_KEY}        vendorName: alchemy        evm:          chainId: 1          statePollerInterval: 10s        rateLimitBudget: alchemy-global        rateLimitAutoTune:          enabled: true          adjustmentPeriod: 30s          # start backing off once 10% of requests are errors          errorRateThreshold: 0.1          # recover slowly (×1.1 per period), cut fast (×0.7)          increaseFactor: 1.1          decreaseFactor: 0.7          minBudget: 1          maxBudget: 100000        routing:          scoreMultipliers:            - network: ""              method: ""              # 0.2× score: use Alchemy only when cheaper upstreams are unavailable              overall: 0.2
rateLimiters:  budgets:    - id: alchemy-global      rules:        - method: "*"          maxCount: 500          period: 1s

4. Per-upstream method failsafe tiers with upstreamDefaults. Production fleets share a single upstreamDefaults.failsafe slice across all upstreams. Heavy methods (eth_getLogs, eth_getBlockReceipts) get a wide timeout with a high quantile floor to avoid cutting off big-range subgraph backfills; light getters get a tight cap to encourage fast failover. Individual upstreams that need special treatment (like a fast internal gRPC reader) override the whole failsafe slice — all-or-nothing:

projects:  - id: main    upstreamDefaults:      # shared across all upstreams; one upstream with its own failsafe[] inherits none of this      autoIgnoreUnsupportedMethods: false      ignoreMethods:        - eth_newFilter        - eth_newBlockFilter        - eth_newPendingTransactionFilter        - eth_getFilterChanges        - eth_getFilterLogs        - eth_uninstallFilter      evm:        getLogsAutoSplittingRangeThreshold: 5000      rateLimitAutoTune:        enabled: true        adjustmentPeriod: 30s        errorRateThreshold: 0.1        increaseFactor: 1.1        decreaseFactor: 0.7        minBudget: 1        maxBudget: 100000      failsafe:        - matchMethod: "eth_getLogs|eth_getBlockReceipts"          timeout:            # 2s floor: big-range queries legitimately take seconds on archive nodes            duration: 15s            quantile: 0.9            minDuration: 2s            maxDuration: 15s          hedge: null          retry: null        - matchMethod: "eth_get*"          timeout:            # 200ms floor: light point-lookups should be fast            duration: 5s            quantile: 0.9            minDuration: 200ms            maxDuration: 5s          hedge: null          retry: null        - matchMethod: "*"          timeout:            duration: 60s            quantile: 0.8            minDuration: 500ms            maxDuration: 60s          hedge: null          retry: null
    upstreams:      - id: eth-reader        endpoint: grpc://reader.internal.svc.cluster.local:50051        vendorName: internal-reader        evm:          chainId: 1        ignoreMethods: ["*"]        allowMethods:          - eth_getBlockByNumber          - eth_getLogs          - eth_getTransactionByHash          - eth_getTransactionReceipt        rateLimitBudget: reader-global        rateLimitAutoTune:          enabled: true          minBudget: 10          maxBudget: 10000          # very sensitive: trip at 1% errors, recover slowly (0.98×)          errorRateThreshold: 0.01          decreaseFactor: 0.98          increaseFactor: 1.1          adjustmentPeriod: 30s        failsafe:          # Overrides upstreamDefaults entirely — tight cap so network-level          # retry races cheaper upstreams instead of bouncing back to the reader          - matchMethod: "*"            timeout:              duration: 500ms              quantile: 0.8              minDuration: 100ms              maxDuration: 500ms            retry:              maxAttempts: 1

5. Per-chain getLogsAutoSplittingRangeThreshold overrides. High-throughput chains like Arbitrum and Sei have providers that reject eth_getLogs queries over large block ranges. Production configs set a tighter threshold on those chains' upstreams (1 000 for Arbitrum, 500 for Sei) while leaving the default 5 000 in upstreamDefaults.evm for chains with more forgiving providers:

projects:  - id: main    upstreamDefaults:      evm:        # generous default for most chains        getLogsAutoSplittingRangeThreshold: 5000
    upstreams:      - id: arbitrum-mainnet-node        endpoint: https://rpc.example.com        evm:          chainId: 42161          # Arbitrum providers reject ranges >1000 blocks for heavy contracts          getLogsAutoSplittingRangeThreshold: 1000          statePollerInterval: 5000ms          blockAvailability:            upper:              latestBlockMinus: 0              probe: blockHeader
      - id: sei-mainnet-node        endpoint: https://sei.rpc.example.com        evm:          chainId: 1329          # Sei's high block rate makes large ranges expensive for providers          getLogsAutoSplittingRangeThreshold: 500          statePollerInterval: 5000ms

6. A Solana upstream pool with upstreamDefaults.svm. type: svm must be explicit on every upstream — it is never inferred from the endpoint or from the network. chain and cluster are the two fields that decide network eligibility, and unlike the network side they are inheritable, so a pool homogeneous across one cluster declares them once. Each SVM upstream gets its own slot/health poller automatically:

projects:  - id: main    upstreamDefaults:      # type is NOT inferred — declaring it here covers the whole pool      type: svm      svm:        # cluster IS inherited on upstreams (it is network identity only on networks)        cluster: mainnet-beta    upstreams:      - id: solana-labs-mainnet        endpoint: https://api.mainnet-beta.solana.com      - id: helius-mainnet        endpoint: https://mainnet.helius-rpc.com/?api-key=${HELIUS_KEY}      # a fork lives in its own network; checkGenesisHash opts the unknown      # (chain, cluster) pair into bootstrap verification      - id: fogo-mainnet        endpoint: https://mainnet.fogo.io        svm:          chain: fogo          cluster: mainnet          checkGenesisHash: true

Request/response behavior

• shouldSkip runs before every Forward call. Skip reasons surface as ErrUpstreamRequestSkipped (HTTP 406) wrapping the underlying reason: ErrUpstreamShadowing, ErrUpstreamSyncing, ErrUpstreamMethodIgnored, ErrUpstreamNotAllowed. (upstream/upstream.go:L1505-1548)
• Block-availability failures produce ErrUpstreamBlockUnavailable (HTTP 503). The network layer classifies them retryable iff block > latest > 0 and distance ≤ maxRetryableBlockDistance (default 128); beyond that they are wrapped as ErrUpstreamRequestSkipped to stop retries. (erpc/networks.go:L1862-1881)
• Computed bounds with finite min > max (e.g. earliestBlockPlus before detection completes) log a WARN and fail open — the upstream serves any block rather than incorrectly rejecting all of them. (upstream/upstream.go:L1199-1210)
• eth_chainId detection retries forever on transport errors; non-numeric chainId or configured-vs-detected mismatch is TaskFatal — the upstream stops retrying permanently. (upstream/upstream.go:L1419-1455)
• Cordon state is an explicit flag on the health tracker (NOT a metric). It survives rolling-window rotation and idle-sweep eviction; only Reset() (test/admin) clears it. erpc_listCordoned only reports wildcard ("*") scope cordons — method-scoped cordons are not listed. (health/tracker.go:L598-615)

Best practices

• Declare block windows explicitly. Use evm.blockAvailability.lower.latestBlockMinus: 128 for pruned full nodes rather than the deprecated nodeType: full — the explicit form is visible in config dumps and survives future refactors.
• Never rely on allowMethods alone. Setting allowMethods without ignoreMethods silently injects ignoreMethods: ["*"], blocking everything else. When you want a partial allow-list alongside a partial ignore-list, set both fields explicitly.
• List all tags on each upstream when mixing with defaults. One tag on an upstream drops the entire upstreamDefaults.tags slice — there is no additive inheritance. Enumerate every intended tag per upstream explicitly.
• Set batchMaxWait to a non-zero value. batchMaxWait: 0 fires the batch immediately via time.AfterFunc(0, ...) — effectively the same as no batching. Use at least 10ms to allow real coalescing.
• Use rateLimitBudget + rateLimitAutoTune together. Auto-tune adjusts the effective budget based on observed error rate (10% threshold, 1-minute cadence by default). Without a budget, the auto-tuner is never created.
• Cordon instead of removing. For emergency isolation — provider outage, billing alarm — call erpc_cordonUpstream via the Admin API. The cordon state survives metric-window rotation and does not need a config change or restart.
• Pick the right earliestBlockPlus probe for your archive node. Most nodes use blockHeader (default). Use traceData only when the upstream has separate pruning boundaries for trace methods; eventLogs when log indices are available further back than state; callState when gating on historical balance/storage availability.

Edge cases & gotchas

1. ws:///wss:// pass config validation but fail at client creation with "websocket client not implemented yet" and retry forever. Use http:///https:// instead.
2. allowMethods without ignoreMethods silently blocks all other methods via an injected ignoreMethods: ["*"]. To allow specific methods while keeping defaults, use ignoreMethods with explicit patterns instead.
3. All-or-nothing tags inheritance: one tag on an upstream drops all upstreamDefaults.tags. List every intended tag explicitly on each upstream.
4. vendorName typo returns nil from vendor lookup silently — the upstream runs as a bare endpoint without error normalization. No warning is logged.
5. Repeated Cordon calls update the reason but not the start timestamp (duration accounting survives reason edits). Changing reason creates a new gauge series; old series stays at 1 until a matching uncordon.
6. erpc_listCordoned only reports wildcard ("*") scope cordons — method-scoped cordons are not listed.
7. Uncordoning a method-scoped cordon does NOT clear a wildcard cordon — IsCordoned checks "*" first.
8. maxAvailableRecentBlocks and blockAvailability set simultaneously: the back-compat synthesis fires only when blockAvailability == nil; explicit blockAvailability silently wins.
9. Lower-bound re-poll race: within 10 blocks of the pruning boundary a fresh poll runs; if the chain advanced, firstAvailable moves up and a block that was valid may flip to unavailable mid-check.
10. batchMaxWait: 0 fires instantly — time.AfterFunc(0, ...) schedules the flush before additional requests can join. Set at least 10ms to coalesce.
11. batchMaxSize: 0 with supportsBatch: true means the batch fires immediately on the first queued request because 0 >= 0. Use a value ≥ 2 to actually coalesce.
12. Proxy URL scheme validation is a case-insensitive prefix check, not URL parsing. socks4:// and other schemes fail at startup even though they are syntactically valid URLs.
13. UniqueUpstreamKey is order-unstable with ≥2 jsonRpc.headers — Go map iteration is randomized, so two calls may produce different keys and re-create the client on each call.
14. Negative or zero block numbers pass block-availability checks — extracted block ≤ 0 is treated as "block not present in request" and the check is skipped (fail-open).
15. Method-support results are cached forever per process. ShouldHandleMethod caches per method name; later edits to Ignore/Allow lists don't invalidate existing entries. Only IgnoreMethod explicitly writes a false entry for its own case.
16. State-poller bootstrap failure does NOT block registration — upstream registers and serves; availability checks that need latest-block data error/fail-open until the poller recovers in background.
17. earliestBlockPlus before detection completes computes 0 + N. If the probe hasn't run yet, earliest = 0, so bound = N. If N > latest, this can produce an invalid range (min > max) that triggers the fail-open WARN path.
18. Forwarded headers are NOT sent on batch requests. req.ForwardHeaders is consulted only in sendSingleRequest; callers relying on bearer-token forwarding must not use batching for those requests.
19. Duplicate JSON-RPC IDs within a batch window flush the current batch. If a second request arrives with the same id as one already queued, the pending batch fires immediately and the new request starts a fresh batch. This preserves ID-to-response mapping correctness but reduces batch efficiency for clients that reuse IDs. (clients/http_json_rpc_client.go:L255-261)
20. ErrUpstreamMalformedResponse is retryable. Raised when a batch response body is neither a JSON array nor a JSON object. Because it is not in the non-retryable allowlist, eRPC retries on another upstream. If all upstreams return malformed responses the request ends as ErrUpstreamsExhausted. HTTP status code on propagation is 400 (method-level), but the JSON-RPC wire response to the caller is still HTTP 200.
21. type is not inferred from the endpoint. A Solana endpoint without type: svm defaults to evm, and bootstrap then tries eth_chainId against it. Set type: svm on the upstream or on upstreamDefaults.
22. An SVM upstream whose svm.chain/svm.cluster do not match the network's is not eligible for it. There is no fuzzy match — an upstream with cluster: mainnet-beta never serves the svm:testnet network. Remember chain: "" means solana, so an upstream omitting chain cannot serve a chain: fogo network.
23. Genesis-hash validation is fail-closed for known clusters. For Solana mainnet-beta/devnet/testnet, both a hash mismatch and a getGenesisHash fetch failure fail the upstream — eRPC refuses to register one it could not verify. checkGenesisHash only affects unknown (chain, cluster) pairs.
24. SVM upstreams are cordoned by their own state poller. Beyond manual and breaker-driven cordons, a failing getHealth or a shred-insert lag above 100 slots removes an SVM upstream from selection until it recovers. See SVM slot tracking & health.

Observability

Source code entry points

• upstream/upstream.go:L136 (opens in a new tab) — Upstream type: construction, bootstrap, detectFeatures, Forward hot path, shouldSkip, ShouldHandleMethod, block-availability assertions, cordon pass-throughs.
• upstream/registry.go:L95 (opens in a new tab) — UpstreamsRegistry: background bootstrap tasks, per-network upstream maps + atomic snapshots, PrepareUpstreamsForNetwork wait algorithm, shadow registration.
• common/defaults.go:L1152 (opens in a new tab) — ApplyDefaults (upstreamDefaults inheritance) + SetDefaults chain (id derivation, allowMethods→ignoreMethods injection, EVM defaults, legacy nodeType→blockAvailability synthesis, vendor shorthand conversion).
• erpc/networks.go:L1919 (opens in a new tab) — checkUpstreamBlockAvailability, resolveEnforceBlockAvailability, handleBlockSkip — single enforcement point for block availability at network layer.
• health/tracker.go:L793 (opens in a new tab) — cordon state storage, Cordon/Uncordon/IsCordoned/CordonedReason, idle-sweep protection.
• clients/http_json_rpc_client.go:L109 (opens in a new tab) — GenericHttpJsonRpcClient transport construction, batch aggregation, gzip pools, error normalization.
• clients/proxy_pool_registry.go:L23 (opens in a new tab) — ProxyPool round-robin selection via atomic counter.
• upstream/upstream_block_availability_test.go (opens in a new tab) — exhaustive availability edge cases: negative/zero blocks, reorgs, pruning races, poll-count assertions.

Related pages

• Selection policies — how upstreams are scored and ordered for each request.
• Rate limiters — budget-based throttling bound to upstream IDs.
• Failsafe — per-upstream retry, hedge, timeout, and circuit-breaker config.
• Matcher / use-upstream — directive that pins a request to a specific upstream or tag.
• Admin API — cordon/uncordon RPCs and listing cordoned upstreams.
• getLogs splitting — auto-splitting thresholds for eth_getLogs and trace_filter.
• SVM slot tracking & health — the per-upstream Solana slot/health poller and its cordoning behavior.
• SVM commitment & finality — how requests are normalized before they reach an SVM upstream.
• Survive provider outages — how upstreams + failsafe combine to keep requests flowing.