# Block tracking & served tip > Source: https://docs.erpc.cloud/reference/evm/block-tracking > eRPC keeps a live pulse on every upstream's chain head and finality, then distills a single honest block number your clients can trust — with no "block not found" surprises. > Format: machine-readable markdown export of the docs page above. > All collapsible AI sections are inlined and fully expanded. # Block tracking & served tip Every upstream sees the chain at a slightly different block. Without coordination, routing a request to the one node that is a block behind causes silent "block not found" failures. eRPC polls every upstream continuously and advertises one **served tip** per axis (`latest`, `finalized`): the highest block at least `servedTip.quorum` (default 2) eligible upstreams already have. **What you get** - Chain head and finality tracked per upstream, refreshed automatically at the right cadence for each chain's block time - One rule for every config: a block is served only once a quorum of upstreams has it — a lone far-ahead or wrong-chain node never defines `latest` - Stalled heads benched automatically, so one frozen node cannot hold `latest` back - Optional per-group guarantee so the advertised tip stays servable by the upstreams your consensus policy requires - Per-upstream lag gauges that feed selection policies so slow nodes are deprioritized before requests reach them - Redis-backed counters share heads across replicas; each replica samples latest-block timestamps on its own debounce schedule. ## Quick taste Illustrative, not a tuned production config — require a strict majority of upstreams instead of the default 2: **Config path:** `projects[].networks[].evm.servedTip` **YAML — `erpc.yaml`:** ```yaml projects: - id: main networks: - architecture: evm evm: chainId: 1 servedTip: # serve the highest block a strict majority of upstreams has quorum: majority ``` **TypeScript — `erpc.ts`:** ```typescript projects: [{ id: "main", networks: [{ architecture: "evm", evm: { chainId: 1, servedTip: { // serve the highest block a strict majority of upstreams has quorum: "majority", }, }, }], }] ``` ## Agent reference Copy one of these prompts into your AI agent session (Claude Code, Cursor, …) — each one points the agent at this page's machine-readable reference so it can do the work correctly: **Prompt Example #1: eliminate block-not-found errors across my upstream fleet** ```text My app gets occasional "block not found" errors right after a new block is produced because some upstreams lag behind. Set servedTip.quorum in my eRPC config so eRPC only advertises a block number enough upstreams actually have, and explain when a stalled upstream gets benched out of the ballot. Read the full reference first: https://docs.erpc.cloud/reference/evm/block-tracking.llms.txt ``` **Prompt Example #2: tune block polling for a fast chain like Arbitrum** ```text I'm running eRPC against Arbitrum (250ms block time) and the default 5s fallbackStatePollerDebounce causes stale served-tip data on startup. Adjust the polling and debounce settings in my eRPC config so the EMA warms up quickly and block tracking stays tight. Reference: https://docs.erpc.cloud/reference/evm/block-tracking.llms.txt ``` **Prompt Example #3: set up guaranteed-method floor for a mixed trace fleet** ```text My eRPC pool mixes trace-capable archive nodes with cheaper non-archive upstreams. Without a floor, the served tip can advance to a block that only the non-archive nodes have, causing trace_block requests to hit "block not found". Configure guaranteedMethods in my eRPC config and explain how the method floor interacts with the quorum. Reference: https://docs.erpc.cloud/reference/evm/block-tracking.llms.txt ``` --- ### Block tracking & served tip — full agent reference ### How it works Each upstream gets its own `EvmStatePoller` background goroutine. On each tick of `statePollerInterval` (default 30 s), three concurrent goroutines fetch the latest block, finalized block, and syncing state. A synchronous `Poll()` runs once at bootstrap time to seed the upstream before any request traffic arrives. Remote head updates do not postpone local latest-block sampling. A local fetch can attach a missing timestamp to the same block already received from a peer. The block-time estimator accepts that matched observation without advancing the head again. Latest-block polling cost scales with replica count. **Routing the `safe` tag.** On op-stack chains, `safe` is node-relative because each verifier chooses its own L1 confirmation depth. If providers in one pool use different depths, they can honestly return different blocks for the same tag. Set `evm.safeBlockSource` to an upstream id/tag selector when one pool should define that answer: ```yaml networks: - architecture: evm evm: chainId: 8453 safeBlockSource: "tier:trusted" upstreams: - id: node-a tags: ["tier:trusted"] - id: node-b tags: ["tier:trusted"] - id: provider-a ``` eRPC keeps the `safe` tag verbatim and routes standard block-tagged JSON-RPC requests only to matching upstreams, so the selected pool defines both block height and hash. Cache reads are skipped because an older entry may have been produced by a different source. Operator routing overrides client `use-upstream`; retry and consensus still run inside the selected JSON-RPC pool, so configure participant thresholds accordingly. `eth_query*` and direct gRPC Query APIs retain their existing behavior and are outside this option's scope. If no source is configured or inherited, routing is unchanged. If trusted sources are unavailable, prefer `finalized` for a provider-independent answer. **Debounce resolution.** To avoid redundant RPC calls, each fetch is gated by `resolveDebounce()`, which picks the first applicable value from: 1. `upstream.evm.statePollerDebounce` — hard per-upstream override; short-circuits all other steps. 2. `tracker.GetNetworkBlockTime × dynamicBlockTimeDebounceMultiplier` (default 0.7) — EMA-derived; active after ≥3 samples. 3. `network.evm.fallbackStatePollerDebounce` (default 5 s) — used while EMA warms up. 4. Hard 1 s floor — applies only if both step 2 and 3 yield zero. On a 12 s Ethereum Mainnet chain this yields ~8.4 s debounce — fresh enough to catch every block, conservative enough that concurrent pollers share one actual RPC call per window. In Redis-backed multi-pod deployments, the shared counter coalesces fetches across replicas too. **Finalized debounce.** The finalized fetch widens that debounce to `(latest − finalized) × EMA block time × finalizedDebounceDepthMultiplier` (default 0.05) when this is longer, using the upstream's own latest and finalized blocks. Finalized advances once per finality window, so polling it at the latest-block cadence mostly returns the number already held. With the default the finalized block trails by at most about 5% of the window: ~43 s on Ethereum Mainnet (~72 blocks × 12 s), ~45 s on an Arbitrum-style L2 with a 15 min window. An upstream behind the network's corroborated finalized block (second-highest across upstreams) keeps the regular debounce until it catches up: once a peer sees the next finalized block, the others sample it at the regular cadence. This also covers vendors whose load balancer mixes stale and fresh backends. Chains with near-instant finality (depth of a few blocks) keep the regular debounce. Unknown finalized, latest at or below finalized, or a cold EMA also keep it. Source: [`architecture/evm/evm_state_poller.go:L449-471`](https://github.com/erpc/erpc/blob/main/architecture/evm/evm_state_poller.go#L449-L471) **Dynamic block-time EMA.** The EMA (`alpha = 0.1`, ~19-sample effective window) lives in `health.Tracker` and updates whenever the corroborated network-level latest block advances, using the on-chain timestamp of the upstream report that holds that head (each upstream's latest timestamp is stored beside its head, so the sample comes from whichever poller reported the block). All head publication for a network is serialized under one per-network lock, so a slow re-derivation can never overwrite a newer one. On fast chains where consecutive blocks share the same integer second, block gaps accumulate and samples are normalized. The EMA is not published until ≥3 samples are collected (requiring 4 total advances, because the first stores the baseline without incrementing the sample counter). Values outside `[10 ms, 120 s]` are rejected and reset to the last valid value to prevent runaway spikes after a chain halt. Consumers receive 0 until warm-up completes. **Per-upstream block counters.** Both `latestBlockShared` and `finalizedBlockShared` are forward-only shared counters: `TryUpdate` rejects values ≤ the current value. A value that would regress more than 1024 blocks fires an `OnLargeRollback` callback and records `erpc_upstream_block_head_large_rollback`. **Network head and `blockHeadLag` / `finalizationLag`.** The tracker's network-level latest and finalized heads are *corroborated*: the second-highest head across the non-benched reporting upstreams, or the only head while a single one has reported (the same order statistic as the served tip's default quorum). When every reporter is benched it falls back to all of them. One upstream can therefore never move the reference every other upstream is measured against — a lone far-ahead report (an endpoint answering for another chain, a corrupted response) is not corroborated, and a benched (stalled) upstream cannot hold the head back. With exactly two reporters the head is the lower one, so a lone high report and a lone stale report are indistinguishable there; per-upstream block-availability checks still gate block-specific requests. The reporting set is every upstream that has reported a latest or finalized head to the tracker. `blockHeadLag = max(0, network_latest − presumed_latest)`; `finalizationLag = max(0, network_finalized − presumed_finalized)`. An upstream's presumed head is its last reported head plus what the network head made since that report, counted for at most one poll period (`max(statePollerInterval, statePollerDebounce or the block time)`): within the period its lag stays what it was at the report, because a stored head ages between polls by design. Without this, an upstream polled every 10s on a 0.4s chain read up to 25 blocks behind late in each poll window, and a selection policy ticking at the same period read the same phase on every tick, excluding a healthy upstream for good on some processes. Past the period the credit freezes, so a missing report counts as lag again; a benched head gets no credit. Both are recomputed on every `SetLatestBlockNumber` / `SetFinalizedBlockNumber` call and exposed as Prometheus gauges; the raw per-upstream `erpc_upstream_latest_block_number` still shows an outlier's own report. These drive selection-policy predicates such as `blockNumberLagAbove`. Source: [`health/tracker.go:L1468-1519`](https://github.com/erpc/erpc/blob/main/health/tracker.go#L1468-L1519) #### Served tip Both axes, every config. `EvmHighestLatestBlockNumber` / `EvmHighestFinalizedBlockNumber` run one pipeline per evaluation (`servedTip`): 1. **Candidates** (`EvmRoutableUpstreams`): the selection policy's eligible upstreams in routing order (all registered upstreams until the policy's first decision), narrowed to the request's `use-upstream` selector when present. 2. **Voter rank** (`tipVoterRank`): each candidate with a head on the axis ranks live 0, benched 1, static cap 2, benched static cap 3. A static cap is a `blockAvailability.upper.exactBlock` the chain has passed: a serving range, not a head observation (`latestBlockMinus` upstreams are live). A candidate that is syncing per `eth_syncing`, has no state poller or has no head yet does not vote. 3. **Ballot** (`evmTipBallot`): only the heads of the best voter rank present vote. A benched head votes only when no non-benched live head exists; a static cap only when no live head exists. 4. **Pick** (`PickServedTip`): the highest block at least `servedTip.quorum` voters already have — the q-th highest head, or the lowest when fewer than q vote. 5. The [regression guard](#regression-guard), then the [WebSocket delivered-head floor](#websocket-delivered-head-floor) (network-wide `latest` only), then the [guarantee floors](#guarantee-subsets). The result is the value served, recorded on the request's [lane](#selector-scoped-served-tip) and [exported as gauges](/reference/metrics.llms.txt#area-3-block-number-and-tip-tracking). The network-wide lane is also evaluated every second, both axes, so its gauges, advance age and guard memory stay current on a network with little traffic; use-upstream lanes are evaluated only by their requests. Nothing is persisted or predicted. Source: [`erpc/networks.go:L870-942`](https://github.com/erpc/erpc/blob/main/erpc/networks.go#L870-L942), [`erpc/networks.go:L430-506`](https://github.com/erpc/erpc/blob/main/erpc/networks.go#L430-L506), [`common/served_tip.go:L112-157`](https://github.com/erpc/erpc/blob/main/common/served_tip.go#L112-L157), [`erpc/networks_served_tip_refresh.go:L9-42`](https://github.com/erpc/erpc/blob/main/erpc/networks_served_tip_refresh.go#L9-L42) **Quorum.** `servedTip.quorum` is an integer ≥ 1 or `"majority"` (`floor(N/2)+1` of N voters), capped at N ([config](/config/projects/networks.llms.txt#config-schema)). Which head is served (heads sorted descending): | Voters N | `quorum: 2` (default) | `quorum: majority` | `quorum: 1` | |---|---|---|---| | 1 | 1st | 1st | 1st | | 2 | 2nd | 2nd | 1st | | 3 | 2nd | 2nd | 1st | | 4 | 2nd | 3rd | 1st | | 5 | 2nd | 3rd | 1st | `quorum: 1` serves the highest head, so a single upstream defines `latest`. A higher quorum trades freshness for servability: more upstreams already have the advertised block. Source: [`common/served_tip.go:L36-56`](https://github.com/erpc/erpc/blob/main/common/served_tip.go#L36-L56) #### Benching The health tracker re-evaluates every reporter on each latest-head report (`evaluateHeadBench`). An upstream is *behind* while a non-benched peer proves the chain went on without it (`headIsBehind`): - **By height:** a peer that moved after the upstream last moved is higher by no more than a plausible lead: 1024 blocks plus twice the blocks the chain produced since the upstream's head was current (by its last change and its head timestamp). Chain progress uses the network block time, else the peer's own, so a process that starts mid-stall still sees the stall. One such peer is enough. Each move of the upstream resets this rule. - **By staleness:** a peer that moved within its own or the upstream's stall window is higher within the plausible lead, and its newest block (never counted as newer than now) is at least the stall window younger than the upstream's head block. This holds while the upstream moves, so it catches a node replaying far behind the tip: after a restore, a restart or a deep rollback. One such peer is enough. Only the upstream's own head timestamp proves it stale: a head stored without one (a peer pod's poll through shared state) keeps the last verdict only while a live peer stays a stall window of blocks ahead. A node at the tip has a head block as young as any honest peer's, so a node that trails by less than the stall window of chain time is never benched. - **By block age:** a peer that moved after the upstream last moved has a newest block at least the stall window younger than the upstream's head block, whatever the heights. This catches a head frozen above a chain that restarted lower (a dead fork), a static far-ahead head, and an endpoint that serves another, slower chain under the same chain id. It takes two such peers while two exist. Behind without a break for its stall window (`headStallAfter`), `max(30s, 3 × max(statePollerInterval, statePollerDebounce or the block time))`, it is benched: its voter rank drops to benched and it leaves the lag reference. Its `blockHeadLag` keeps growing so `blockNumberLagAbove` routes around it: while its head has not moved since the bench (`movedSinceBench`), the head counts as no higher than the network head at the bench (a head frozen above a dead fork lags by what the network made since, even after the network passes its height); once it moves again, it lags by its real distance. It rejoins once it has moved since being benched (a move in the evaluation that benched it does not count), no rule above still proves it behind, and its head reached the corroborated non-benched head (second-highest, or the only one); or when no non-benched peer has a head. A voter at or above that head never lowers `latest`, and a lone far-ahead head cannot keep it out. A bench or rejoin also re-derives the finalized lag reference. Stall windows run on the monotonic clock, so a wall-clock step benches nobody. Each bench increments `erpc_upstream_head_benched_total` and logs a WARN. Bench state is per process. Source: [`health/tracker.go:L1463-1653`](https://github.com/erpc/erpc/blob/main/health/tracker.go#L1463-L1653) - **Nobody is benched** for a chain halt itself (no peer moves; a node already a stall window behind when the chain halted is benched like any trailing node), by a moving wrong-chain head above (its lead is implausible), or by one wrong-chain peer whose blocks are younger (the block-age rule needs two). - **Boundary:** a node that trails by less than the stall window of chain time (for example, 1-3 blocks on a 2s chain) is never benched. It votes with its lower head, which holds the pick back only when it is needed for the quorum (N=2 with the default quorum). `quorum: 1` serves the highest head instead. - **Boundary:** eRPC compares heights and block times, never block hashes, so it cannot tell a live fork from the canonical chain. During a halt, a node that moves within the plausible lead benches the halted ones after their stall window, and then defines `latest` alone. A benched dead-fork node that starts producing blocks again rejoins once its head reaches the corroborated head. - **Boundary:** the block-age rule cannot tell two heads frozen above one moving peer (a dead fork the minority left) from two halted honest nodes and one wrong-chain peer. It keeps the frozen pair, so a dead fork that most upstreams stayed on still defines `latest`. - **Boundary:** two chains under one chain id (an endpoint pointed at the wrong chain) differ by far more than any plausible lead, so only block age tells them apart. Until the slower chain goes twice its stall window without a block, quorum 2 over the pair serves the lower head; then the slower chain is benched, right or wrong. A wrong chain that makes blocks more often than that is never benched. Exclude such an endpoint in config. - **Boundary:** benching a voter that sat between the pick and a slower voter lowers `latest` to that slower voter's head, as any upstream leaving the ballot does. The regression guard holds only drops beyond `maxRegressionBlocks`. - **Boundary:** a `latestBlockMinus: N` upstream rejoins on its raw head but votes N below it, so its rejoin can lower `latest` by up to N. - **Boundary:** rejoin needs the benched head to reach the bar at one report. A node that always trails the second-highest head by a block or two rejoins late; until then it serves traffic while its lag stays under the policy threshold, but does not vote. - **Boundary:** a bench changes the ballots at once, but routing leaves the benched upstream at the selection policy's next evaluation (`evalInterval`, 15s by default). Until then a request for the new `latest` can reach it; without a configured availability bound nothing gates it, so its missing-data error is retried on another upstream. #### Regression guard When a pick falls more than `maxRegressionBlocks` (0 = 1024) below the corroborated evidence — the second-highest live head, or the last live-corroborated pick — the lane serves its last corroborated pick instead, never above the highest live head (`guardServedTipRegression`). It holds for at most 1 minute, then fails open and serves the pick as computed. In-process only; selectors without a lane get no guard; `-1` disables it. Each transition increments `erpc_network_served_tip_regression_total`. Source: [`erpc/networks.go:L944-1102`](https://github.com/erpc/erpc/blob/main/erpc/networks.go#L944-L1102) #### WebSocket delivered-head floor Network-wide `latest` (no `use-upstream`) is lifted to the highest head already delivered to a `newHeads` subscriber by this instance, or by any instance through the shared counter `network///latestBlock` (`applyDeliveredHeadFloor`). The floor applies only within `maxRetryableBlockDistance` of the ballot's highest head (or when no head is known). The guarantee floors run after it, so it never lifts `latest` past one. Source: [`erpc/networks.go:L1381-1409`](https://github.com/erpc/erpc/blob/main/erpc/networks.go#L1381-L1409) #### eth_blockNumber With `enforceHighestBlock` on (the default) and the served head known, eRPC answers `eth_blockNumber` from the served head without dispatching, scoped to the request's `use-upstream` selector. Each answer triggers `EvmRefreshLatestBlock`: in routing order, quorum-many voters of the best voter rank present — the voters the ballot counts — refresh their pollers in the background, each only when its stored head is older than its poll debounce, one refresh in flight per poller. The answer is the head as of the last refresh, so a client that alone polls every N seconds reads a head up to N seconds old (never older than `statePollerInterval`, and fresher when other traffic refreshes it); a request that follows it sees the refreshed head. With the head unknown (cold pollers) the request is forwarded. Every response that reaches the post-forward hook — an upstream answer or a cache hit — is pinned exactly to the served head when it is known: higher values are lowered, lower values are raised (an upstream answer below it increments `erpc_upstream_stale_latest_block_total`). `enforce-highest-block=false` forwards and returns the upstream's answer unchanged. Source: [`architecture/evm/eth_blockNumber.go:L13-150`](https://github.com/erpc/erpc/blob/main/architecture/evm/eth_blockNumber.go#L13-L150), [`erpc/networks.go:L761-795`](https://github.com/erpc/erpc/blob/main/erpc/networks.go#L761-L795) #### Guarantee subsets `guaranteedMethods` and `guaranteedFor` clamp the advertised tip down to the lowest majority pick of every configured guarantee subset, so it is never a block a required subset cannot serve. `guaranteedMethods` names a subset by method support; `guaranteedFor` names one by upstream id/tag selector — the same vocabulary as `use-upstream` and `consensus.requiredParticipants`. Each subset runs the same ballot (`guaranteedSubsetTip` → `evmTipBallot`, same voter ranks) at `quorum: majority` whatever the network quorum, folded to the minimum in `guaranteedFloor`; only the membership predicate differs. A majority, never a minimum, so a single frozen member is outvoted inside its group. A subset with no voters constrains nothing (the tip is never pinned to 0), and the clamp can only lower the tip — a group ahead of the network pick never raises it. A subset whose ballot holds only static caps floors the tip only when no REGISTERED member is live (`anyRegisteredLiveHead`): live members that are merely absent from the eligible set (lag-excluded, cordoned, mid-restart) must not hand back a frozen cap. Source: [`erpc/networks.go:L1411-1500`](https://github.com/erpc/erpc/blob/main/erpc/networks.go#L1411-L1500) #### Selector-scoped served tip With a `use-upstream` selector, `EvmRoutableUpstreams` narrows the candidates to the matching upstreams; the pick is the same rule over that subset. `servedTipLaneFor` returns the request's lane: the network-wide lane (`lane="all"`) without a selector, a group's own lane for a simple group selector, else none. A lane holds no pick state: only its gauge label, advance-age anchor and regression-guard memory. A selector without a lane gets the same pick with no guard and no gauges; only the network-wide lane gets the WebSocket floor. Which selectors form a lane, naming and the 16-lane cap: [Lanes](/reference/lanes.llms.txt). Source: [`erpc/networks.go:L564-655`](https://github.com/erpc/erpc/blob/main/erpc/networks.go#L564-L655) #### Syncing, finality and earliest block **Syncing state machine.** `eth_syncing` is called with a 5 s sub-context on each tick. The upstream is declared fully synced only after 4 consecutive "not syncing" responses (`synced counter ≥ 4`); any syncing response resets the counter to 1. The non-standard `{Ok: bool}` response shape is supported: `Ok=false` is treated as still-syncing. After 10 consecutive failures with no prior success, `skipSyncingCheck` is set permanently and the state is fixed at `Unknown` for the process lifetime. If `upstreams[*].evm.skipSyncingCheck: true` is set in config, `eth_syncing` is never called at all and the state is immediately fixed at `NotSyncing` — use this for nodes that always return a syncing object regardless of actual sync status. **Skip-on-failure semantics.** Both `skipLatestBlockCheck` and `skipFinalizedCheck` follow the same pattern: 10 consecutive failures with no prior successful poll sets the skip flag permanently and the poller stops calling that method. Once the upstream has had at least one success (`*SuccessfulOnce = true`), failures are only logged and never trigger the skip. This prevents false-positive disablement during transient degradations. Source: [`architecture/evm/evm_state_poller.go:L420-L445`](https://github.com/erpc/erpc/blob/main/architecture/evm/evm_state_poller.go#L420-L445) **`IsBlockFinalized` decision tree:** 1. Both `finalizedBlock == 0` and `latestBlock == 0` → `ErrFinalizedBlockUnavailable`. 2. `finalizedBlock > 0` → `blockNumber ≤ finalizedBlock`. 3. `finalizedBlock == 0` and `latestBlock > 0` → infer `finalizedBlock = latestBlock − FallbackFinalityDepth` (default 1024) and compare. **Earliest block detection.** When `blockAvailability.lower.earliestBlockPlus` is configured, `binarySearchEarliest` finds the first block in `[0, latestBlock]` where a probe RPC call returns data. Probe types: `blockHeader` (`eth_getBlockByNumber`), `eventLogs` (`eth_getLogs`, requires ≥1 log entry), `callState` (`eth_getBalance`), `traceData` (tries `trace_block`, `debug_traceBlockByHash`, `trace_replayBlockTransactions` in order). Results are stored in cross-pod shared counters. If `blockAvailability.*.updateRate` is set, a scheduler goroutine re-runs the search at that interval. ### Config schema #### Upstream-level (`upstreams[*].evm.*`) | Field | Type | Default | Behavior / footguns | |---|---|---|---| | `upstreams[*].evm.statePollerInterval` | Duration | `30s` | How often `Poll()` fires. **`0` disables the poller entirely**: no background goroutine, no initial synchronous poll. Consequences: `LatestBlock()` returns 0, `FinalizedBlock()` returns 0, `SyncingState()` returns `Unknown`, block-availability checks fail-open, `IsBlockFinalized` returns `ErrFinalizedBlockUnavailable` immediately. Intentional opt-out for chains where polling is expensive. Source: [`architecture/evm/evm_state_poller.go:L147-160`](https://github.com/erpc/erpc/blob/main/architecture/evm/evm_state_poller.go#L147-L160) | | `upstreams[*].evm.statePollerDebounce` | Duration | `0` (defer to dynamic/fallback) | Hard per-upstream override for the debounce window. When non-zero, short-circuits the entire `resolveDebounce` algorithm — EMA path and `fallbackStatePollerDebounce` are both bypassed. Source: [`architecture/evm/evm_state_poller.go:L360-363`](https://github.com/erpc/erpc/blob/main/architecture/evm/evm_state_poller.go#L360-L363) | | `upstreams[*].evm.blockAvailability.lower.updateRate` | Duration | `0` (freeze after first detection) | Re-detection cadence for `earliestBlockPlus`-based lower bounds. `0` means the binary-search result is never refreshed within a process lifetime. **Ignored for `latestBlockMinus` and `exactBlock` bounds.** The scheduler goroutine is NOT restarted on config hot-reload. Source: [`architecture/evm/evm_state_poller.go:L694-700`](https://github.com/erpc/erpc/blob/main/architecture/evm/evm_state_poller.go#L694-L700) | | `upstreams[*].evm.blockAvailability.upper.updateRate` | Duration | `0` (freeze after first detection) | Same semantics as `lower.updateRate`. The minimum non-zero rate across lower and upper bounds for the same probe governs the scheduler goroutine. Source: [`architecture/evm/evm_state_poller.go:L694-700`](https://github.com/erpc/erpc/blob/main/architecture/evm/evm_state_poller.go#L694-L700) | #### Network-level (`networks[*].evm.*`) | Field | Type | Default | Behavior / footguns | |---|---|---|---| | `networks[*].evm.fallbackStatePollerDebounce` | Duration | `5s` | Debounce used when EMA is not yet available (step 3 of `resolveDebounce`). Setting to `0` causes the 1 s hard floor to apply during EMA warm-up. Source: [`common/defaults.go:L1975-1976`](https://github.com/erpc/erpc/blob/main/common/defaults.go#L1975-L1976) | | `networks[*].evm.fallbackFinalityDepth` | int64 | `1024` | Blocks subtracted from latest to infer finalized when the upstream does not support `eth_getBlockByNumber("finalized")`. Applied only when `finalizedBlock == 0` and `latestBlock > 0`. Source: [`common/defaults.go:L1975`](https://github.com/erpc/erpc/blob/main/common/defaults.go#L1975) | | `networks[*].evm.dynamicBlockTimeDebounceMultiplier` | \*float64 | `0.7` | Scales the EMA block time to compute the debounce. **`0` is silently treated as unset** — the default 0.7 is used. Valid useful range: `0.1`–`1.0`; values above `1.0` make the debounce longer than one block time, causing stale data between polls. To eliminate the EMA-derived debounce, use `statePollerDebounce` on the upstream instead. Source: [`architecture/evm/evm_state_poller.go:L368-372`](https://github.com/erpc/erpc/blob/main/architecture/evm/evm_state_poller.go#L368-L372) | | `networks[*].evm.finalizedDebounceDepthMultiplier` | \*float64 | `0.05` | Finalized polls wait until the upstream's finalized block is older than `(latest − finalized) × EMA block time × multiplier`, when that is longer than the regular debounce. `0` disables it: finalized uses the regular debounce, as before. Also throttles the forced finalized re-check of `blockAvailability` with finalized confidence. Source: [`common/defaults.go:L2446`](https://github.com/erpc/erpc/blob/main/common/defaults.go#L2446) | `networks[*].evm.servedTip.*` keys (`quorum`, `maxRegressionBlocks`, `guaranteedMethods`, `guaranteedFor`) and the removed keys with their migration: [Networks config](/config/projects/networks.llms.txt#config-schema). ### Worked examples **1. Majority quorum for a five-provider Mainnet pool.** The default quorum serves the 2nd-highest head; `majority` serves the 3rd of 5, so three providers already have every advertised block and fewer interpolated requests land on a node without it: **Config path:** `networks[].evm` **YAML — `erpc.yaml`:** ```yaml networks: - architecture: evm evm: chainId: 1 servedTip: quorum: majority ``` **TypeScript — `erpc.ts`:** ```typescript networks: [{ architecture: "evm", evm: { chainId: 1, servedTip: { quorum: "majority" }, }, }] ``` **2. Faster polling for a sub-second chain (Arbitrum).** The EMA auto-tunes the debounce to ~70% of block time, but `fallbackStatePollerDebounce` should be shortened so warm-up doesn't bottleneck fresh starts: **Config path:** `networks[].evm` **YAML — `erpc.yaml`:** ```yaml networks: - architecture: evm evm: chainId: 42161 fallbackStatePollerDebounce: 200ms ``` **TypeScript — `erpc.ts`:** ```typescript networks: [{ architecture: "evm", evm: { chainId: 42161, fallbackStatePollerDebounce: "200ms", }, }] ``` **3. Guaranteed method floor for a mixed trace/non-trace fleet.** Some upstreams support `trace_*`; others do not. Without `guaranteedMethods`, the served tip could advance to a block no trace-capable upstream has yet: **Config path:** `networks[].evm.servedTip` **YAML — `erpc.yaml`:** ```yaml networks: - architecture: evm evm: chainId: 1 servedTip: guaranteedMethods: ["trace_*"] ``` **TypeScript — `erpc.ts`:** ```typescript networks: [{ architecture: "evm", evm: { chainId: 1, servedTip: { guaranteedMethods: ["trace_*"], }, }, }] ``` **4. Keep the advertised tip servable by the group consensus requires.** A mixed pool whose own nodes trail the third-party majority by a few blocks: `consensus.requiredParticipants` demands them in the winning group, but `"latest"` resolves to a block they do not have yet, so every request becomes a composition dispute. Name the same tag in `guaranteedFor` and the tip tracks whatever the group can actually serve — no cushion constant to size or revisit: **Config path:** `networks[].evm.servedTip` **YAML — `erpc.yaml`:** ```yaml networks: - architecture: evm evm: chainId: 1 servedTip: # the same tag consensus.requiredParticipants names guaranteedFor: ["type:internal"] ``` **TypeScript — `erpc.ts`:** ```typescript networks: [{ architecture: "evm", evm: { chainId: 1, servedTip: { // the same tag consensus.requiredParticipants names guaranteedFor: ["type:internal"], }, }, }] ``` **5. Disable the poller for a static archive upstream.** When an archive node is used only for historical queries and tracking chain head is unnecessary or costly: **Config path:** `upstreams[].evm` **YAML — `erpc.yaml`:** ```yaml upstreams: - id: archive-node evm: statePollerInterval: 0 ``` **TypeScript — `erpc.ts`:** ```typescript upstreams: [{ id: "archive-node", evm: { statePollerInterval: 0 }, }] ``` ### Request/response behavior - A block-tagged request for `"latest"` / `"finalized"` is interpolated to the served tip, which at least `quorum` voters already have. Routing does not exclude upstreams below it; per-upstream block-availability checks and retries handle a node that lacks the block. - `ErrFinalizedBlockUnavailable` is retryable (N:yes). Requests are automatically retried on a different upstream. On the wire it maps to HTTP 200 with JSON-RPC error code `-32603`. - `ErrUpstreamBlockUnavailable` is produced by `CheckBlockRangeAvailability` (used for `eth_getLogs`, `trace_filter`, `arbtrace_filter`) when both block bounds are outside the upstream's availability window. Retryable at the network level. - The `SuggestLatestBlock` fast path (called when a response is observed during request forwarding) is non-blocking and uses no debounce. `SuggestFinalizedBlock` uses `TryLock` and is async — suggestions may be dropped under high concurrency. ### Best practices - Use `guaranteedMethods: ["trace_*"]` when mixing archive and non-archive upstreams — prevents the served tip from advancing past what trace-capable nodes have. - Lower `fallbackStatePollerDebounce` to 200–500 ms on sub-second chains (Arbitrum, Base, Optimism) so warm-up doesn't stall on a 5 s default. - Avoid setting `dynamicBlockTimeDebounceMultiplier: 0`; it is silently ignored and the default 0.7 is used. To impose a fixed debounce, use `statePollerDebounce` on the upstream instead. - Monitor `erpc_upstream_block_head_lag` per upstream, and alert on `erpc_network_served_tip_advance_age_seconds` (stuck tip), `erpc_network_served_tip_regression_total` and `erpc_upstream_head_benched_total` ([metrics reference](/reference/metrics.llms.txt#area-3-block-number-and-tip-tracking)). ### Edge cases & gotchas 1. **`statePollerInterval: 0` disables the poller silently.** All block numbers return 0, syncing state is `Unknown`, `IsBlockFinalized` returns `ErrFinalizedBlockUnavailable` immediately, and block availability checks fail-open. The upstream is not excluded from selection but its head is 0, so it never votes. 2. **Two upstreams with the default quorum serve the lower head.** One stalled node holds `latest` until it is benched (its stall window: 90s at the default 30s `statePollerInterval`); then the other votes alone. A node that is catching up slowly is not benched and holds `latest` at its head until `eth_syncing` reports it syncing. 3. **A rogue head needs `quorum − 1` accomplices.** One far-future or wrong-chain head never moves the pick at quorum ≥ 2. 4. **`dynamicBlockTimeDebounceMultiplier: 0` is silently treated as unset.** The guard `> 0` causes the value to fall through to the default 0.7. 5. **`eventLogs` probe requires ≥1 log entry.** A block with no events returns an empty array, treated as "not available" — the binary search may incorrectly mark data-sparse blocks as unavailable. 6. **`updateRate` goroutine is NOT restarted on config hot-reload.** Only a process restart picks up a changed `updateRate` for `earliestBlockPlus` bounds. Setting `updateRate` alongside `latestBlockMinus` produces a validation warning (not an error); the value is silently ignored at runtime. 7. **`ErrFinalizedBlockUnavailable` is retryable.** It carries no explicit opt-out flag, so `IsRetryableTowardNetwork` returns true. On the wire it maps to HTTP 200 with JSON-RPC error code `-32603`. 8. **`OnValue` callback passes `blockTimestamp=0` for cross-pod replications.** Only the replica that actually fetches feeds the EMA with a real timestamp; replicas receiving shared-counter updates use `blockTimestamp=0`, so only one pod per network feeds the EMA per debounce window. 9. **`ErrFinalizedBlockUnavailable` never fires if only `latestBlock == 0`.** The error requires BOTH counters to be 0. If the upstream supports `eth_getBlockByNumber("latest")` but not `"finalized"`, `IsBlockFinalized` falls back to `latestBlock − FallbackFinalityDepth` instead of returning the error. 10. **`SuggestFinalizedBlock` may drop updates under load.** It uses `TryLock`; if an update is already in progress the suggestion is discarded. Finalized block updates may lag under high request volume. 11. **`SuggestLatestBlock` monotonic within a process but not across pod restarts.** It calls `TryUpdate` which the shared counter rejects if the value is ≤ current. Across pod restarts the shared counter (Redis) retains the last value, so a restarted upstream reporting a lower block number after a reorg is silently ignored for up to 1024 blocks of rollback. 12. **`eth_syncing` result `{Ok: false}` is interpreted as SYNCING.** The non-standard `{Ok: bool}` response shape is handled by `!Ok`: `Ok=false` means not-ok = still syncing. Source: [`architecture/evm/evm_state_poller.go:L1087-L1100`](https://github.com/erpc/erpc/blob/main/architecture/evm/evm_state_poller.go#L1087-L1100) 13. **`guaranteedFor` on a ONE-member group lets that upstream set the tip.** The clamp is the group's majority, and a group of one has no majority to outvote a stall — a single frozen member holds the network tip at its head. Naming exactly one upstream is declaring it mandatory; name ≥3 for a group that tolerates one stall. Same property `guaranteedMethods` already has for a method only one upstream supports. 14. **Finalized block polling is permanently skipped on unsupported chains.** After 10 consecutive `eth_getBlockByNumber("finalized")` failures without any prior success, `skipFinalizedCheck = true`. `IsBlockFinalized` then falls back exclusively to `latestBlock − FallbackFinalityDepth`. On PoW chains or chains without EIP-4895 finality, this is the only reachable path. 15. **Finalized trails by up to `finalizedDebounceDepthMultiplier` of the finality window.** Response finality uses the serving upstream's own finalized block and cache hits the lowest across upstreams, so requests for blocks finalized within that span stay classified unfinalized (cache TTL, finality-scoped policies) until the first upstream polls again; the rest then catch up at the regular debounce. A `finalized` response or `SuggestFinalizedBlock` inside the window still advances it at once. ### Observability | Metric | Type | Labels | When it fires | |---|---|---|---| | `erpc_upstream_latest_block_number` | gauge | project, vendor, network, upstream | Updated on every `SetLatestBlockNumber` call; `upstream="*"` = corroborated network head (second-highest non-benched) | | `erpc_upstream_finalized_block_number` | gauge | project, vendor, network, upstream | Updated on every `SetFinalizedBlockNumber` call | | `erpc_upstream_block_head_lag` | gauge | project, vendor, network, upstream | Recomputed on each `SetLatestBlockNumber`; value = `network_latest − presumed_latest` (corroborated network head; presumed head = last report plus the network head's progress since, for at most one poll period); a benched head that has not moved since its bench counts as no higher than the network head at the bench | | `erpc_upstream_finalization_lag` | gauge | project, vendor, network, upstream | Recomputed on each `SetFinalizedBlockNumber`; value = `network_finalized − presumed_finalized` (corroborated network head, same presumed-head credit) | | `erpc_upstream_latest_block_polled_total` | counter | project, vendor, network, upstream | Incremented before each actual `eth_getBlockByNumber("latest")` call (debounce elapsed). Two paths increment this counter: (1) proactive timer-driven `Poll()` on the background goroutine, and (2) forced-fresh polls triggered by block-availability checks (`EvmAssertBlockAvailability` / `EvmIsBlockFinalized`) when a request arrives for a block above the current known latest. Both paths still respect the debounce window — if a pod already fetched within the window, `TryUpdateIfStale` returns the cached value without incrementing. | | `erpc_upstream_finalized_block_polled_total` | counter | project, vendor, network, upstream | Same dual-path semantics as `latest_block_polled_total` for the finalized axis | | `erpc_upstream_block_head_large_rollback` | gauge | project, vendor, network, upstream, finality | Fired when a new value would regress >1024 blocks. `finality` is the head axis (`latest` or `finalized`) — the two pollers would otherwise share one series | | `erpc_network_dynamic_block_time_milliseconds` | gauge | project, network | Current EMA block-time estimate in ms; absent until ≥3 samples | | `erpc_network_latest_block_timestamp_distance_seconds` | gauge | project, network, origin | `(now_ms − block.timestamp × 1000) / 1000`; `origin="evm_state_poller"` | Served-tip and bench metrics (`erpc_network_served_tip_*`, `erpc_upstream_head_benched_total`): [metrics reference](/reference/metrics.llms.txt#area-3-block-number-and-tip-tracking). **OTel trace spans** (under detailed tracing): `EvmStatePoller.PollLatestBlockNumber`, `EvmStatePoller.PollFinalizedBlockNumber`, `Network.EvmHighestLatestBlockNumber`, `Network.EvmHighestFinalizedBlockNumber`, `Evm.ExtractBlockReferenceFromRequest`, `Evm.ExtractBlockReferenceFromResponse`. The network span includes attributes: `served_tip.axis`, `served_tip.tip`, `served_tip.freshest`, `served_tip.inputs`. Source: [`erpc/networks.go:L924-931`](https://github.com/erpc/erpc/blob/main/erpc/networks.go#L924-L931) **Notable log messages:** | Level | Message | Trigger | |---|---|---| | `INFO` | `"bootstrapped evm state poller to track upstream latest/finalized blocks and syncing states"` | Successful `Bootstrap` | | `INFO` | `"node is marked as fully synced"` | `synced counter >= 4` consecutive not-syncing responses | | `WARN` | `"upstream does not support fetching syncing state ... after N consecutive failures, will give up"` | `skipSyncingCheck` flip (10 failures, no prior success) | | `WARN` | `"upstream does not support fetching latest block number ... after N consecutive failures, will give up"` | `skipLatestBlockCheck` flip | | `WARN` | `"upstream does not support fetching finalized block number ... after N consecutive failures, will give up"` | `skipFinalizedCheck` flip | | `WARN` | `"failed to poll evm state"` | `Poll()` returns a non-cancel error | | `INFO` | `"initial earliest block detection completed for this instance"` | Binary search for `earliestBlockPlus` bound completes | | `INFO` | `"started periodic scheduler for earliest block availability bound"` | `updateRate` configured and scheduler goroutine launched | | `WARN` | `"benched upstream head: it stayed behind peers that kept advancing; ..."` | Upstream benched | | `INFO` | `"upstream head rejoined: it caught up with its non-benched peers"` | Benched upstream rejoined | | `ERROR` | `"served tip regressed far below the corroborated live head; serving the last corroborated tip"` | Regression guard starts holding | | `ERROR` | `"served tip regression outlasted the guard window; failing open and serving the current pick"` | Hold exceeded 1 minute | | `WARN` | `"served tip recovered; the regression guard is no longer holding"` | Guard back to healthy | ### Source code entry points - [`architecture/evm/evm_state_poller.go:L145-L204`](https://github.com/erpc/erpc/blob/main/architecture/evm/evm_state_poller.go#L145-L204) — `Bootstrap`, `Poll`, `PollLatestBlockNumber`, `PollFinalizedBlockNumber`; debounce resolution; syncing state machine - [`common/served_tip.go:L36-L157`](https://github.com/erpc/erpc/blob/main/common/served_tip.go#L36-L157) — `ServedTipQuorum`, `PickServedTip` - [`erpc/networks.go:L430-L506`](https://github.com/erpc/erpc/blob/main/erpc/networks.go#L430-L506) — `tipVoterRank`, `evmTipBallot` - [`erpc/networks.go:L564-L655`](https://github.com/erpc/erpc/blob/main/erpc/networks.go#L564-L655) — `EvmRoutableUpstreams`, `requestSelector`, `servedTipLaneFor` - [`erpc/networks.go:L748-L795`](https://github.com/erpc/erpc/blob/main/erpc/networks.go#L748-L795) — `EvmHighestLatestBlockNumber`, `EvmRefreshLatestBlock` - [`erpc/networks.go:L870-L1102`](https://github.com/erpc/erpc/blob/main/erpc/networks.go#L870-L1102) — `servedTip`, `guardServedTipRegression` - [`erpc/networks.go:L1369-L1532`](https://github.com/erpc/erpc/blob/main/erpc/networks.go#L1369-L1532) — `EvmHighestFinalizedBlockNumber`, `applyDeliveredHeadFloor`, `guaranteedFloor`, `guaranteedSubsetTip`, `anyRegisteredLiveHead`, `observeServedTipMetrics` - [`architecture/evm/eth_blockNumber.go`](https://github.com/erpc/erpc/blob/main/architecture/evm/eth_blockNumber.go) — `eth_blockNumber` pre-forward answer and post-forward pin - [`health/tracker.go:L1428-L1664`](https://github.com/erpc/erpc/blob/main/health/tracker.go#L1428-L1664) — `corroboratedNetworkHead`, `headStallAfter`, `evaluateHeadBench`, `movedSinceBench`, `headIsBehind`, `benchPlausibleLead`, `IsHeadBenched` - [`health/tracker.go:L1666-L1918`](https://github.com/erpc/erpc/blob/main/health/tracker.go#L1666-L1918) — `SetLatestBlockNumber`, EMA update (`updateBlockTimeSample`, `sampleBlockTime`), `BlockHeadLag` computation - [`health/tracker.go:L1937-L2033`](https://github.com/erpc/erpc/blob/main/health/tracker.go#L1937-L2033) — `SetFinalizedBlockNumber`, `rederiveNetworkFinalized`, `FinalizationLag` computation - [`common/defaults.go:L1718-L1977`](https://github.com/erpc/erpc/blob/main/common/defaults.go#L1718-L1977) — `DefaultEvmFinalityDepth=1024`, `DefaultEvmStatePollerDebounce=5s`, `DefaultDynamicBlockTimeDebounceMultiplier=0.7` - [`common/served_tip_test.go`](https://github.com/erpc/erpc/blob/main/common/served_tip_test.go) — unit tests for `PickServedTip` and quorum parsing - [`erpc/networks_served_tip_test.go`](https://github.com/erpc/erpc/blob/main/erpc/networks_served_tip_test.go) — integration tests: quorum, voter ranks, benching, static caps, selector scoping, guarantee floors ### Related pages - [Selection policies](/config/projects/selection-policies.llms.txt) — uses `blockNumberLagAbove` and `blockHeadLag` produced by this subsystem to deprioritize lagging upstreams. - [Upstream config](/config/projects.llms.txt) — where `evm.statePollerInterval`, `evm.statePollerDebounce`, and `evm.blockAvailability` are set. - [Network config](/config/projects/networks.llms.txt) — where `evm.servedTip`, `evm.fallbackStatePollerDebounce`, and `evm.dynamicBlockTimeDebounceMultiplier` are set. - [Cache](/config/database.llms.txt) — uses `ExtractBlockReferenceFromRequest` / `ExtractBlockReferenceFromResponse` to build cache keys from block numbers and tags. - [eth\_getLogs splitting](/reference/evm/getlogs-splitting.llms.txt) — calls `CheckBlockRangeAvailability` from this subsystem to gate range queries. --- ## Navigation (machine-readable surface) - Up: [All pages index](https://docs.erpc.cloud/llms.txt) - Root index of every page: [llms.txt](https://docs.erpc.cloud/llms.txt) · everything in one file: [llms-full.txt](https://docs.erpc.cloud/llms-full.txt) ### Sibling pages - [getLogs auto-splitting](https://docs.erpc.cloud/reference/evm/getlogs-splitting.llms.txt) — Large eth_getLogs queries silently split into parallel chunks and merge back — no more range-limit errors from any provider, no client changes required. - [Method Handlers](https://docs.erpc.cloud/reference/evm/method-handlers.llms.txt) — eRPC's per-method EVM hooks eliminate entire classes of provider inconsistency — stale blocks, duplicate broadcasts, range-exceeded traces — before your application ever sees them.