# Block tracking & served tip > Source: https://docs.erpc.cloud/reference/evm/block-tracking > eRPC keeps a live pulse on every upstream's chain head and finality, then distills a single honest block number your clients can trust — with no "block not found" surprises. > Format: machine-readable markdown export of the docs page above. > All collapsible AI sections are inlined and fully expanded. # Block tracking & served tip Every upstream sees the chain at a slightly different block. Without coordination, routing a request to the one node that is a block behind causes silent "block not found" failures. eRPC polls every upstream continuously and advertises one **served tip** per axis (`latest`, `finalized`): the highest block at least `servedTip.quorum` (default 2) eligible upstreams already have. **What you get** - Chain head and finality tracked per upstream, refreshed automatically at the right cadence for each chain's block time - One rule for every config: a block is served only once a quorum of upstreams has it — a lone far-ahead or wrong-chain node never defines `latest` - Stalled heads benched automatically, so one frozen node cannot hold `latest` back - `finalized` never steps back while an upstream still reports the block already served, even when a lagging group of upstreams becomes the quorum; replicas sharing state serve the highest one any of them served - Optional per-group guarantee so the advertised tip stays servable by the upstreams your consensus policy requires - Per-upstream lag gauges that feed selection policies so slow nodes are deprioritized before requests reach them - Redis-backed counters share heads across replicas; each replica samples latest-block timestamps on its own debounce schedule. ## Quick taste Illustrative, not a tuned production config — require a strict majority of upstreams instead of the default 2: **Config path:** `projects[].networks[].evm.servedTip` **YAML — `erpc.yaml`:** ```yaml projects: - id: main networks: - architecture: evm evm: chainId: 1 servedTip: # serve the highest block a strict majority of upstreams has quorum: majority ``` **TypeScript — `erpc.ts`:** ```typescript projects: [{ id: "main", networks: [{ architecture: "evm", evm: { chainId: 1, servedTip: { // serve the highest block a strict majority of upstreams has quorum: "majority", }, }, }], }] ``` ## Agent reference Copy one of these prompts into your AI agent session (Claude Code, Cursor, …) — each one points the agent at this page's machine-readable reference so it can do the work correctly: **Prompt Example #1: eliminate block-not-found errors across my upstream fleet** ```text My app gets occasional "block not found" errors right after a new block is produced because some upstreams lag behind. Set servedTip.quorum in my eRPC config so eRPC only advertises a block number enough upstreams actually have, and explain when a stalled upstream gets benched out of the ballot. Read the full reference first: https://docs.erpc.cloud/reference/evm/block-tracking.llms.txt ``` **Prompt Example #2: tune block polling for a fast chain like Arbitrum** ```text I'm running eRPC against Arbitrum (250ms block time) and want block tracking tight from the first second after startup. Adjust the polling and debounce settings in my eRPC config so the EMA warms up quickly and block tracking stays tight. Reference: https://docs.erpc.cloud/reference/evm/block-tracking.llms.txt ``` **Prompt Example #3: set up guaranteed-method floor for a mixed trace fleet** ```text My eRPC pool mixes trace-capable archive nodes with cheaper non-archive upstreams. Without a floor, the served tip can advance to a block that only the non-archive nodes have, causing trace_block requests to hit "block not found". Configure guaranteedMethods in my eRPC config and explain how the method floor interacts with the quorum. Reference: https://docs.erpc.cloud/reference/evm/block-tracking.llms.txt ``` --- ### Block tracking & served tip — full agent reference ### How it works Each upstream gets its own `EvmStatePoller` background goroutine. On each tick of `statePollerInterval` (default 30 s), three concurrent goroutines fetch the latest block, finalized block, and syncing state. A synchronous `Poll()` runs once at bootstrap time to seed the upstream before any request traffic arrives. Remote head updates do not postpone local latest-block sampling. A local fetch can attach a missing timestamp to the same block already received from a peer. The block-time estimator accepts that matched observation without advancing the head again. Latest-block polling cost scales with replica count. **Routing the `safe` tag.** On op-stack chains, `safe` is node-relative because each verifier chooses its own L1 confirmation depth. If providers in one pool use different depths, they can honestly return different blocks for the same tag. Set `evm.safeBlockSource` to an upstream id/tag selector when one pool should define that answer: ```yaml networks: - architecture: evm evm: chainId: 8453 safeBlockSource: "tier:trusted" upstreams: - id: node-a tags: ["tier:trusted"] - id: node-b tags: ["tier:trusted"] - id: provider-a ``` eRPC keeps the `safe` tag verbatim and routes standard block-tagged JSON-RPC requests only to matching upstreams, so the selected pool defines both block height and hash. Cache reads are skipped because an older entry may have been produced by a different source. Operator routing overrides client `use-upstream`; retry and consensus still run inside the selected JSON-RPC pool, so configure participant thresholds accordingly. `eth_query*` and direct gRPC Query APIs retain their existing behavior and are outside this option's scope. If no source is configured or inherited, routing is unchanged. If trusted sources are unavailable, prefer `finalized` for a provider-independent answer. **Debounce resolution.** To avoid redundant RPC calls, each fetch is gated by `resolveDebounce()`, which picks the first applicable value from: 1. `upstream.evm.statePollerDebounce` — hard per-upstream override; short-circuits all other steps. 2. `tracker.GetNetworkBlockTime × dynamicBlockTimeDebounceMultiplier` (default 0.7) — set by the first block-time sample, which needs 2 timestamp changes after the first observed header. 3. `network.evm.fallbackStatePollerDebounce` (no default) — when set, used until step 2 applies. 4. The 1 s cold block time × the multiplier (0.7 s) — until step 2 applies. On a 12 s Ethereum Mainnet chain this yields ~8.4 s debounce — fresh enough to catch every block, conservative enough that concurrent pollers share one actual RPC call per window. In Redis-backed multi-pod deployments, the shared counter coalesces fetches across replicas too. **Finalized debounce.** The finalized fetch widens that debounce to `(latest − finalized) × EMA block time × finalizedDebounceDepthMultiplier` (default 0.05) when this is longer, using the upstream's own latest and finalized blocks. Finalized advances once per finality window, so polling it at the latest-block cadence mostly returns the number already held. With the default the finalized block trails by at most about 5% of the window: ~43 s on Ethereum Mainnet (~72 blocks × 12 s), ~45 s on an Arbitrum-style L2 with a 15 min window. An upstream behind the network's corroborated finalized block (second-highest across upstreams) keeps the regular debounce until it catches up: once a peer sees the next finalized block, the others sample it at the regular cadence. This also covers vendors whose load balancer mixes stale and fresh backends. Chains with near-instant finality (depth of a few blocks) keep the regular debounce. Unknown finalized, latest at or below finalized, or a cold EMA also keep it. Source: [`architecture/evm/evm_state_poller.go:L449-471`](https://github.com/erpc/erpc/blob/main/architecture/evm/evm_state_poller.go#L449-L471) **Head fetch back-off.** An upstream's head fetches back off after polls in a row that brought nothing: node failures (a transport error, a timeout, a 5xx, a 429, a 401 or 403), and answers that brought no head above the one the poller held as the fetch started, while the tracker [benched](#benching) the upstream. Each one doubles the wait after the last fetch's send, from the poll cadence up to 60 s (`evm.AskMaxBackoff`). An answer ends the failure count; an answer with a higher head, or any answer while the upstream is not benched, ends the idle count, and the idle count holds nothing back once the bench clears. A benched node that moves slower than the cadence also waits between its blocks, up to about twice its block interval. While it waits, an Ask fetches nothing and answers `AskBackoff` (counted as `"idle"` in `erpc_head_ask_total` when only the idle count holds it back, else `"backoff"`), the block clock gives the upstream no turn, and, while the idle count holds it back, the finalized poll waits at least as long. The bench reads the longer cadence too, so the stall window and the window in which a slow poll is not counted as lag grow with it: a frozen node that still answers is polled about once per 60 s, and a node that starts to move again is seen at most one wait later. A halted chain benches no one, so its upstreams keep their cadence. Source: [`architecture/evm/evm_state_poller.go:L1023-1041`](https://github.com/erpc/erpc/blob/main/architecture/evm/evm_state_poller.go#L1023-L1041) **Dynamic block-time EMA.** The EMA (`alpha = 0.1`, ~19-sample effective window) lives in `health.Tracker` and updates whenever the corroborated network-level latest block advances, using the on-chain timestamp of the upstream report that holds that head (each upstream's latest timestamp is stored beside its head, so the sample comes from whichever poller reported the block). All head publication for a network is serialized under one per-network lock, so a slow re-derivation can never overwrite a newer one. Source headers (pushes and poll-source polls) sample it too, once the network's confirmed head reaches them, so a lagging or bogus header is no sample. A sample spans two observations whose timestamps differ: seconds elapsed over blocks elapsed; blocks that share a timestamp extend the span. Spans start at the first timestamp change after the first observation, so the first sample needs 2 timestamp changes after it, and seeds the EMA. A sample above 3× the EMA counts as 3× the EMA, so one span across a chain halt never sets W to its 60 s cap; values outside `[10 ms, 120 s]` are rejected. `GetNetworkBlockTime` returns 0 until the first sample; the chain tracker's W and the poll cadence use the 1 s cold block time meanwhile (poll cadence 0.7 s). **Per-upstream block counters.** Both `latestBlockShared` and `finalizedBlockShared` are forward-only shared counters: `TryUpdate` rejects values ≤ the current value. A value more than 1024 blocks lower applies only once the lower side moves: a later reading above the first lower one, with no reading near the counter in between. Applied, it fires an `OnLargeRollback` callback and records `erpc_upstream_block_head_large_rollback`. A provider that sometimes answers from a node stuck far behind repeats one value, so its drops never reach `latest` or `finalized`; a real deep reorg or a chain restarted lower applies one reading later, and lower readings alone for 5 minutes apply too. **Network head and `blockHeadLag` / `finalizationLag`.** The tracker's network-level latest and finalized heads are *corroborated*: the second-highest head across the non-benched reporting upstreams, or the only head while a single one has reported (the same order statistic as the served tip's default quorum). When every reporter is benched it falls back to all of them. One upstream can therefore never move the reference every other upstream is measured against — a lone far-ahead report (an endpoint answering for another chain, a corrupted response) is not corroborated, and a benched (stalled) upstream cannot hold the head back. With exactly two reporters the head is the lower one, so a lone high report and a lone stale report are indistinguishable there; per-upstream block-availability checks still gate block-specific requests. The reporting set is every upstream that has reported a latest or finalized head to the tracker. `blockHeadLag = max(0, network_latest − presumed_latest)`; `finalizationLag = max(0, network_finalized − presumed_finalized)`. An upstream's presumed head is its last reported head plus what the network head made since that report, counted for at most one poll period (`max(statePollerInterval, statePollerDebounce or the block time)`): within the period its lag stays what it was at the report, because a stored head ages between polls by design. Without this, an upstream polled every 10s on a 0.4s chain read up to 25 blocks behind late in each poll window, and a selection policy ticking at the same period read the same phase on every tick, excluding a healthy upstream for good on some processes. Past the period the credit freezes, so a missing report counts as lag again; a benched head gets no credit. Both are recomputed on every `SetLatestBlockNumber` / `SetFinalizedBlockNumber` call and exposed as Prometheus gauges; the raw per-upstream `erpc_upstream_latest_block_number` still shows an outlier's own report. These drive selection-policy predicates such as `blockNumberLagAbove`. Source: [`health/tracker.go:L1468-1519`](https://github.com/erpc/erpc/blob/main/health/tracker.go#L1468-L1519) #### One view (latest) `latest` is one view per lane, `{Head, Firm}`, published by the lane's chain tracker as one pointer. HTTP reads (`eth_blockNumber`, an interpolated `latest`, cache keys) and every stream (ws `newHeads`, ws `logs`, gRPC `StreamBlocks`) read that one state, so they never show two heads. - **Head:** the highest pinned block (parent-hash linked) that at least one counted hash vote holds, undisputed, at most each guarantee subset's floor. It only moves back when the pins move (a reorg), which streams replay as a proper reorg. - **Undisputed:** no hash voter votes another branch at that height, unless `servedTip.quorum` voters on the pinned branch outnumber it: one upstream on a minority fork never holds the view back. - **Firm:** the highest block at or below `Head` that `servedTip.quorum` voters hold. It decides routing: a block above `Firm` goes to its holders first. - **Held:** a block above `Firm` that the view pins exists, so a request for it asks no upstream for its head, even where a method only some upstreams serve keeps that method's `latest` below it. Its first attempt and its one retry go to the upstream that most likely holds it soonest, by the block's first sight plus that upstream's measured hold time, and wait for it, at most W. Each answer is a new measurement, and no head fetch is sent for it. - **Block clock:** while a stream needs heads, or a `latest` read came within 2 W, the tracker spends the voters' poll turns at the times each block may arrive. A voter's `statePollerInterval` is its read budget: it takes at most one turn per block and per interval, and its full poll skips the latest fetch the turn made. The turns form a ladder across the block's arrival spread; the first sight stops it, and the turns not sent fetch at once as hash votes. With no turn due, the fastest voter fetches once per block. Budget: at most max(1, Σ min(1, block time / interval)) + 0.05 head fetches per new block. A wss `newHeads` push skips the ladder. - **Idle edge:** the first `latest` read after that fetches `servedTip.quorum` voters at once; tag reads wait until the view reaches the fetched head (at most W); `eth_blockNumber` never waits: it answers the view at once. #### Served tip Both axes but `latest`'s number, every config. `Network.EvmHead` returns one `Head` per request: `Latest` and `LatestHash` from the view above; `Finalized`, `Leader` and the ceiling from `computeHead`, when an upstream head, a bench or the eligible set changes, and every 1s. Each of these runs one pipeline: 1. **Candidates** (`tipCandidates`): the selection policy's eligible upstreams in routing order (all registered upstreams until the policy's first decision), narrowed to the request's `use-upstream` selector when present and to the upstreams that serve the request's method (`ignoreMethods`/`allowMethods`, the filter routing applies; all of them when none does). An upstream that does not serve the method never raises the block a request is pinned to. `eth_blockNumber` keeps every voter: it reports the view's `Head`. 2. **Voter rank** (`tipVoterRank`): each candidate with a head on the axis ranks live 0, benched 1, static cap 2, benched static cap 3. A static cap is a `blockAvailability.upper.exactBlock` the chain has passed: a serving range, not a head observation (`latestBlockMinus` upstreams are live). A candidate that is syncing per `eth_syncing`, has no state poller, has no head yet or is on a minority fork casts no number vote. 3. **Ballot** (`evmTipBallot`): only the heads of the best voter rank present vote. A benched head votes only when no non-benched live head exists; a static cap only when no live head exists. One voter set (`voterSet`) applies these rules for the ballots, `Leader`, the refresh, catch-up and band Asks, the hash voters and topic admission; a hash vote keeps a voter on a minority fork, and topic admission also keeps a poller with no head yet. 4. **Pick** (`PickServedTip`): the highest block at least `servedTip.quorum` voters already have — the q-th highest head, or the lowest when fewer than q vote. 5. The [regression guard](#regression-guard), then the [guarantee floors](#guarantee-subsets), then the [finalized floor](#finalized-floor) (`finalized` only). The result is the value served, recorded on the request's [lane](#selector-scoped-served-tip) and [exported as gauges](/reference/metrics.llms.txt#area-3-block-number-and-tip-tracking). The network-wide lane is also evaluated every second, both axes, so its gauges, advance age and guard memory stay current on a network with little traffic; use-upstream lanes are evaluated only by their requests. Only the network-wide [finalized floor](#finalized-floor) is shared; nothing is predicted. Source: [`erpc/networks.go:L870-942`](https://github.com/erpc/erpc/blob/main/erpc/networks.go#L870-L942), [`erpc/networks.go:L430-506`](https://github.com/erpc/erpc/blob/main/erpc/networks.go#L430-L506), [`common/served_tip.go:L112-157`](https://github.com/erpc/erpc/blob/main/common/served_tip.go#L112-L157), [`erpc/networks_served_tip_refresh.go:L9-42`](https://github.com/erpc/erpc/blob/main/erpc/networks_served_tip_refresh.go#L9-L42) **Quorum.** `servedTip.quorum` is an integer ≥ 1 or `"majority"` (`floor(N/2)+1` of N voters), capped at N ([config](/config/projects/networks.llms.txt#config-schema)). Which head is served (heads sorted descending): | Voters N | `quorum: 2` (default) | `quorum: majority` | `quorum: 1` | |---|---|---|---| | 1 | 1st | 1st | 1st | | 2 | 2nd | 2nd | 1st | | 3 | 2nd | 2nd | 1st | | 4 | 2nd | 3rd | 1st | | 5 | 2nd | 3rd | 1st | `quorum: 1` serves the highest head, so a single upstream defines `latest`. A higher quorum trades freshness for servability: more upstreams already have the advertised block. Source: [`common/served_tip.go:L36-56`](https://github.com/erpc/erpc/blob/main/common/served_tip.go#L36-L56) #### Benching The health tracker re-evaluates every reporter on each latest-head report (`evaluateHeadBench`). An upstream is *behind* while a non-benched peer proves the chain went on without it (`headIsBehind`): - **By height:** a peer that moved after the upstream last moved is higher by no more than a plausible lead: 1024 blocks plus twice the blocks the chain produced since the upstream's head was current (by its last change and its head timestamp). Chain progress uses the network block time, else the peer's own, so a process that starts mid-stall still sees the stall. One such peer is enough. Each move of the upstream resets this rule. - **By staleness:** a peer that moved within its own or the upstream's stall window is higher within the plausible lead, and its newest block (never counted as newer than now) is at least the stall window younger than the upstream's head block. This holds while the upstream moves, so it catches a node replaying far behind the tip: after a restore, a restart or a deep rollback. One such peer is enough. Only the upstream's own head timestamp proves it stale: a head stored without one (a peer pod's poll through shared state) keeps the last verdict only while a live peer stays a stall window of blocks ahead. A node at the tip has a head block as young as any honest peer's, so a node that trails by less than the stall window of chain time is never benched. - **By block age:** a peer that moved after the upstream last moved has a newest block at least the stall window younger than the upstream's head block, whatever the heights. This catches a head frozen above a chain that restarted lower (a dead fork), a static far-ahead head, and an endpoint that serves another, slower chain under the same chain id. It takes two such peers while two exist. Behind without a break for its stall window (`headStallAfter`), `max(30s, 3 × max(statePollerInterval, statePollerDebounce or the block time))`, it is benched: its voter rank drops to benched and it leaves the lag reference. Its `blockHeadLag` keeps growing so `blockNumberLagAbove` routes around it: while its head has not moved since the bench (`movedSinceBench`), the head counts as no higher than the network head at the bench (a head frozen above a dead fork lags by what the network made since, even after the network passes its height); once it moves again, it lags by its real distance. It rejoins once it has moved since being benched (a move in the evaluation that benched it does not count), no rule above still proves it behind, and its head reached the corroborated non-benched head (second-highest, or the only one); or when no non-benched peer has a head. A voter at or above that head never lowers `latest`, and a lone far-ahead head cannot keep it out. A bench or rejoin also re-derives the finalized lag reference. Stall windows run on the monotonic clock, so a wall-clock step benches nobody. Each bench increments `erpc_upstream_head_benched_total` and logs a WARN. Bench state is per process. Source: [`health/tracker.go:L1463-1653`](https://github.com/erpc/erpc/blob/main/health/tracker.go#L1463-L1653) - **Nobody is benched** for a chain halt itself (no peer moves; a node already a stall window behind when the chain halted is benched like any trailing node), by a moving wrong-chain head above (its lead is implausible), or by one wrong-chain peer whose blocks are younger (the block-age rule needs two). - **Boundary:** a node that trails by less than the stall window of chain time (for example, 1-3 blocks on a 2s chain) is never benched. It votes with its lower head, which holds the pick back only when it is needed for the quorum (N=2 with the default quorum). `quorum: 1` serves the highest head instead. - **Boundary:** eRPC compares heights and block times, never block hashes, so it cannot tell a live fork from the canonical chain. During a halt, a node that moves within the plausible lead benches the halted ones after their stall window, and then defines `latest` alone. A benched dead-fork node that starts producing blocks again rejoins once its head reaches the corroborated head. - **Boundary:** the block-age rule cannot tell two heads frozen above one moving peer (a dead fork the minority left) from two halted honest nodes and one wrong-chain peer. It keeps the frozen pair, so a dead fork that most upstreams stayed on still defines `latest`. - **Boundary:** two chains under one chain id (an endpoint pointed at the wrong chain) differ by far more than any plausible lead, so only block age tells them apart. Until the slower chain goes twice its stall window without a block, quorum 2 over the pair serves the lower head; then the slower chain is benched, right or wrong. A wrong chain that makes blocks more often than that is never benched. Exclude such an endpoint in config. - **Boundary:** benching a voter that sat between the pick and a slower voter lowers `latest` to that slower voter's head, as any upstream leaving the ballot does. The regression guard holds only drops beyond `maxRegressionBlocks`; `finalized` holds at the [finalized floor](#finalized-floor). - **Boundary:** a `latestBlockMinus: N` upstream rejoins on its raw head but votes N below it, so its rejoin can lower `latest` by up to N. - **Boundary:** rejoin needs the benched head to reach the bar at one report. A node that always trails the second-highest head by a block or two rejoins late; until then it serves traffic while its lag stays under the policy threshold, but does not vote. - **Boundary:** a bench changes the ballots at once, but routing leaves the benched upstream at the selection policy's next evaluation (`evalInterval`, 15s by default). Until then a request for the new `latest` can reach it; without a configured availability bound nothing gates it, so its missing-data error is retried on another upstream. #### Frozen vs benched A **frozen** upstream is a benched one that still answers: it holds every block up to its own finalized one, so the selection policy's [held ranking](/config/projects/selection-policies.llms.txt) lets it serve client reads of those finalized block numbers while the lag rule keeps it out of everything else. `Tracker.IsHeadFrozen` reads three facts from atomics each time it is called, so it never waits for another evaluation: - **Benched** (above). - **Head unchanged for its stall window:** at least `headStallAfter` since its stored head last changed. `evaluateHeadBench` stores `headChangedAt + headStallAfter` for every reporter, so a frozen node that moves again stops being frozen at once and needs a full stall window more. - **Still answers:** its last head report is at most twice its fetch cadence old (the backed-off cadence `headStallAfter` uses). Every answered local `eth_getBlockByNumber("latest")` poll reports, also when the number did not change, so each instance sees it on its own polls. | | Benched | Frozen | |---|---|---| | Head | stayed behind peers that kept moving | benched, and unchanged for its stall window | | Answers polls | any (a dead node is benched too) | a report within 2 × fetch cadence | | Served-tip ballots, lag reference | left out | left out | | Tip ranking (`blockNumberLagAbove`) | excluded by its lag | excluded by its lag | | Client reads of finalized block numbers | never | by the held ranking, up to its own finalized block | - **Boundary:** a node whose polls fail, are refused or time out stops being frozen two fetch cadences after its last answer; until then it may still be tried, and its failure is retried on the next upstream. - **Boundary:** a push (WebSocket) source reports only new heads; a stalled one reports nothing, so it is not frozen. - **Boundary:** a node that moves, however slowly, is never frozen: a replay far behind the tip stays out of every request. #### Regression guard `finalized` axis only. When a pick falls more than `maxRegressionBlocks` below the corroborated evidence — the second-highest live head, or the last live-corroborated pick — the lane serves its last corroborated pick instead, never above the highest live head (`guardServedTipRegression`). Unset (`0`), the tolerance is 1024 widened to twice the lane's typical finality step: the lower median of its last eight finalized advances, once two were seen. A chain that finalizes in batches (base: median 1,914 blocks a step) thus does not hold when one voter lags a batch; an explicit value is used as configured. It holds for at most 1 minute, then fails open and serves the pick as computed. In-process only; selectors without a lane get no guard; `-1` disables it. Each transition increments `erpc_network_served_tip_regression_total`. `latest` needs no guard: its view moves back only when the pins move. #### Subscriber delivery ws `newHeads`, ws `logs` and gRPC `StreamBlocks` send up to the lane's view `Head`, read from the same state HTTP reads: a block streams the instant `eth_blockNumber` shows it, and never before. A block a reorg removes is replayed as removed (ws `removed: true`, gRPC resend) before the new branch. A subscription on a `use-upstream` connection reads the network-wide pins cut at its lane's `Head`. A request whose method only some upstreams serve reads its lane's view capped at the block `servedTip.quorum` of those upstreams hold or most likely hold (the block's first sight plus each upstream's measured hold time passed, at most its head plus the blocks the chain made since its last report), so it may be pinned below a block subscribers already got. An upstream that still lacks that block answers missing data, and the request's retry goes to one that holds it; `Firm` counts proven holders only. Model, reorgs and wait bounds: [WebSocket](/operation/websocket.llms.txt#how-heads-reach-subscribers). Source: [`architecture/evm/chain_tracker.go`](https://github.com/erpc/erpc/blob/main/architecture/evm/chain_tracker.go) #### eth_blockNumber With `enforceHighestBlock` on (the default) and the view known, eRPC answers `eth_blockNumber` from the view's `Head` without dispatching, scoped to the request's `use-upstream` selector; `erigon_blockNumber` likewise with no param or `"latest"`, `"finalized"` answers `Head.Finalized`, and any other param is forwarded as is, unpinned. It never waits: at the idle edge it answers the view, as old as the voters' last polls, and the edge's fetch serves the next read. An interpolated `"latest"` tag of `eth_getBlockByNumber` and its family reads the same view; at the idle edge it waits until the view reaches the fetched head, at most W. A `"finalized"` request takes `Head.Finalized` at once and waits for its finalized refresh only at its idle edge (item 15 below). The head is taken before the cache, so a cache hit is pinned to the same head. With no view yet (cold start) the request is forwarded. Every response that reaches the post-forward hook — an upstream answer or a cache hit — is pinned exactly to the head when it is known: higher values are lowered, lower values are raised (an upstream answer below it increments `erpc_upstream_stale_latest_block_total`). `enforce-highest-block=false` forwards and returns the upstream's answer unchanged. Source: [`architecture/evm/eth_blockNumber.go`](https://github.com/erpc/erpc/blob/main/architecture/evm/eth_blockNumber.go), [`erpc/networks.go`](https://github.com/erpc/erpc/blob/main/erpc/networks.go) An `eth_getBlockByNumber("latest")` that the cache misses, or that no cache holds, goes to its upstream as the `"latest"` tag, as before the chain tracker: each such read is that upstream's poll of its head, at no extra request. Upstreams whose evidence shows them behind the head (a head counter below it after the ordering credit, a newest report that found them behind the network head, and no measured hold wait passed since the head's first sight) go last; the rest go fastest first. Its header goes through the same checks as a poll's and votes, so a block a client read sights first moves the view at once. The answer is served only once the view pins it at or above the request's head, waiting at most 50 ms for the tracker to take the vote; otherwise (a lagging upstream, another branch, a header the tracker cannot verify) the request serves the view's head block, from the cache or by number from a holder, the upstream that answered included: its answer counts as no result, so it stays available to the read and its retries. A `use-upstream` directive, a consensus policy or `servedTip.followCacheHead` keeps the number. `erpc_chain_tracker_latest_read_total{outcome}` counts them: `served`, `head`, `failed`. Source: [`erpc/network_latest_read.go`](https://github.com/erpc/erpc/blob/main/erpc/network_latest_read.go) #### Cache head A network whose cache has a head-aware realtime policy (the source the anchor below reads) polls that connector's head while the network-wide lane has a reader: every eighth of a block or of the TTL, whichever is sooner (50ms–1s), each fetch allowed up to the TTL, with or without `followCacheHead`. Without a reader it fetches nothing; the connector's own head poll runs once a minute. Each new head young enough for the TTL is used once: - **The cache's own voter's poll:** when a lane voter's endpoint host:port is the connector's server (Prism's upstream), the head is that voter's poll of `latest`: the same request to the same server. It passes intake's checks and votes as the voter's own poll would, so the view moves when the cache holds a block, at most an eighth of a block after it. The block clock takes it as a push: it hedges a block only a slot after its predicted sight, since earlier the poll brings it first. Each other voter whose turn falls in that block votes once it most likely holds the block (its measured hold time, as for a held block), at most a block later, not at the poll's sight, when vendors still lack it: so a method only the vendors serve (`eth_call`) gets its `latest` quorum on the new block. - **A sighting:** with no such voter, the head tells the network-wide block clock that its block exists, and the clock fetches it at once as the block's one probe, to a voter whose turn falls in that block once that voter most likely holds it, else to the clock's fastest voter. A by-hash answer above `Latest` is the same proof, and a cache answer prefers the cache's own upstream. A later proof of a block already probed fetches nothing. - **Methods:** a gRPC BDS upstream serves only the methods its client maps (blocks, receipts, logs, traces, balances). For any other method (`eth_call`) it is no server, so that method's `latest` stops at a block the upstreams that serve it hold. #### Cache-head anchor `servedTip.followCacheHead: true` lowers the `Latest` a network-wide read takes to the newest block the network's cache serves, and keeps its `Finalized` and `Firm` at or below it. The source is the first realtime cache policy, with a TTL, on a head-aware connector (the `grpc` driver) that serves `eth_getBlockByNumber("latest")` for the network. While that connector's newest block is young enough for the policy's TTL — the [realtime age guard](/config/database/evm-json-rpc-cache.llms.txt)'s own rule, in whole seconds — the read's `Latest` stops at it and takes the view's pinned hash there: `eth_blockNumber`, `eth_getBlockByNumber("latest")` and tag interpolation name a block the cache already holds, so a client's follow-up reads of it (receipts, logs, the block by number) hit the cache instead of upstreams. Older or unknown: reads take the view's `Head`. Source: [`architecture/evm/cache_head_anchor.go`](https://github.com/erpc/erpc/blob/main/architecture/evm/cache_head_anchor.go), `anchorHead` in [`erpc/network_head.go`](https://github.com/erpc/erpc/blob/main/erpc/network_head.go) - **Only lowers:** `min(Head, cache head)`, so a cache ahead of the upstreams changes nothing. - **Never steps back:** a fresh cache head below the `Latest` reads last took holds `Latest` there, for at most one TTL. A cache that does not catch up within its bound (one reporting another chain, say) stops anchoring until it goes stale or reaches that value, so it cannot freeze `Latest`. A reorg or walk-back the view makes lowers `Latest` as it does without the anchor. - **Read Head only:** the view, its pins, intake, hash votes and the refusal ceiling stay on the tracker's own head, and ws `newHeads`, ws `logs`, gRPC `StreamBlocks` and taps keep sending up to the view's `Head`. So `eth_blockNumber` may sit below the newest block a subscriber got, never above it, and a glitching cache cannot delay, reorder or drop a stream's block. - **Network-wide only:** a read on a `use-upstream` selector, or of a method only some upstreams serve, takes its lane's Head as computed. - **One connector:** it assumes the connector that serves latest blocks also serves the follow-up reads. - **Poll:** the cache head's poll above. - **Signal:** `time() − erpc_cache_connector_latest_block_timestamp_seconds` is the cache head's age; an absent or stalled series means no fresh head. - **Side effect:** the age guard reads the same connector head for realtime responses without a block timestamp (`eth_getLogs`, `eth_blockNumber`), so it judges them by a freshly polled head. #### Guarantee subsets `guaranteedMethods` and `guaranteedFor` clamp the advertised tip down to the lowest majority pick of every configured guarantee subset, so it is never a block a required subset cannot serve. `guaranteedMethods` names a subset by method support; `guaranteedFor` names one by upstream id/tag selector — the same vocabulary as `use-upstream` and `consensus.requiredParticipants`. Each subset runs the same ballot (`guaranteedSubsetPick` → `evmTipBallot`, same voter ranks) at `quorum: majority` whatever the network quorum, folded to the minimum in `guaranteedFloor`; only the membership predicate differs. A majority, never a minimum, so a single frozen member is outvoted inside its group. A subset with no voters constrains nothing (the tip is never pinned to 0), and the clamp can only lower the tip — a group ahead of the network pick never raises it. A subset whose ballot holds only static caps floors the tip only when no REGISTERED member is live (`anyRegisteredLiveHead`): live members that are merely absent from the eligible set (lag-excluded, cordoned, mid-restart) must not hand back a frozen cap. Source: [`erpc/networks.go:L1411-1500`](https://github.com/erpc/erpc/blob/main/erpc/networks.go#L1411-L1500) #### Finalized floor On `finalized`, each lane serves the higher of the tip computed above and its floor: the highest `finalized` it served since an evaluation last found no voter reporting it (`holdFinalized`). Finality never reverts, so a lower tip while a voter still reports the floor means the voters changed — a leader left the ballot (excluded, benched, syncing), or a lagger rejoined a lone leader, and a lagging group became the quorum — not the chain. The floor holds only while a voter of the lane's ballot, and of every constraining guarantee subset, reports a finalized block at or above it, and only a drop of at most `maxRegressionBlocks` (0 = 1024, also when the guard is off): a deeper drop is the [regression guard](#regression-guard)'s, so a far-ahead voter once served alone cannot hold it up. The first evaluation outside those bounds drops it to the computed tip, and a voter that returns later does not restore it. A held value never exceeds the lane's served `latest` of the same evaluation: above it, `finalized` serves `latest`, never less than the computed tip, since every voter's finalized is at or below its own latest. A request interpolated to the floor block can go to any upstream whose head passed it: routing checks configured bounds only, and response finality is the serving upstream's own. Per lane, computed under the lane's lock with the ballot it compares; a selector without a lane gets no floor, and `latest` has none. On the network-wide lane the floor is also shared across instances through the counter `network///finalizedBlock` (`shareFinalized`, `publishFinalized`), so a client moved to another instance does not see `finalized` step back. A computed value above the shared one raises it (a held value never publishes): the in-process copy at once, the store in the background, so the head computation never waits on it. A computed value below the shared one is served as the shared one under the same rule: at most the tolerance above the computed value, and while the same backing reaches it. Any other shared value is ignored. Only raises are published: a value the chain left behind stays in the counter until the chain passes it, unserved by the tolerance bound. Another instance's raise reaches this one's `Head` at its next evaluation, within 1s. Source: [`erpc/networks.go:L186-245`](https://github.com/erpc/erpc/blob/main/erpc/networks.go#L186-L245), [`erpc/network_head.go:L356-379`](https://github.com/erpc/erpc/blob/main/erpc/network_head.go#L356-L379) #### Selector-scoped served tip With a `use-upstream` selector, `EvmRoutableUpstreams` narrows the candidates to the matching upstreams; the pick is the same rule over that subset. `laneFor` returns the selector's lane: the network-wide lane (`lane="all"`) without a selector, a group's own lane for a simple group selector, else none. A lane holds no pick state: only its gauge label, advance-age anchor, regression-guard memory and finalized floor. A selector without a lane gets the same pick with no guard, no finalized floor and no gauges. A request whose method only some of its selector's upstreams serve (`tipScope`) gets the same stateless pick over those upstreams: its selector lane's view capped at the block `servedTip.quorum` of them hold, `Finalized` and `Leader` from their ballot. Each method's voter set is built on its first request and rebuilt when the network's upstreams or an upstream's method filter (an auto-ignore included) change, so a request resolves it with one lookup; when every upstream filters methods alike, every method uses its selector's lane. Which selectors form a lane, naming and the 16-lane cap: [Lanes](/reference/lanes.llms.txt). Source: [`erpc/networks.go:L621-807`](https://github.com/erpc/erpc/blob/main/erpc/networks.go#L621-L807) #### Syncing, finality and earliest block **Syncing state machine.** `eth_syncing` is called with a 5 s sub-context on each tick. The upstream is declared fully synced only after 4 consecutive "not syncing" responses (`synced counter ≥ 4`); any syncing response resets the counter to 1. The non-standard `{Ok: bool}` response shape is supported: `Ok=false` is treated as still-syncing. After 10 consecutive failures with no prior success, the syncing check is given up (see below) and the state is `Unknown`. If `upstreams[*].evm.skipSyncingCheck: true` is set in config, `eth_syncing` is never called at all and the state is immediately fixed at `NotSyncing` — use this for nodes that always return a syncing object regardless of actual sync status. **Give-up semantics.** Latest, finalized and the syncing check share one give-up: 10 refusals in a row with no prior success give the method up for 10 minutes (`evm.PollerGiveUpExpiry`); then it is retried once, a refused retry gives it up again, and an answer ends the give-up for good. Once the method has answered at least once, refusals are only logged. Latest and finalized count empty answers, unsupported, ignored and client errors as refusals; finalized also counts missing-data errors once the upstream's latest holds a block, since a node without the finalized tag can answer so. A missing-data error before that, or on latest, is no refusal: an upstream that starts without blocks is polled at its cadence until it holds one. The syncing check counts every error. While finalized is given up, and through its retries, finality is derived from latest. Source: [`architecture/evm/evm_state_poller.go`](https://github.com/erpc/erpc/blob/main/architecture/evm/evm_state_poller.go) (`giveUp`) **`IsBlockFinalized` decision tree** (it compares against `FinalizedBlock()`, the upstream's vote for `Head.Finalized`): 1. Both `finalizedBlock == 0` and `latestBlock == 0` → `ErrFinalizedBlockUnavailable`. 2. `finalizedBlock > 0` → `blockNumber ≤ finalizedBlock`. 3. `finalizedBlock == 0`, the upstream not yet known to lack the finalized tag → nothing is finalized: the upstream reports 0 until its tag first answers. 4. `finalizedBlock == 0`, the upstream known to lack the finalized tag (`skipFinalizedCheck`, item 14 below) and `latestBlock > 0` → infer `finalizedBlock = latestBlock − depth`, the deeper of `FallbackFinalityDepth` (default 1024) and 30 minutes of chain, and compare. The 30 minutes count at the block time floor: the shorter of the EMA and what the blocks seen since start, or since the last halt, allow, so an estimate running long at a cold start never shortens them; 1s while the block time is unknown. **Earliest block detection.** When `blockAvailability.lower.earliestBlockPlus` is configured, `binarySearchEarliest` finds the first block in `[0, latestBlock]` where a probe RPC call returns data. Probe types: `blockHeader` (`eth_getBlockByNumber`), `eventLogs` (`eth_getLogs`, requires ≥1 log entry), `callState` (`eth_getBalance`), `traceData` (tries `trace_block`, `debug_traceBlockByHash`, `trace_replayBlockTransactions` in order). Results are stored in cross-pod shared counters. If `blockAvailability.*.updateRate` is set, a scheduler goroutine re-runs the search at that interval. ### Config schema #### Upstream-level (`upstreams[*].evm.*`) | Field | Type | Default | Behavior / footguns | |---|---|---|---| | `upstreams[*].evm.statePollerInterval` | Duration | `30s` | How often `Poll()` fires. **`0` disables the poller entirely**: no background goroutine, no initial synchronous poll. Consequences: `LatestBlock()` returns 0, `FinalizedBlock()` returns 0, `SyncingState()` returns `Unknown`, block-availability checks fail-open, `IsBlockFinalized` returns `ErrFinalizedBlockUnavailable` immediately. Intentional opt-out for chains where polling is expensive. Source: [`architecture/evm/evm_state_poller.go:L147-160`](https://github.com/erpc/erpc/blob/main/architecture/evm/evm_state_poller.go#L147-L160) | | `upstreams[*].evm.statePollerDebounce` | Duration | `0` (defer to dynamic/fallback) | Hard per-upstream override for the debounce window. When non-zero, short-circuits the entire `resolveDebounce` algorithm — EMA path and `fallbackStatePollerDebounce` are both bypassed. Source: [`architecture/evm/evm_state_poller.go:L360-363`](https://github.com/erpc/erpc/blob/main/architecture/evm/evm_state_poller.go#L360-L363) | | `upstreams[*].evm.blockAvailability.lower.updateRate` | Duration | `0` (freeze after first detection) | Re-detection cadence for `earliestBlockPlus`-based lower bounds. `0` means the binary-search result is never refreshed within a process lifetime. **Ignored for `latestBlockMinus` and `exactBlock` bounds.** The scheduler goroutine is NOT restarted on config hot-reload. Source: [`architecture/evm/evm_state_poller.go:L694-700`](https://github.com/erpc/erpc/blob/main/architecture/evm/evm_state_poller.go#L694-L700) | | `upstreams[*].evm.blockAvailability.upper.updateRate` | Duration | `0` (freeze after first detection) | Same semantics as `lower.updateRate`. The minimum non-zero rate across lower and upper bounds for the same probe governs the scheduler goroutine. Source: [`architecture/evm/evm_state_poller.go:L694-700`](https://github.com/erpc/erpc/blob/main/architecture/evm/evm_state_poller.go#L694-L700) | #### Network-level (`networks[*].evm.*`) | Field | Type | Default | Behavior / footguns | |---|---|---|---| | `networks[*].evm.fallbackStatePollerDebounce` | Duration | unset | Debounce used while the block time is not yet known (step 3 of `resolveDebounce`). Unset, the 1 s cold block time × the multiplier (0.7 s) applies. Source: [`architecture/evm/evm_state_poller.go:L505-530`](https://github.com/erpc/erpc/blob/main/architecture/evm/evm_state_poller.go#L505-L530) | | `networks[*].evm.fallbackFinalityDepth` | int64 | `1024` | Blocks subtracted from latest to infer finalized when the upstream does not support `eth_getBlockByNumber("finalized")`. Applied only when `skipFinalizedCheck` is set, `finalizedBlock == 0` and `latestBlock > 0`. Source: [`common/defaults.go:L1975`](https://github.com/erpc/erpc/blob/main/common/defaults.go#L1975) | | `networks[*].evm.dynamicBlockTimeDebounceMultiplier` | \*float64 | `0.7` | Scales the EMA block time to compute the debounce. **`0` is silently treated as unset** — the default 0.7 is used. Valid useful range: `0.1`–`1.0`; values above `1.0` make the debounce longer than one block time, causing stale data between polls. To eliminate the EMA-derived debounce, use `statePollerDebounce` on the upstream instead. Source: [`architecture/evm/evm_state_poller.go:L368-372`](https://github.com/erpc/erpc/blob/main/architecture/evm/evm_state_poller.go#L368-L372) | | `networks[*].evm.finalizedDebounceDepthMultiplier` | \*float64 | `0.05` | Finalized polls wait until the upstream's finalized block is older than `(latest − finalized) × EMA block time × multiplier`, when that is longer than the regular debounce. `0` disables it: finalized uses the regular debounce, as before. Also throttles the forced finalized re-check of `blockAvailability` with finalized confidence. Source: [`common/defaults.go:L2446`](https://github.com/erpc/erpc/blob/main/common/defaults.go#L2446) | `networks[*].evm.servedTip.*` keys (`quorum`, `maxRegressionBlocks`, `guaranteedMethods`, `guaranteedFor`, `followCacheHead`) and the removed keys with their migration: [Networks config](/config/projects/networks.llms.txt#config-schema). ### Worked examples **1. Majority quorum for a five-provider Mainnet pool.** The default quorum serves the 2nd-highest head; `majority` serves the 3rd of 5, so three providers already have every advertised block and fewer interpolated requests land on a node without it: **Config path:** `networks[].evm` **YAML — `erpc.yaml`:** ```yaml networks: - architecture: evm evm: chainId: 1 servedTip: quorum: majority ``` **TypeScript — `erpc.ts`:** ```typescript networks: [{ architecture: "evm", evm: { chainId: 1, servedTip: { quorum: "majority" }, }, }] ``` **2. Faster polling for a sub-second chain (Arbitrum).** The block time auto-tunes the debounce to ~70% of block time from its first sample, 2 timestamp changes after the first header; before that the cold debounce is 0.7 s (the 1 s cold block time × 0.7). `fallbackStatePollerDebounce` shortens those first headers: **Config path:** `networks[].evm` **YAML — `erpc.yaml`:** ```yaml networks: - architecture: evm evm: chainId: 42161 fallbackStatePollerDebounce: 200ms ``` **TypeScript — `erpc.ts`:** ```typescript networks: [{ architecture: "evm", evm: { chainId: 42161, fallbackStatePollerDebounce: "200ms", }, }] ``` **3. Guaranteed method floor for a mixed trace/non-trace fleet.** Some upstreams support `trace_*`; others do not. Without `guaranteedMethods`, the served tip could advance to a block no trace-capable upstream has yet: **Config path:** `networks[].evm.servedTip` **YAML — `erpc.yaml`:** ```yaml networks: - architecture: evm evm: chainId: 1 servedTip: guaranteedMethods: ["trace_*"] ``` **TypeScript — `erpc.ts`:** ```typescript networks: [{ architecture: "evm", evm: { chainId: 1, servedTip: { guaranteedMethods: ["trace_*"], }, }, }] ``` **4. Keep the advertised tip servable by the group consensus requires.** A mixed pool whose own nodes trail the third-party majority by a few blocks: `consensus.requiredParticipants` demands them in the winning group, but `"latest"` resolves to a block they do not have yet, so every request becomes a composition dispute. Name the same tag in `guaranteedFor` and the tip tracks whatever the group can actually serve — no cushion constant to size or revisit: **Config path:** `networks[].evm.servedTip` **YAML — `erpc.yaml`:** ```yaml networks: - architecture: evm evm: chainId: 1 servedTip: # the same tag consensus.requiredParticipants names guaranteedFor: ["type:internal"] ``` **TypeScript — `erpc.ts`:** ```typescript networks: [{ architecture: "evm", evm: { chainId: 1, servedTip: { // the same tag consensus.requiredParticipants names guaranteedFor: ["type:internal"], }, }, }] ``` **5. Disable the poller for a static archive upstream.** When an archive node is used only for historical queries and tracking chain head is unnecessary or costly: **Config path:** `upstreams[].evm` **YAML — `erpc.yaml`:** ```yaml upstreams: - id: archive-node evm: statePollerInterval: 0 ``` **TypeScript — `erpc.ts`:** ```typescript upstreams: [{ id: "archive-node", evm: { statePollerInterval: 0 }, }] ``` ### Request/response behavior - A block-tagged request for `"latest"` / `"finalized"` is interpolated to the served tip, which at least `quorum` voters already have; `"finalized"` may instead be the [finalized floor](#finalized-floor), which a voter still reports finalized. Routing does not exclude upstreams below it; per-upstream block-availability checks and retries handle a node that lacks the block. - `ErrFinalizedBlockUnavailable` is retryable (N:yes). Requests are automatically retried on a different upstream. On the wire it maps to HTTP 200 with JSON-RPC error code `-32603`. - `ErrUpstreamBlockUnavailable` is produced by `CheckBlockRangeAvailability` (used for `eth_getLogs`, `trace_filter`, `arbtrace_filter`) when both block bounds are outside the upstream's availability window. Retryable at the network level. - The `SuggestLatestBlock` fast path (called when a response is observed during request forwarding) is non-blocking and uses no debounce. `SuggestFinalizedBlock` uses `TryLock` and is async — suggestions may be dropped under high concurrency. ### Best practices - Use `guaranteedMethods: ["trace_*"]` when mixing archive and non-archive upstreams — prevents the served tip from advancing past what trace-capable nodes have. - Lower `fallbackStatePollerDebounce` to 200–500 ms on sub-second chains (Arbitrum, Base, Optimism) to poll faster than the 0.7 s cold cadence during warm-up. - Avoid setting `dynamicBlockTimeDebounceMultiplier: 0`; it is silently ignored and the default 0.7 is used. To impose a fixed debounce, use `statePollerDebounce` on the upstream instead. - Monitor `erpc_upstream_block_head_lag` per upstream, and alert on `erpc_network_served_tip_advance_age_seconds` (stuck tip), `erpc_network_served_tip_regression_total` and `erpc_upstream_head_benched_total` ([metrics reference](/reference/metrics.llms.txt#area-3-block-number-and-tip-tracking)). ### Edge cases & gotchas 1. **`statePollerInterval: 0` disables the poller silently.** All block numbers return 0, syncing state is `Unknown`, `IsBlockFinalized` returns `ErrFinalizedBlockUnavailable` immediately, and block availability checks fail-open. The upstream is not excluded from selection but its head is 0, so it never votes. 2. **Two upstreams with the default quorum serve the lower head.** One stalled node holds `latest` until it is benched (its stall window: 90s at the default 30s `statePollerInterval`); then the other votes alone. A node that is catching up slowly is not benched and holds `latest` at its head until `eth_syncing` reports it syncing. 3. **A rogue head needs `quorum − 1` accomplices.** One far-future or wrong-chain head never moves the pick at quorum ≥ 2. 4. **`dynamicBlockTimeDebounceMultiplier: 0` is silently treated as unset.** The guard `> 0` causes the value to fall through to the default 0.7. 5. **`eventLogs` probe requires ≥1 log entry.** A block with no events returns an empty array, treated as "not available" — the binary search may incorrectly mark data-sparse blocks as unavailable. 6. **`updateRate` goroutine is NOT restarted on config hot-reload.** Only a process restart picks up a changed `updateRate` for `earliestBlockPlus` bounds. Setting `updateRate` alongside `latestBlockMinus` produces a validation warning (not an error); the value is silently ignored at runtime. 7. **`ErrFinalizedBlockUnavailable` is retryable.** It carries no explicit opt-out flag, so `IsRetryableTowardNetwork` returns true. On the wire it maps to HTTP 200 with JSON-RPC error code `-32603`. 8. **`OnValue` callback passes `blockTimestamp=0` for cross-pod replications.** Only the replica that actually fetches feeds the EMA with a real timestamp; replicas receiving shared-counter updates use `blockTimestamp=0`, so only one pod per network feeds the EMA per debounce window. 9. **`ErrFinalizedBlockUnavailable` never fires if only `latestBlock == 0`.** The error requires BOTH counters to be 0. An upstream with a latest block but no finalized block yet answers "not finalized"; only once it is known to lack the finalized tag (item 14) does `IsBlockFinalized` compare against `latestBlock −` the derived depth. 10. **`SuggestFinalizedBlock` may drop updates under load.** It uses `TryLock`; if an update is already in progress the suggestion is discarded. Finalized block updates may lag under high request volume. 11. **`SuggestLatestBlock` monotonic within a process but not across pod restarts.** It calls `TryUpdate` which the shared counter rejects if the value is ≤ current. Across pod restarts the shared counter (Redis) retains the last value, so a restarted upstream reporting a lower block number after a reorg is silently ignored for up to 1024 blocks of rollback. 12. **`eth_syncing` result `{Ok: false}` is interpreted as SYNCING.** The non-standard `{Ok: bool}` response shape is handled by `!Ok`: `Ok=false` means not-ok = still syncing. Source: [`architecture/evm/evm_state_poller.go:L1087-L1100`](https://github.com/erpc/erpc/blob/main/architecture/evm/evm_state_poller.go#L1087-L1100) 13. **`guaranteedFor` on a ONE-member group lets that upstream set the tip.** The clamp is the group's majority, and a group of one has no majority to outvote a stall — a single frozen member holds the network tip at its head. Naming exactly one upstream is declaring it mandatory; name ≥3 for a group that tolerates one stall. Same property `guaranteedMethods` already has for a method only one upstream supports. 14. **Finalized block polling is given up on unsupported chains.** After 10 consecutive `eth_getBlockByNumber("finalized")` refusals without any prior success, finalized is given up and retried every 10 minutes. `IsBlockFinalized` then falls back exclusively to `latestBlock −` the derived depth, also during the retries. On PoW chains or chains without EIP-4895 finality, this is the only reachable path. 15. **Finalized trails by up to `finalizedDebounceDepthMultiplier` of the finality window.** Reads and cache hits classify a block as finalized against the request's `Head.Finalized`; an upstream's answer needs that upstream's own finalized block too, and an upstream on a minority fork never makes one final. Only the full poll and a `finalized` tag request move `Head.Finalized`: a request that names the `finalized` tag (any method, any param) starts the lane's finalized refresh in the background and takes `Head.Finalized` at once. Only when fewer than the quorum's voters reported finalized within W does it wait for the refresh, at most W, before interpolation and the cache lookup. A numbered read above `Head.Finalized` polls nothing. 16. **Instances agree on network-wide `finalized` once their own voters report it.** An instance serves another's value only while one of its voters reports it finalized, so during poll lag two instances behind one load balancer can still differ. Use-upstream lanes keep a separate floor per instance. 17. **`finalized` steps back when every voter at the floor leaves at once, or by more than `maxRegressionBlocks`.** For example, one policy evaluation excludes the whole leading group: the floor drops to the pick of the voters left. A deeper drop is held by the regression guard for at most 1 minute instead. 18. **A voter within `maxRegressionBlocks` above the honest ones can hold the floor while it stays.** The floor never passes the served `latest`, so such a voter can lift `finalized` only into blocks the honest upstreams already have. ### Observability | Metric | Type | Labels | When it fires | |---|---|---|---| | `erpc_upstream_latest_block_number` | gauge | project, vendor, network, upstream | Updated on every `SetLatestBlockNumber` call; `upstream="*"` = corroborated network head (second-highest non-benched) | | `erpc_upstream_finalized_block_number` | gauge | project, vendor, network, upstream | Updated on every `SetFinalizedBlockNumber` call | | `erpc_upstream_block_head_lag` | gauge | project, vendor, network, upstream | Recomputed on each `SetLatestBlockNumber`; value = `network_latest − presumed_latest` (corroborated network head; presumed head = last report plus the network head's progress since, for at most one poll period); a benched head that has not moved since its bench counts as no higher than the network head at the bench | | `erpc_upstream_finalization_lag` | gauge | project, vendor, network, upstream | Recomputed on each `SetFinalizedBlockNumber`; value = `network_finalized − presumed_finalized` (corroborated network head, same presumed-head credit) | | `erpc_upstream_latest_block_polled_total` | counter | project, vendor, network, upstream | Incremented before each actual `eth_getBlockByNumber("latest")` call (debounce elapsed). Two paths increment this counter: (1) proactive timer-driven `Poll()` on the background goroutine, and (2) forced-fresh polls triggered by block-availability checks (`EvmAssertBlockAvailability` / `EvmIsBlockFinalized`) when a request arrives for a block above the current known latest. Both paths still respect the debounce window — if a pod already fetched within the window, `TryUpdateIfStale` returns the cached value without incrementing. | | `erpc_upstream_finalized_block_polled_total` | counter | project, vendor, network, upstream | Same dual-path semantics as `latest_block_polled_total` for the finalized axis | | `erpc_upstream_block_head_large_rollback` | gauge | project, vendor, network, upstream, finality | Fired when a new value would regress >1024 blocks. `finality` is the head axis (`latest` or `finalized`) — the two pollers would otherwise share one series | | `erpc_network_dynamic_block_time_milliseconds` | gauge | project, network | Current EMA block-time estimate in ms; absent until ≥3 samples | | `erpc_network_latest_block_timestamp_distance_seconds` | gauge | project, network, origin | `(now_ms − block.timestamp × 1000) / 1000`; `origin="evm_state_poller"` | Served-tip and bench metrics (`erpc_network_served_tip_*`, `erpc_upstream_head_benched_total`): [metrics reference](/reference/metrics.llms.txt#area-3-block-number-and-tip-tracking). **OTel trace spans** (under detailed tracing): `EvmStatePoller.Ask`, `EvmStatePoller.PollFinalizedBlockNumber`, `ChainTracker.Fetch`, `Tap.Read`, `Evm.ExtractBlockReferenceFromRequest`, `Evm.ExtractBlockReferenceFromResponse`. **Notable log messages:** | Level | Message | Trigger | |---|---|---| | `INFO` | `"bootstrapped evm state poller to track upstream latest/finalized blocks and syncing states"` | Successful `Bootstrap` | | `INFO` | `"node is marked as fully synced"` | `synced counter >= 4` consecutive not-syncing responses | | `WARN` | `"upstream refused eth_syncing in evm state poller N times with no success, giving it up"` | syncing check given up (10 refusals, no prior success) | | `WARN` | `"upstream refused latest in evm state poller N times with no success, giving it up"` | latest given up | | `WARN` | `"upstream refused finalized in evm state poller N times with no success, giving it up"` | finalized given up | | `WARN` | `"failed to poll evm state"` | `Poll()` returns a non-cancel error | | `INFO` | `"initial earliest block detection completed for this instance"` | Binary search for `earliestBlockPlus` bound completes | | `INFO` | `"started periodic scheduler for earliest block availability bound"` | `updateRate` configured and scheduler goroutine launched | | `WARN` | `"benched upstream head: it stayed behind peers that kept advancing; ..."` | Upstream benched | | `INFO` | `"upstream head rejoined: it caught up with its non-benched peers"` | Benched upstream rejoined | | `ERROR` | `"served tip regressed far below the corroborated live head; serving the last corroborated tip"` | Regression guard starts holding | | `ERROR` | `"served tip regression outlasted the guard window; failing open and serving the current pick"` | Hold exceeded 1 minute | | `WARN` | `"served tip recovered; the regression guard is no longer holding"` | Guard back to healthy | ### Source code entry points - [`architecture/evm/evm_state_poller.go`](https://github.com/erpc/erpc/blob/main/architecture/evm/evm_state_poller.go) — `Bootstrap`, `Poll`, `Ask`, `PollLatestBlockNumber`, `PollFinalizedBlockNumber`; debounce resolution; syncing state machine - [`common/served_tip.go:L36-L157`](https://github.com/erpc/erpc/blob/main/common/served_tip.go#L36-L157) — `ServedTipQuorum`, `PickServedTip` - [`erpc/networks.go`](https://github.com/erpc/erpc/blob/main/erpc/networks.go) — `evmTipBallot`, `EvmRoutableUpstreams`, `requestSelector`, `laneFor`, `EvmAskHead`, `guardServedTipRegression`, `holdFinalized`, `guaranteedFloor`, `guaranteedSubsetPick`, `anyRegisteredLiveHead`, `bindLaneGauges` - [`erpc/network_intake.go`](https://github.com/erpc/erpc/blob/main/erpc/network_intake.go) — `voterSet`, `tipVoterRank` - [`erpc/network_head.go`](https://github.com/erpc/erpc/blob/main/erpc/network_head.go) — `EvmHead`, `EvmLaneHead`, `computeHead`, `markLanes`, `refreshDirtyHeads` - [`architecture/evm/eth_blockNumber.go`](https://github.com/erpc/erpc/blob/main/architecture/evm/eth_blockNumber.go) — `eth_blockNumber` pre-forward answer and post-forward pin - [`health/tracker.go:L1428-L1664`](https://github.com/erpc/erpc/blob/main/health/tracker.go#L1428-L1664) — `corroboratedNetworkHead`, `headStallAfter`, `evaluateHeadBench`, `movedSinceBench`, `headIsBehind`, `benchPlausibleLead`, `IsHeadBenched`, `IsHeadFrozen` - [`health/tracker.go:L1666-L1918`](https://github.com/erpc/erpc/blob/main/health/tracker.go#L1666-L1918) — `SetLatestBlockNumber`, EMA update (`updateBlockTimeSample`, `sampleBlockTime`), `BlockHeadLag` computation - [`health/tracker.go:L1937-L2033`](https://github.com/erpc/erpc/blob/main/health/tracker.go#L1937-L2033) — `SetFinalizedBlockNumber`, `rederiveNetworkFinalized`, `FinalizationLag` computation - [`common/defaults.go:L1718-L1977`](https://github.com/erpc/erpc/blob/main/common/defaults.go#L1718-L1977) — `DefaultEvmFinalityDepth=1024`, `DefaultDynamicBlockTimeDebounceMultiplier=0.7` - [`common/served_tip_test.go`](https://github.com/erpc/erpc/blob/main/common/served_tip_test.go) — unit tests for `PickServedTip` and quorum parsing - [`erpc/networks_served_tip_test.go`](https://github.com/erpc/erpc/blob/main/erpc/networks_served_tip_test.go) — integration tests: quorum, voter ranks, benching, static caps, selector scoping, guarantee floors, finalized floor ### Related pages - [Selection policies](/config/projects/selection-policies.llms.txt) — uses `blockNumberLagAbove` and `blockHeadLag` produced by this subsystem to deprioritize lagging upstreams. - [Upstream config](/config/projects.llms.txt) — where `evm.statePollerInterval`, `evm.statePollerDebounce`, and `evm.blockAvailability` are set. - [Network config](/config/projects/networks.llms.txt) — where `evm.servedTip`, `evm.fallbackStatePollerDebounce`, and `evm.dynamicBlockTimeDebounceMultiplier` are set. - [Cache](/config/database.llms.txt) — uses `ExtractBlockReferenceFromRequest` / `ExtractBlockReferenceFromResponse` to build cache keys from block numbers and tags. - [eth\_getLogs splitting](/reference/evm/getlogs-splitting.llms.txt) — calls `CheckBlockRangeAvailability` from this subsystem to gate range queries. --- ## Navigation (machine-readable surface) - Up: [All pages index](https://docs.erpc.cloud/llms.txt) - Root index of every page: [llms.txt](https://docs.erpc.cloud/llms.txt) · everything in one file: [llms-full.txt](https://docs.erpc.cloud/llms-full.txt) ### Sibling pages - [getLogs auto-splitting](https://docs.erpc.cloud/reference/evm/getlogs-splitting.llms.txt) — Large eth_getLogs queries silently split into parallel chunks and merge back — no more range-limit errors from any provider, no client changes required. - [Method Handlers](https://docs.erpc.cloud/reference/evm/method-handlers.llms.txt) — eRPC's per-method EVM hooks eliminate entire classes of provider inconsistency — stale blocks, duplicate broadcasts, range-exceeded traces — before your application ever sees them.