Block tracking & served tip

How it works

Each upstream gets its own EvmStatePoller background goroutine. On each tick of statePollerInterval (default 30 s), three concurrent goroutines fetch the latest block, finalized block, and syncing state. A synchronous Poll() runs once at bootstrap time to seed the upstream before any request traffic arrives.

Remote head updates do not postpone local latest-block sampling. A local fetch can attach a missing timestamp to the same block already received from a peer. The block-time estimator accepts that matched observation without advancing the head again. Latest-block polling cost scales with replica count.

Routing the safe tag. On op-stack chains, safe is node-relative because each verifier chooses its own L1 confirmation depth. If providers in one pool use different depths, they can honestly return different blocks for the same tag. Set evm.safeBlockSource to an upstream id/tag selector when one pool should define that answer:

networks:
  - architecture: evm
    evm:
      chainId: 8453
      safeBlockSource: "tier:trusted"
 
upstreams:
  - id: node-a
    tags: ["tier:trusted"]
  - id: node-b
    tags: ["tier:trusted"]
  - id: provider-a

eRPC keeps the safe tag verbatim and routes standard block-tagged JSON-RPC requests only to matching upstreams, so the selected pool defines both block height and hash. Cache reads are skipped because an older entry may have been produced by a different source. Operator routing overrides client use-upstream; retry and consensus still run inside the selected JSON-RPC pool, so configure participant thresholds accordingly. eth_query* and direct gRPC Query APIs retain their existing behavior and are outside this option's scope. If no source is configured or inherited, routing is unchanged. If trusted sources are unavailable, prefer finalized for a provider-independent answer.

Debounce resolution. To avoid redundant RPC calls, each fetch is gated by resolveDebounce(), which picks the first applicable value from:

1. upstream.evm.statePollerDebounce — hard per-upstream override; short-circuits all other steps.
2. tracker.GetNetworkBlockTime × dynamicBlockTimeDebounceMultiplier (default 0.7) — EMA-derived; active after ≥3 samples.
3. network.evm.fallbackStatePollerDebounce (default 5 s) — used while EMA warms up.
4. Hard 1 s floor — applies only if both step 2 and 3 yield zero.

On a 12 s Ethereum Mainnet chain this yields ~8.4 s debounce — fresh enough to catch every block, conservative enough that concurrent pollers share one actual RPC call per window. In Redis-backed multi-pod deployments, the shared counter coalesces fetches across replicas too.

Dynamic block-time EMA. The EMA (alpha = 0.1, ~19-sample effective window) lives in health.Tracker and updates whenever the corroborated network-level latest block advances, using the on-chain timestamp of the upstream report that holds that head (each upstream's latest timestamp is stored beside its head, so the sample comes from whichever poller reported the block). All head publication for a network is serialized under one per-network lock, so a slow re-derivation can never overwrite a newer one. On fast chains where consecutive blocks share the same integer second, block gaps accumulate and samples are normalized. The EMA is not published until ≥3 samples are collected (requiring 4 total advances, because the first stores the baseline without incrementing the sample counter). Values outside [10 ms, 120 s] are rejected and reset to the last valid value to prevent runaway spikes after a chain halt. Consumers receive 0 until warm-up completes.

Per-upstream block counters. Both latestBlockShared and finalizedBlockShared are forward-only shared counters: TryUpdate rejects values ≤ the current value. A value that would regress more than 1024 blocks fires an OnLargeRollback callback and records erpc_upstream_block_head_large_rollback.

Network head and blockHeadLag / finalizationLag. The tracker's network-level latest and finalized heads are corroborated: the second-highest head across the reporting upstreams, or the only head while a single upstream has reported (the same order statistic PickServedTip uses for its lag reference). One upstream can therefore never move the reference every other upstream is measured against — a lone far-ahead report (an endpoint answering for another chain, a corrupted response) is not corroborated, and one stale upstream cannot hold the head back either. With exactly two reporters the head is the lower one, so a lone high report and a lone stale report are indistinguishable there; per-upstream block-availability checks still gate block-specific requests. The reporting set is the upstream that polled plus every upstream in the network's request-metrics index (an upstream that has never served a request is not in it). blockHeadLag = max(0, network_latest − upstream_latest); finalizationLag = max(0, network_finalized − upstream_finalized). Both are recomputed on every SetLatestBlockNumber / SetFinalizedBlockNumber call and exposed as Prometheus gauges; the raw per-upstream erpc_upstream_latest_block_number still shows an outlier's own report. These drive selection-policy predicates such as blockNumberLagAbove.

Served tip: default mode vs. majority mode. By default, EvmHighestLatestBlockNumber returns the corroborated head across all non-syncing, policy-eligible upstreams: the second-highest effective head, or the only one (PickServedTip(...).Freshest, the same statistic the tracker's blockHeadLag uses). One upstream can never define it. It can still advertise a block only two upstreams have seen.

When servedTip.enabledFor includes "latest" or "finalized", the pure function ComputeServedTipCandidate runs instead:

1. Drop inputs with BlockNumber ≤ 0.
2. Velocity gate (requires a prior served anchor AND a known block time): compute expectedMax = lastServedBlock + ceil(elapsedBlocks × VelocitySlack) + VelocityBufferBlocks (defaults: VelocitySlack = 2.0, VelocityBufferBlocks = 5); discard any upstream reporting a block above expectedMax. MaxEligible is set to the maximum of the remaining (post-gate) inputs, so velocity-dropped upstreams never inflate the lag gauge.
3. Sort remaining inputs ascending.
4. Greedy clustering: split into clusters when adjacent inputs differ by more than clusterDelta blocks (auto-derived from block time; 2 for Ethereum, up to 10 for sub-100 ms chains).
5. Pick the dominant cluster (largest by member count; ties go to the chain-forward cluster).
6. Return the minimum of the dominant cluster as the candidate.

The result is monotonically clamped via a cross-pod shared counter so the served tip never regresses, even across pods.

Guarantee subsets (guaranteedMethods, guaranteedFor). After the pick, the advertised tip is clamped down to the lowest majority of every configured guarantee subset, so it is never a block a required subset cannot serve. guaranteedMethods names a subset by method support; guaranteedFor names one by upstream id/tag selector — the same vocabulary as use-upstream and consensus.requiredParticipants. Both run through the same ballot and the same min-across-subsets fold in guaranteedFloor; only the membership predicate differs. Each subset is clamped to its OWN MAJORITY, never its minimum, so a single frozen member is outvoted inside its group. An empty or unmatched subset constrains nothing (the tip is never pinned to 0), and the clamp can only lower the tip — a group ahead of the network majority never raises it. Source: erpc/networks.go:L1505-L1600 (opens in a new tab)

Selector-scoped served tip. When a use-upstream selector is present, tipCandidateUpstreams narrows to the matching upstreams. A "simple group selector" is a single glob pattern (optionally negated with ! at position 0), containing no whitespace or ()|&, characters, of length ≤128, that matches 2–(N−1) upstreams. For such selectors, a per-group servedTipPartition is lazily materialized: partitionKeyFor resolves the matched upstream ID set, sorts it, SHA-256 hashes the sorted \0-joined IDs, truncates to 8 bytes, and encodes as hex (key = "grp:" + hex). Equivalent selectors (same matched set) share one partition. Up to 16 partitions are allowed per network; beyond this, extra selectors fall back to stateless cluster-min. Boolean-expression selectors (containing |, &, (, )) always use the stateless path. Source: erpc/networks.go:L466-L506 (opens in a new tab)

Syncing state machine. eth_syncing is called with a 5 s sub-context on each tick. The upstream is declared fully synced only after 4 consecutive "not syncing" responses (synced counter ≥ 4); any syncing response resets the counter to 1. The non-standard {Ok: bool} response shape is supported: Ok=false is treated as still-syncing. After 10 consecutive failures with no prior success, skipSyncingCheck is set permanently and the state is fixed at Unknown for the process lifetime. If upstreams[*].evm.skipSyncingCheck: true is set in config, eth_syncing is never called at all and the state is immediately fixed at NotSyncing — use this for nodes that always return a syncing object regardless of actual sync status.

Skip-on-failure semantics. Both skipLatestBlockCheck and skipFinalizedCheck follow the same pattern: 10 consecutive failures with no prior successful poll sets the skip flag permanently and the poller stops calling that method. Once the upstream has had at least one success (*SuccessfulOnce = true), failures are only logged and never trigger the skip. This prevents false-positive disablement during transient degradations. Source: architecture/evm/evm_state_poller.go:L420-L445 (opens in a new tab)

IsBlockFinalized decision tree:

1. Both finalizedBlock == 0 and latestBlock == 0 → ErrFinalizedBlockUnavailable.
2. finalizedBlock > 0 → blockNumber ≤ finalizedBlock.
3. finalizedBlock == 0 and latestBlock > 0 → infer finalizedBlock = latestBlock − FallbackFinalityDepth (default 1024) and compare.

Earliest block detection. When blockAvailability.lower.earliestBlockPlus is configured, binarySearchEarliest finds the first block in [0, latestBlock] where a probe RPC call returns data. Probe types: blockHeader (eth_getBlockByNumber), eventLogs (eth_getLogs, requires ≥1 log entry), callState (eth_getBalance), traceData (tries trace_block, debug_traceBlockByHash, trace_replayBlockTransactions in order). Results are stored in cross-pod shared counters. If blockAvailability.*.updateRate is set, a scheduler goroutine re-runs the search at that interval.

Config schema

Upstream-level (upstreams[*].evm.*)

Network-level (networks[*].evm.*)

Worked examples

1. Enable cluster-min served tip for a multi-provider Mainnet setup. Cluster mode ensures every provider in the dominant group has the advertised block before eRPC routes to it — eliminates "block not found" on eth_getBlockByNumber after a new block:

networks:  - architecture: evm    evm:      chainId: 1      servedTip:        enabledFor: ["latest", "finalized"]

2. Faster polling for a sub-second chain (Arbitrum). The EMA auto-tunes the debounce to ~70% of block time, but fallbackStatePollerDebounce should be shortened so warm-up doesn't bottleneck fresh starts. Reduce polling interval to keep lag tight:

networks:  - architecture: evm    evm:      chainId: 42161      fallbackStatePollerDebounce: 200ms      servedTip:        enabledFor: ["latest"]

3. Guaranteed method floor for a mixed trace/non-trace fleet. Some upstreams support trace_*; others do not. Without guaranteedMethods, the served tip could advance to a block no trace-capable upstream has yet:

networks:  - architecture: evm    evm:      chainId: 1      servedTip:        enabledFor: ["latest"]        guaranteedMethods: ["trace_*"]

4. Keep the advertised tip servable by the group consensus requires. A mixed pool whose own nodes trail the third-party majority by a few blocks: consensus.requiredParticipants demands them in the winning group, but "latest" resolves to a block they do not have yet, so every request becomes a composition dispute. Name the same tag in guaranteedFor and the tip tracks whatever the group can actually serve — no cushion constant to size or revisit:

networks:  - architecture: evm    evm:      chainId: 1      servedTip:        enabledFor: ["latest"]        # the same tag consensus.requiredParticipants names        guaranteedFor: ["type:internal"]

5. Disable the poller for a static archive upstream. When an archive node is used only for historical queries and tracking chain head is unnecessary or costly:

upstreams:  - id: archive-node    evm:      statePollerInterval: 0

Request/response behavior

• In cluster mode, eRPC gates routing so only upstreams at or above the served tip are eligible. A request for "latest" receives a block number ≤ the served tip, guaranteed to be present on the upstream that handles it.
• ErrFinalizedBlockUnavailable is retryable (N:yes). Requests are automatically retried on a different upstream. On the wire it maps to HTTP 200 with JSON-RPC error code -32603.
• ErrUpstreamBlockUnavailable is produced by CheckBlockRangeAvailability (used for eth_getLogs, trace_filter, arbtrace_filter) when both block bounds are outside the upstream's availability window. Retryable at the network level.
• erpc_network_served_tip_block_number and erpc_network_served_tip_lag_blocks are only emitted in majority mode — they are absent in default mode. Alerting rules referencing them will silently produce no data in default deployments. [erpc/networks.go:L521-527]
• The SuggestLatestBlock fast path (called when a response is observed during request forwarding) is non-blocking and uses no debounce. SuggestFinalizedBlock uses TryLock and is async — suggestions may be dropped under high concurrency.

Best practices

• Enable cluster mode (servedTip.enabledFor: ["latest", "finalized"]) in any production deployment with multiple upstreams — it eliminates the class of "block not found" failures caused by routing to a lagging node.
• Lower fallbackStatePollerDebounce to 200–500 ms on sub-second chains (Arbitrum, Base, Optimism) so warm-up doesn't stall on a 5 s default.
• Use guaranteedMethods: ["trace_*"] when mixing archive and non-archive upstreams — prevents the served tip from advancing past what trace-capable nodes have.
• Avoid setting dynamicBlockTimeDebounceMultiplier: 0; it is silently ignored and the default 0.7 is used. To impose a fixed debounce, use statePollerDebounce on the upstream instead.
• Monitor erpc_upstream_block_head_lag per upstream. A node consistently lagging more than clusterDelta blocks will be excluded from the dominant cluster and effectively deprioritized.
• Alert on erpc_network_served_tip_lag_blocks rising with a flat erpc_network_served_tip_block_number — this is the diagnostic signature of a monotonic clamp hold-down after fleet regression (all nodes temporarily reported lower blocks but the counter held the old high-water mark).
• Don't alert on erpc_network_served_tip_lag_blocks == 0 as a health signal in cluster mode — during a total outage all inputs are zero and MaxEligible - served < 0 clamps to 0, masking the problem. Cross-alert with erpc_upstream_latest_block_number being flat instead.

Edge cases & gotchas

1. statePollerInterval: 0 disables the poller silently. All block numbers return 0, syncing state is Unknown, IsBlockFinalized returns ErrFinalizedBlockUnavailable immediately, and block availability checks fail-open. The upstream is not excluded from selection but contributes BlockNumber=0, which cluster mode drops at step 1.
2. Velocity gate is inactive on cold start. If lastServedBlock == 0 (cold start) or BlockTimeSeconds == 0 (EMA not warmed up), all candidates pass to clustering. A misbehaving upstream reporting a far-future block during cold start can become the sole cluster and win.
3. Monotonic clamp survives fleet regression. Per-upstream shared counters are forward-only. A reorg or mass node reboot causing all upstreams to report lower blocks is silently ignored — the counter holds the previous high-water mark. erpc_network_served_tip_lag_blocks grows unboundedly; a flat served_tip_block_number alongside rising lag is the diagnostic signature.
4. dynamicBlockTimeDebounceMultiplier: 0 is silently treated as unset. The guard > 0 causes the value to fall through to the default 0.7.
5. "safe" is an alias for "finalized" in enabledFor. Listing "safe" enables cluster mode on the finalized axis only.
6. Selector-scoped tip is stateless for boolean-expression selectors. Selectors containing |, &, (, ), or multiple ! characters fail isSimpleGroupSelector and always use stateless cluster-min (no monotonic guarantee, no metrics).
7. Maximum 16 selector partitions per network. Beyond the cap, additional selectors fall back to stateless cluster-min. Equivalent selectors (same matched upstream set) dedup to one partition.
8. clusterDelta auto-derive uses the EMA block time. During warm-up (fewer than 3 EMA samples), blockTimeSec = 0 and clusterDelta defaults to 2. A 2-block spread between an early and late upstream will erroneously split into two clusters until the EMA warms up (typically within the first 3 produced blocks).
9. eventLogs probe requires ≥1 log entry. A block with no events returns an empty array, treated as "not available" — the binary search may incorrectly mark data-sparse blocks as unavailable.
10. updateRate goroutine is NOT restarted on config hot-reload. Only a process restart picks up a changed updateRate for earliestBlockPlus bounds. Setting updateRate alongside latestBlockMinus produces a validation warning (not an error); the value is silently ignored at runtime.
11. ErrFinalizedBlockUnavailable is retryable. It carries no explicit opt-out flag, so IsRetryableTowardNetwork returns true. On the wire it maps to HTTP 200 with JSON-RPC error code -32603.
12. erpc_network_served_tip_lag_blocks reports 0 during a total outage. When all upstreams return 0 or blocks below the clamp, MaxEligible = 0 and lag = MaxEligible − served < 0 is clamped to 0. A lag of 0 with a flat served_tip_block_number indicates total outage, not healthy operation.
13. OnValue callback passes blockTimestamp=0 for cross-pod replications. Only the replica that actually fetches feeds the EMA with a real timestamp; replicas receiving shared-counter updates use blockTimestamp=0, so only one pod per network feeds the EMA per debounce window.
14. ErrFinalizedBlockUnavailable never fires if only latestBlock == 0. The error requires BOTH counters to be 0. If the upstream supports eth_getBlockByNumber("latest") but not "finalized", IsBlockFinalized falls back to latestBlock − FallbackFinalityDepth instead of returning the error.
15. SuggestFinalizedBlock may drop updates under load. It uses TryLock; if an update is already in progress the suggestion is discarded. Finalized block updates may lag under high request volume.
16. SuggestLatestBlock monotonic within a process but not across pod restarts. It calls TryUpdate which the shared counter rejects if the value is ≤ current. Across pod restarts the shared counter (Redis) retains the last value, so a restarted upstream reporting a lower block number after a reorg is silently ignored for up to 1024 blocks of rollback.
17. eth_syncing result {Ok: false} is interpreted as SYNCING. The non-standard {Ok: bool} response shape is handled by !Ok: Ok=false means not-ok = still syncing. Source: architecture/evm/evm_state_poller.go:L1087-L1100 (opens in a new tab)
18. erpc_network_served_tip_lag_blocks uses MaxEligible (post-velocity-gate), not MaxObserved. A far-future tip from a misbehaving upstream is velocity-dropped and does not inflate the lag gauge. The gauge accurately reflects deliberate lag vs. the highest plausible eligible upstream tip.
19. guaranteedFor on a ONE-member group lets that upstream set the tip. The clamp is the group's majority, and a group of one has no majority to outvote a stall — a single frozen member holds the network tip at its head. Naming exactly one upstream is declaring it mandatory; name ≥3 for a group that tolerates one stall. Same property guaranteedMethods already has for a method only one upstream supports.
20. guaranteedFor constrains nothing when its group is absent from the eligible set. If every matching upstream is lag-excluded, cordoned or mid-restart, the group ballot has no inputs and the clamp is skipped rather than pinning the tip to 0. A group whose members are all static serving-range caps floors the tip only when no REGISTERED member is live — the same transient-vs-permanent rule guaranteedMethods applies, so an eligibility collapse cannot hand back a frozen cap. Source: erpc/networks.go:L1654-L1675 (opens in a new tab)
21. guaranteedFor is subject to enabledFor. The clamp lives in the served-tip pipeline, so it is a no-op on any axis not listed in enabledFor — setting guaranteedFor without enabledFor changes nothing.
22. Finalized block polling is permanently skipped on unsupported chains. After 10 consecutive eth_getBlockByNumber("finalized") failures without any prior success, skipFinalizedCheck = true. IsBlockFinalized then falls back exclusively to latestBlock − FallbackFinalityDepth. On PoW chains or chains without EIP-4895 finality, this is the only reachable path.

Observability

OTel trace spans (under detailed tracing): EvmStatePoller.PollLatestBlockNumber, EvmStatePoller.PollFinalizedBlockNumber, Network.EvmHighestLatestBlockNumber, Network.EvmHighestFinalizedBlockNumber, Evm.ExtractBlockReferenceFromRequest, Evm.ExtractBlockReferenceFromResponse. The network span includes attributes: served_tip.axis, served_tip.candidate, served_tip.max_observed, served_tip.cluster_count, served_tip.dominant_size, served_tip.outliers_count, served_tip.velocity_dropped, served_tip.lag_vs_max. Source: erpc/networks.go:L777-793

Notable log messages:

Source code entry points

• architecture/evm/evm_state_poller.go:L145-L204 (opens in a new tab) — Bootstrap, Poll, PollLatestBlockNumber, PollFinalizedBlockNumber; debounce resolution; syncing state machine
• architecture/evm/served_tip.go:L114-L246 (opens in a new tab) — ComputeServedTipCandidate: velocity gate, greedy clustering, dominant cluster, resolveAutoClusterDelta
• erpc/networks.go:L466-L851 (opens in a new tab) — clusteredServedTip, gatherEvmTipInputsForMethod, tipCandidateUpstreams, servedTipPartitionFor, partitionKeyFor, isSimpleGroupSelector, guaranteedMethodFloor, observeServedTipMetrics
• health/tracker.go:L1306-L1463 (opens in a new tab) — SetLatestBlockNumber, EMA update (updateBlockTimeSample), GetNetworkBlockTime, BlockHeadLag computation
• health/tracker.go:L1504-L1550 (opens in a new tab) — SetFinalizedBlockNumber, FinalizationLag computation
• common/defaults.go:L1718-L1977 (opens in a new tab) — DefaultEvmFinalityDepth=1024, DefaultEvmStatePollerDebounce=5s, DefaultDynamicBlockTimeDebounceMultiplier=0.7
• architecture/evm/served_tip_test.go (opens in a new tab) — unit tests for clustering, velocity gate, clusterDelta auto-derive
• erpc/networks_served_tip_test.go (opens in a new tab) — integration tests: cluster-min, monotonic clamp, selector scoping, partition dedup, guaranteed method floor

Related pages

• Selection policies — uses blockNumberLagAbove and blockHeadLag produced by this subsystem to deprioritize lagging upstreams.
• Upstream config — where evm.statePollerInterval, evm.statePollerDebounce, and evm.blockAvailability are set.
• Network config — where evm.servedTip, evm.fallbackStatePollerDebounce, and evm.dynamicBlockTimeDebounceMultiplier are set.
• Cache — uses ExtractBlockReferenceFromRequest / ExtractBlockReferenceFromResponse to build cache keys from block numbers and tags.
• eth_getLogs splitting — calls CheckBlockRangeAvailability from this subsystem to gate range queries.