Block tracking & served tip

How it works

Each upstream gets its own EvmStatePoller background goroutine. On each tick of statePollerInterval (default 30 s), three concurrent goroutines fetch the latest block, finalized block, and syncing state. A synchronous Poll() runs once at bootstrap time to seed the upstream before any request traffic arrives.

Debounce resolution. To avoid redundant RPC calls, each fetch is gated by resolveDebounce(), which picks the first applicable value from:

1. upstream.evm.statePollerDebounce — hard per-upstream override; short-circuits all other steps.
2. tracker.GetNetworkBlockTime × dynamicBlockTimeDebounceMultiplier (default 0.7) — EMA-derived; active after ≥3 samples.
3. network.evm.fallbackStatePollerDebounce (default 5 s) — used while EMA warms up.
4. Hard 1 s floor — applies only if both step 2 and 3 yield zero.

On a 12 s Ethereum Mainnet chain this yields ~8.4 s debounce — fresh enough to catch every block, conservative enough that concurrent pollers share one actual RPC call per window. In Redis-backed multi-pod deployments, the shared counter coalesces fetches across replicas too.

Dynamic block-time EMA. The EMA (alpha = 0.1, ~19-sample effective window) lives in health.Tracker and updates whenever an upstream advances the network-level latest block. On fast chains where consecutive blocks share the same integer second, block gaps accumulate and samples are normalized. The EMA is not published until ≥3 samples are collected (requiring 4 total SetLatestBlockNumber calls, because the first call stores the baseline without incrementing the sample counter). Values outside [10 ms, 120 s] are rejected and reset to the last valid value to prevent runaway spikes after a chain halt. Consumers receive 0 until warm-up completes.

Per-upstream block counters. Both latestBlockShared and finalizedBlockShared are forward-only shared counters: TryUpdate rejects values ≤ the current value. A value that would regress more than 1024 blocks fires an OnLargeRollback callback and records erpc_upstream_block_head_large_rollback.

blockHeadLag and finalizationLag. blockHeadLag = max_network_latest − upstream_latest; finalizationLag = max_network_finalized − upstream_finalized. Both are block-number deltas recomputed on every SetLatestBlockNumber / SetFinalizedBlockNumber call and exposed as Prometheus gauges. A value of 0 means the upstream is at or ahead of the network maximum. These drive selection-policy predicates such as blockNumberLagAbove.

Served tip: max mode vs. cluster mode. By default (max mode), EvmHighestLatestBlockNumber returns the simple maximum across all non-syncing, policy-eligible upstreams. This is fast but can advertise a block only one upstream has seen.

When servedTip.enabledFor includes "latest" or "finalized", the pure function ComputeServedTipCandidate runs instead:

1. Drop inputs with BlockNumber ≤ 0.
2. Velocity gate (requires a prior served anchor AND a known block time): compute expectedMax = lastServedBlock + ceil(elapsedBlocks × VelocitySlack) + VelocityBufferBlocks (defaults: VelocitySlack = 2.0, VelocityBufferBlocks = 5); discard any upstream reporting a block above expectedMax. MaxEligible is set to the maximum of the remaining (post-gate) inputs, so velocity-dropped upstreams never inflate the lag gauge.
3. Sort remaining inputs ascending.
4. Greedy clustering: split into clusters when adjacent inputs differ by more than clusterDelta blocks (auto-derived from block time; 2 for Ethereum, up to 10 for sub-100 ms chains).
5. Pick the dominant cluster (largest by member count; ties go to the chain-forward cluster).
6. Return the minimum of the dominant cluster as the candidate.

The result is monotonically clamped via a cross-pod shared counter so the served tip never regresses, even across pods.

Selector-scoped served tip. When a use-upstream selector is present, tipCandidateUpstreams narrows to the matching upstreams. A "simple group selector" is a single glob pattern (optionally negated with ! at position 0), containing no whitespace or ()|&, characters, of length ≤128, that matches 2–(N−1) upstreams. For such selectors, a per-group servedTipPartition is lazily materialized: partitionKeyFor resolves the matched upstream ID set, sorts it, SHA-256 hashes the sorted \0-joined IDs, truncates to 8 bytes, and encodes as hex (key = "grp:" + hex). Equivalent selectors (same matched set) share one partition. Up to 16 partitions are allowed per network; beyond this, extra selectors fall back to stateless cluster-min. Boolean-expression selectors (containing |, &, (, )) always use the stateless path. Source: erpc/networks.go:L466-L506 (opens in a new tab)

Syncing state machine. eth_syncing is called with a 5 s sub-context on each tick. The upstream is declared fully synced only after 4 consecutive "not syncing" responses (synced counter ≥ 4); any syncing response resets the counter to 1. The non-standard {Ok: bool} response shape is supported: Ok=false is treated as still-syncing. After 10 consecutive failures with no prior success, skipSyncingCheck is set permanently and the state is fixed at Unknown for the process lifetime.

Skip-on-failure semantics. Both skipLatestBlockCheck and skipFinalizedCheck follow the same pattern: 10 consecutive failures with no prior successful poll sets the skip flag permanently and the poller stops calling that method. Once the upstream has had at least one success (*SuccessfulOnce = true), failures are only logged and never trigger the skip. This prevents false-positive disablement during transient degradations. Source: architecture/evm/evm_state_poller.go:L420-L445 (opens in a new tab)

IsBlockFinalized decision tree:

1. Both finalizedBlock == 0 and latestBlock == 0 → ErrFinalizedBlockUnavailable.
2. finalizedBlock > 0 → blockNumber ≤ finalizedBlock.
3. finalizedBlock == 0 and latestBlock > 0 → infer finalizedBlock = latestBlock − FallbackFinalityDepth (default 1024) and compare.

Earliest block detection. When blockAvailability.lower.earliestBlockPlus is configured, binarySearchEarliest finds the first block in [0, latestBlock] where a probe RPC call returns data. Probe types: blockHeader (eth_getBlockByNumber), eventLogs (eth_getLogs, requires ≥1 log entry), callState (eth_getBalance), traceData (tries trace_block, debug_traceBlockByHash, trace_replayBlockTransactions in order). Results are stored in cross-pod shared counters. If blockAvailability.*.updateRate is set, a scheduler goroutine re-runs the search at that interval.

Config schema

Upstream-level (upstreams[*].evm.*)

Network-level (networks[*].evm.*)

Worked examples

1. Enable cluster-min served tip for a multi-provider Mainnet setup. Cluster mode ensures every provider in the dominant group has the advertised block before eRPC routes to it — eliminates "block not found" on eth_getBlockByNumber after a new block:

networks:  - architecture: evm    evm:      chainId: 1      servedTip:        enabledFor: ["latest", "finalized"]

2. Faster polling for a sub-second chain (Arbitrum). The EMA auto-tunes the debounce to ~70% of block time, but fallbackStatePollerDebounce should be shortened so warm-up doesn't bottleneck fresh starts. Reduce polling interval to keep lag tight:

networks:  - architecture: evm    evm:      chainId: 42161      fallbackStatePollerDebounce: 200ms      servedTip:        enabledFor: ["latest"]

3. Guaranteed method floor for a mixed trace/non-trace fleet. Some upstreams support trace_*; others do not. Without guaranteedMethods, the served tip could advance to a block no trace-capable upstream has yet:

networks:  - architecture: evm    evm:      chainId: 1      servedTip:        enabledFor: ["latest"]        guaranteedMethods: ["trace_*"]

4. Disable the poller for a static archive upstream. When an archive node is used only for historical queries and tracking chain head is unnecessary or costly:

upstreams:  - id: archive-node    evm:      statePollerInterval: 0

Request/response behavior

• In cluster mode, eRPC gates routing so only upstreams at or above the served tip are eligible. A request for "latest" receives a block number ≤ the served tip, guaranteed to be present on the upstream that handles it.
• ErrFinalizedBlockUnavailable is retryable (N:yes). Requests are automatically retried on a different upstream. On the wire it maps to HTTP 200 with JSON-RPC error code -32603.
• ErrUpstreamBlockUnavailable is produced by CheckBlockRangeAvailability (used for eth_getLogs, trace_filter, arbtrace_filter) when both block bounds are outside the upstream's availability window. Retryable at the network level.
• erpc_network_served_tip_block_number and erpc_network_served_tip_lag_blocks are only emitted in cluster mode — they are absent in max mode. Alerting rules referencing them will silently produce no data in default deployments. [erpc/networks.go:L521-527]
• The SuggestLatestBlock fast path (called when a response is observed during request forwarding) is non-blocking and uses no debounce. SuggestFinalizedBlock uses TryLock and is async — suggestions may be dropped under high concurrency.

Best practices

• Enable cluster mode (servedTip.enabledFor: ["latest", "finalized"]) in any production deployment with multiple upstreams — it eliminates the class of "block not found" failures caused by routing to a lagging node.
• Lower fallbackStatePollerDebounce to 200–500 ms on sub-second chains (Arbitrum, Base, Optimism) so warm-up doesn't stall on a 5 s default.
• Use guaranteedMethods: ["trace_*"] when mixing archive and non-archive upstreams — prevents the served tip from advancing past what trace-capable nodes have.
• Avoid setting dynamicBlockTimeDebounceMultiplier: 0; it is silently ignored and the default 0.7 is used. To impose a fixed debounce, use statePollerDebounce on the upstream instead.
• Monitor erpc_upstream_block_head_lag per upstream. A node consistently lagging more than clusterDelta blocks will be excluded from the dominant cluster and effectively deprioritized.
• Alert on erpc_network_served_tip_lag_blocks rising with a flat erpc_network_served_tip_block_number — this is the diagnostic signature of a monotonic clamp hold-down after fleet regression (all nodes temporarily reported lower blocks but the counter held the old high-water mark).
• Don't alert on erpc_network_served_tip_lag_blocks == 0 as a health signal in cluster mode — during a total outage all inputs are zero and MaxEligible - served < 0 clamps to 0, masking the problem. Cross-alert with erpc_upstream_latest_block_number being flat instead.

Edge cases & gotchas

1. statePollerInterval: 0 disables the poller silently. All block numbers return 0, syncing state is Unknown, IsBlockFinalized returns ErrFinalizedBlockUnavailable immediately, and block availability checks fail-open. The upstream is not excluded from selection but contributes BlockNumber=0, which cluster mode drops at step 1.
2. Velocity gate is inactive on cold start. If lastServedBlock == 0 (cold start) or BlockTimeSeconds == 0 (EMA not warmed up), all candidates pass to clustering. A misbehaving upstream reporting a far-future block during cold start can become the sole cluster and win.
3. Monotonic clamp survives fleet regression. Per-upstream shared counters are forward-only. A reorg or mass node reboot causing all upstreams to report lower blocks is silently ignored — the counter holds the previous high-water mark. erpc_network_served_tip_lag_blocks grows unboundedly; a flat served_tip_block_number alongside rising lag is the diagnostic signature.
4. dynamicBlockTimeDebounceMultiplier: 0 is silently treated as unset. The guard > 0 causes the value to fall through to the default 0.7.
5. "safe" is an alias for "finalized" in enabledFor. Listing "safe" enables cluster mode on the finalized axis only.
6. Selector-scoped tip is stateless for boolean-expression selectors. Selectors containing |, &, (, ), or multiple ! characters fail isSimpleGroupSelector and always use stateless cluster-min (no monotonic guarantee, no metrics).
7. Maximum 16 selector partitions per network. Beyond the cap, additional selectors fall back to stateless cluster-min. Equivalent selectors (same matched upstream set) dedup to one partition.
8. clusterDelta auto-derive uses the EMA block time. During warm-up (fewer than 3 EMA samples), blockTimeSec = 0 and clusterDelta defaults to 2. A 2-block spread between an early and late upstream will erroneously split into two clusters until the EMA warms up (typically within the first 3 produced blocks).
9. eventLogs probe requires ≥1 log entry. A block with no events returns an empty array, treated as "not available" — the binary search may incorrectly mark data-sparse blocks as unavailable.
10. updateRate goroutine is NOT restarted on config hot-reload. Only a process restart picks up a changed updateRate for earliestBlockPlus bounds. Setting updateRate alongside latestBlockMinus produces a validation warning (not an error); the value is silently ignored at runtime.
11. ErrFinalizedBlockUnavailable is retryable. It carries no explicit opt-out flag, so IsRetryableTowardNetwork returns true. On the wire it maps to HTTP 200 with JSON-RPC error code -32603.
12. erpc_network_served_tip_lag_blocks reports 0 during a total outage. When all upstreams return 0 or blocks below the clamp, MaxEligible = 0 and lag = MaxEligible − served < 0 is clamped to 0. A lag of 0 with a flat served_tip_block_number indicates total outage, not healthy operation.
13. OnValue callback passes blockTimestamp=0 for cross-pod replications. Only the replica that actually fetches feeds the EMA with a real timestamp; replicas receiving shared-counter updates use blockTimestamp=0, so only one pod per network feeds the EMA per debounce window.
14. ErrFinalizedBlockUnavailable never fires if only latestBlock == 0. The error requires BOTH counters to be 0. If the upstream supports eth_getBlockByNumber("latest") but not "finalized", IsBlockFinalized falls back to latestBlock − FallbackFinalityDepth instead of returning the error.
15. SuggestFinalizedBlock may drop updates under load. It uses TryLock; if an update is already in progress the suggestion is discarded. Finalized block updates may lag under high request volume.
16. SuggestLatestBlock monotonic within a process but not across pod restarts. It calls TryUpdate which the shared counter rejects if the value is ≤ current. Across pod restarts the shared counter (Redis) retains the last value, so a restarted upstream reporting a lower block number after a reorg is silently ignored for up to 1024 blocks of rollback.
17. eth_syncing result {Ok: false} is interpreted as SYNCING. The non-standard {Ok: bool} response shape is handled by !Ok: Ok=false means not-ok = still syncing. Source: architecture/evm/evm_state_poller.go:L1087-L1100 (opens in a new tab)
18. erpc_network_served_tip_lag_blocks uses MaxEligible (post-velocity-gate), not MaxObserved. A far-future tip from a misbehaving upstream is velocity-dropped and does not inflate the lag gauge. The gauge accurately reflects deliberate lag vs. the highest plausible eligible upstream tip.
19. Finalized block polling is permanently skipped on unsupported chains. After 10 consecutive eth_getBlockByNumber("finalized") failures without any prior success, skipFinalizedCheck = true. IsBlockFinalized then falls back exclusively to latestBlock − FallbackFinalityDepth. On PoW chains or chains without EIP-4895 finality, this is the only reachable path.

Observability

OTel trace spans (under detailed tracing): EvmStatePoller.PollLatestBlockNumber, EvmStatePoller.PollFinalizedBlockNumber, Network.EvmHighestLatestBlockNumber, Network.EvmHighestFinalizedBlockNumber, Evm.ExtractBlockReferenceFromRequest, Evm.ExtractBlockReferenceFromResponse. The network span includes attributes: served_tip.axis, served_tip.candidate, served_tip.max_observed, served_tip.cluster_count, served_tip.dominant_size, served_tip.outliers_count, served_tip.velocity_dropped, served_tip.lag_vs_max. Source: erpc/networks.go:L777-793

Notable log messages:

Source code entry points

• architecture/evm/evm_state_poller.go:L145-L204 (opens in a new tab) — Bootstrap, Poll, PollLatestBlockNumber, PollFinalizedBlockNumber; debounce resolution; syncing state machine
• architecture/evm/served_tip.go:L114-L246 (opens in a new tab) — ComputeServedTipCandidate: velocity gate, greedy clustering, dominant cluster, resolveAutoClusterDelta
• erpc/networks.go:L466-L851 (opens in a new tab) — clusteredServedTip, gatherEvmTipInputsForMethod, tipCandidateUpstreams, servedTipPartitionFor, partitionKeyFor, isSimpleGroupSelector, guaranteedMethodFloor, observeServedTipMetrics
• health/tracker.go:L1306-L1463 (opens in a new tab) — SetLatestBlockNumber, EMA update (updateBlockTimeSample), GetNetworkBlockTime, BlockHeadLag computation
• health/tracker.go:L1504-L1550 (opens in a new tab) — SetFinalizedBlockNumber, FinalizationLag computation
• common/defaults.go:L1718-L1977 (opens in a new tab) — DefaultEvmFinalityDepth=1024, DefaultEvmStatePollerDebounce=5s, DefaultDynamicBlockTimeDebounceMultiplier=0.7
• architecture/evm/served_tip_test.go (opens in a new tab) — unit tests for clustering, velocity gate, clusterDelta auto-derive
• erpc/networks_served_tip_test.go (opens in a new tab) — integration tests: cluster-min, monotonic clamp, selector scoping, partition dedup, guaranteed method floor

Related pages

• Selection policies — uses blockNumberLagAbove and blockHeadLag produced by this subsystem to deprioritize lagging upstreams.
• Upstream config — where evm.statePollerInterval, evm.statePollerDebounce, and evm.blockAvailability are set.
• Network config — where evm.servedTip, evm.fallbackStatePollerDebounce, and evm.dynamicBlockTimeDebounceMultiplier are set.
• Cache — uses ExtractBlockReferenceFromRequest / ExtractBlockReferenceFromResponse to build cache keys from block numbers and tags.
• eth_getLogs splitting — calls CheckBlockRangeAvailability from this subsystem to gate range queries.