Monitoring & metrics

How it works

The metrics HTTP server binds :<port> (all interfaces) and registers promhttp.Handler() as the root handler. Because there is no path routing, every URL path on the metrics port — GET /, GET /metrics, GET /health — returns the identical full Prometheus text exposition. There is no TLS, no auth, and no gzip on the metrics endpoint.

All 79 counters and 23 gauges register eagerly at package-init time via promauto. The 17 LabeledHistogram instances (request-duration, cache-operation durations, consensus duration, etc.) register lazily during erpc.Init after the label-filter and bucket config has been applied. Changing histogramDropLabels after init panics — Prometheus disallows changing a registered metric's label set.

An idle-eviction sweep runs every ~5 minutes and calls DeleteLabelValues on stale erpc_upstream_request_duration_seconds and erpc_rate_limits_total series to prevent cardinality runaway under method-flood attacks. Counters emitted through telemetry.CounterHandle (upstream request/error totals, network success/hedge counters, etc.) are swept on the same cadence with their own, more conservative threshold — metrics.counterIdleEvictionAfter, default 24h — so counter series whose label combinations go quiet (stale method × user × agent_name tuples) are eventually released instead of being re-emitted on every scrape for the process lifetime. A swept series that becomes active again restarts at zero, the same semantics as a process restart. Remaining high-cardinality metrics are bounded only by distinct (project, network, upstream, method) tuples.

LabeledHistogram label filtering. WithLabelValues always receives the full canonical label set in schema order; the wrapper projects down to the active (post-filter) subset before forwarding to the underlying prometheus.HistogramVec. Call sites never need to know which labels survived. Length mismatches panic immediately to surface wiring bugs. ActiveLabelValues returns the projected subset so handle-caches key on the effective labels, preventing duplicate cache entries when multiple full-label tuples collapse to the same series.

Idle-series eviction detail. Sweep fires every 10 rotation ticks of rotateMetricsLoop (default tick = 30s, so sweep ≈ every 5 min). sweepIdle walks upsMetrics and ntwMetrics sync.Maps and deletes entries not accessed in the last 30 min. For upstream_request_duration_seconds and rate_limits_total it additionally calls DeleteLabelValues on the Prometheus registry. Cordoned entries and wildcard ("*") rollups are never evicted.

errorLabelMode global. compact mode produces short stable codes like ErrEndpointCapacityExceeded or ErrUpstreamRequest/ErrJsonRpcExceptionInternal/-32000. verbose mode emits the full message chain, potentially including block numbers or IP addresses — unbounded cardinality risk. Default before erpc.Init runs is verbose; Init sets it to the config value (default compact). Series created before Init carry verbose labels and will coexist until they expire naturally.

compact label output in full: StandardError → string(be.Base().Code) (e.g. ErrEndpointCapacityExceeded); for ErrFailsafeRetryExceeded/ErrUpstreamRequest/ErrUpstreamRequestSkipped appends cause code (ErrUpstreamRequest/ErrEndpointTransportFailure); for ErrJsonRpcExceptionInternal cause appends numeric code (ErrUpstreamRequest/ErrJsonRpcExceptionInternal/-32000). context.DeadlineExceeded → ContextDeadlineExceeded. context.Canceled → ContextCanceled. Plain error → GenericError. Unknown → UnknownError. Source: common/errors.go:L46-66.

Handle caching. CounterHandle, GaugeHandle, and ObserverHandle in telemetry/handles.go (opens in a new tab) cache prometheus.Counter/Gauge/Observer children in sync.Maps keyed by {Vec pointer, '\x1f'-joined label values}. This avoids a per-observation map lookup and mutex contention inside the Prometheus library. For LabeledHistogram observers the key uses post-filter label values so the cache is filter-aware. ResetHandleCache wipes all three maps and is called after SetHistogramBuckets when Vecs are re-created. Source: telemetry/handles.go:L86-105.

registerOrReuse idempotency. Calling SetHistogramBuckets twice with the same bucket string and the same label filter is idempotent: the second call returns the already-registered LabeledHistogram silently and the Prometheus registry is not touched again. If the label set differs (filter changed after first registration) it panics. Source: telemetry/metrics.go:L978-995.

networkAlias resolver. At startup erpc.Init installs a common.NetworkAliasResolver callback that maps raw EVM chain IDs to human-readable network aliases. Components that only know a numeric chainId (e.g. the gRPC cache connector) use this resolver so their metric network labels match the alias used by every other metric in the system. Source: erpc/init.go:L62-77.

Config schema

All fields under metrics. in the YAML root config. Struct: common/config.go:L2543-2564. Defaults: common/defaults.go:L749-767. Validation: common/validation.go:L131-161.

Hardcoded server constants (not configurable):

• Bind address: ":<port>" (all interfaces) — erpc/init.go:L149
• Handler: promhttp.Handler() registered as root; every URL path returns full metrics output — erpc/init.go:L150
• Protocol: plain HTTP only — no TLS, no auth, no gzip
• ReadHeaderTimeout: 10 seconds
• Graceful shutdown budget: 5 seconds — erpc/init.go:L162-168

Histogram bucket constants — only DefaultHistogramBuckets are configurable via metrics.histogramBuckets. All others are hard-coded:

Source: telemetry/metrics.go:L731-932.

Worked examples

All patterns below are distilled from real production fleets; comments explain the non-obvious choices.

1. Single-region deployment — compact labels, tighter buckets. Used by edge and data-indexing deployments where latency spans ~5ms cache hits through ~10s heavy getLogs. The default [0.05, 0.5, 5, 30] buckets are too coarse for sub-100ms percentile analysis, so buckets are customised to match observed p50–p99:

metrics:  enabled: true  # default port; Prometheus scrapes http://<host>:4001/metrics  port: 4001  # ALWAYS compact in production — verbose embeds block numbers and IPs in labels,  # creating unbounded cardinality that crashes managed Prometheus scrapers  errorLabelMode: compact  # buckets tuned to observed p50~5ms, p99~3s for an indexing fleet;  # adjust for your workload — coarser is fine for purely human dashboards  histogramBuckets: "0.010,0.030,0.050,0.100,0.250,0.500,1,3,5,10,30"

2. Multi-region aggregator — cardinality reduction with surgical label restore. Used when many API keys (users) flow through one node and Grafana Cloud caps scrape body size. Drop user and composite globally; restore user only on network_request_duration_seconds to keep per-client latency SLOs without multiplying every other histogram's cardinality. Coarser buckets (8 boundaries) halve bucket count vs. the 11-boundary indexer shape above:

metrics:  enabled: true  port: 4001  errorLabelMode: compact  # 8 boundaries tuned for p50~100ms, p99~6s on a multi-region aggregator  histogramBuckets: "0.020,0.100,0.300,0.500,1,3,6,10"  histogramDropLabels:    # user × composite × other labels can create thousands of histogram series;    # dropping these two globally is the single biggest cardinality win    - user    - composite  histogramLabelOverrides:    # restore user ONLY here so per-client latency SLOs remain queryable    network_request_duration_seconds:      - user

3. Edge node with fast p50 — sub-10ms granularity for cache-hit visibility. When the cache hit rate is high, most responses return in under 5ms. The default 50ms lower bucket obscures all cache-hit latency in a single bucket. Tighter low-end boundaries make p50/p75 meaningful:

metrics:  enabled: true  port: 4001  errorLabelMode: compact  # fine sub-10ms buckets for cache-hit p50 visibility; top end covers heavy getLogs  histogramBuckets: "0.005,0.050,0.500,1,3,10,30"

4. Disabling metrics entirely for test or embedded use. Set ERPC_NOMETRICS=1 before process start to replace the default Prometheus registry with an empty no-op registry — all erpc_* metrics and the stock go_*/process_* collectors are silenced. Alternatively set enabled: false to skip the HTTP server while keeping in-process counters alive (useful for integration tests that read metrics programmatically).

5. Scrape config and alerting. The bundled files at monitoring/prometheus/ (opens in a new tab) set scrape_interval: 10s targeting host.docker.internal:4001. Bring up the full Grafana + Prometheus stack with docker compose up grafana prometheus from the repo root. Six bundled alert rules fire on: high upstream error rate (>5% for 5 min), p95 latency > 1s, high/low request rate, and rate-limit events — but two rules reference stale metric names (erpc_upstream_request_self_rate_limited_total, erpc_network_request_self_rate_limited_total); replace them with erpc_rate_limits_total.

Best practices

• Always set errorLabelMode: compact in production. verbose mode can embed block numbers and IP addresses in label values, creating unbounded cardinality that crashes managed Prometheus scrapers.
• Start with histogramDropLabels: [user, agent_name] if your deployment has many API keys or clients — these two labels multiply every LabeledHistogram's bucket count by the number of distinct values.
• Managed Prometheus scrapers (Grafana Cloud, hosted Prometheus) typically cap the /metrics response body at 10–64 MB, and the cap is usually enforced on the response whole: past it the scrape is rejected outright, so every series for that instance goes dark rather than degrading. Counters — not histograms — are normally the bulk of a large payload, because an unbounded label like a client-supplied user-agent is re-emitted on every scrape for the life of the process. Reach for counterDropLabels (and counterIdleEvictionAfter) before raising the scraper's limit; the label set is unbounded and will re-cross any new ceiling.
• counterDropLabels is the counter-side twin of histogramDropLabels, with one important difference: counter labels are more likely to be load-bearing downstream. agent_name or user on a request counter often feeds billing or per-tenant attribution, so check consumers before dropping and use counterLabelOverrides to spare the specific counters that need it.
• Never set histogramDropLabels after startup — the Prometheus registry rejects label-set changes and the process panics. Plan your drop list before first deploy.
• Use histogramLabelOverrides to surgically restore a dropped label on exactly one histogram (e.g. keep user only on network_request_duration_seconds for per-client SLO analysis).
• Tighten histogramBuckets to your actual observed p50–p99 range (e.g. "0.01,0.05,0.1,0.5,1,2,5") — the default [0.05, 0.5, 5, 30] is intentionally coarse for broad compatibility, not precision.
• Do not alert on erpc_network_hedge_delay_seconds — it is registered but dormant (no production Observe call). Use erpc_network_hedged_request_total and erpc_network_request_duration_seconds for hedge-related alerting instead.

Key metrics by scenario

Is eRPC healthy right now?

Upstream performance and availability

Cache effectiveness

Rate limiting and traffic shaping

Consensus and resilience

Block tracking and tip

Solana (SVM) upstreams

SVM upstreams feed the same tracker as EVM, so the gauges below populate — but the metric names say "block" while the values are slots. There is no separate erpc_svm_* family.

Ingestion lag (maxShredInsertSlot − processedSlot) has no gauge — it is logged and drives cordoning only. See SVM slot tracking & health.

Request multiplexing and static responses

Upstream selection and probing

gRPC BDS resilience

Shadow testing

CORS

Edge cases & gotchas

1. hostV4/hostV6 are unused. The server always binds all interfaces (":<port>"). Setting metrics.hostV4: 127.0.0.1 does NOT restrict scrape access to loopback. Use a firewall or network policy instead. (erpc/init.go:L149)

2. metrics.enabled defaults to nil (disabled) under go test. Integration tests that start a full erpc.Init will NOT spin up a metrics server unless they set Enabled: true explicitly. (common/defaults.go:L750-752)

3. errorLabelMode change does not retroactively fix accumulated series. Any series created before erpc.Init sets the mode carry verbose labels; subsequent series carry compact labels. Both appear as parallel label-value sets until the verbose series expire.

4. erpc_rate_limiter_budget_decision_total is a dead metric. Registered at startup so it appears in /metrics but has zero production call sites and always returns 0. (telemetry/metrics.go:L515-519)

5. erpc_network_hedge_delay_seconds is dormant. Registered and appears in /metrics but the only Observe call is in a test file. Do not alert on it. (telemetry/metrics.go:L813)

6. erpc_cors_requests_total project label receives the URL path, not the project ID. Values look like /myproject/evm/1 instead of myproject. Dashboard queries on CORS metrics must use path-style values. (erpc/http_server.go:L1020)

7. erpc_ristretto_cache_current_cost requires memory.emitMetrics: true. Without this flag the gauge is registered but always 0 — the collection goroutine never starts. This is the only metric requiring an explicit config opt-in to be meaningful. (data/memory.go:L71)

8. erpc_upstream_cordoned category label is the cordon scope (method string), not a request-category. "*" = wholesale cordon; a specific method string = per-method cordon. Do not join this category with category on request-accounting metrics. (health/tracker.go:L458-465)

9. erpc_auth_failed_total always has strategy="database". Only the database auth strategy records this counter. For other strategies, monitor erpc_network_failed_request_total{error=~"ErrAuthUnauthorized.*"} instead. (auth/strategy_database.go:L467-472)

10. erpc_upstream_attempt_outcome_total is_hedge and is_retry are "true"/"false" strings, not "1"/"0". Use {is_hedge="true"} in Prometheus queries. (upstream/upstream.go:L80-85)

11. Idle sweep protects only upstream_request_duration_seconds and rate_limits_total Prometheus series. All other high-cardinality metrics (e.g. upstream_request_total, network_request_duration_seconds) are never DeleteLabelValues'd — cardinality bounded only by distinct label tuples.

12. Two bundled alerting rules reference stale metric names. erpc_upstream_request_self_rate_limited_total and erpc_network_request_self_rate_limited_total no longer exist. Use erpc_rate_limits_total as the replacement. (monitoring/prometheus/alert.rules:L1)

13. erpc_upstream_block_head_large_rollback is silent for rollbacks ≤ 1024 blocks, and sticky above them. DefaultToleratedBlockHeadRollback = 1024 is the threshold. The gauge is Set to the rollback size and is never reset, so it holds that value for the life of the process: > 0 means "this process saw a rollback at some point", not "a rollback is happening now", and rate()/increase() are meaningless on it. Alert on changes(...[window]) instead. Split by the finality label — the finalized axis is dominated by upstreams whose write granularity exceeds 1024 blocks, which registers as a large rollback by construction. (architecture/evm/evm_state_poller.go:L27)

14. erpc_selection_* metrics carry a method label. If the ingress exposes many distinct RPC methods to untrusted callers, a flood of unique method names creates unbounded series. Add method to a Prometheus relabeling drop rule for untrusted deployments.

15. Changing histogramDropLabels after the first SetHistogramBuckets call panics. Re-calling SetHistogramBuckets with a different filter is intentionally not supported. Plan the drop list before first deploy. (telemetry/metrics.go:L942-950)

16. erpc_network_evm_block_range_requested_total uses a dynamic tip-relative bucket label. When tip is known, ComputeBlockHeatmapBucket produces human-readable labels ("TIP", "L100k", "100k-200k", "119m-120m", etc.) — potentially unbounded. When tip is unknown it falls back to static 100k-aligned labels. Plan Prometheus cardinality accordingly. (erpc/block_heatmap.go:L87)

17. ERPC_NOMETRICS=1 drops the default go_*/process_* collectors too. It replaces the entire default registry with a fresh empty one before any promauto init fires. (cmd/erpc/initflags.go:L22-28)

18. erpc_upstream_cordoned vendor label is "n/a" when vendor is not configured. Ensure dashboard JOINs between cordoned metrics and upstream request metrics match "n/a" for unvendored upstreams. (upstream/upstream.go:L275-286)

19. Idle sweep fires at rotation-tick granularity, not in real-time. With default 5-minute window and 10 rolling buckets, each tick is 30 seconds; sweep fires every 10 ticks ≈ every 5 minutes. An idle series is evicted up to idleEvictionAfter + 5 min (≈35 min) after its last observation, not exactly at 30 minutes. (health/tracker.go:L557-563)

20. Cordoned upstreams are never swept. If an upstream is cordoned, its tracker entry is preserved regardless of idle time. The erpc_upstream_cordoned{...} = 1 gauge remains in /metrics until the upstream is uncordoned and subsequently stays idle for idleEvictionAfter. (health/tracker.go:L603-608)

21. ParseHistogramBuckets silently sorts the input. If buckets are provided out of order they are sorted automatically; no warning is emitted. The result is valid Prometheus buckets, but it may differ from the intended order if you relied on a specific sequence. (telemetry/metrics.go:L997-1015)

22. erpc_network_dynamic_block_time_milliseconds is bounded [10ms, 120s] and requires min 3 samples. The EMA (α=0.1) needs at least 3 block-time samples (4 total block observations) before the gauge emits. Chains faster than 10ms report the 10ms floor; chains with no new blocks for >2 min report the 120s ceiling. Returns 0 during startup before enough samples accumulate. (health/tracker.go:L1381-1458)

23. erpc_cors_preflight_requests_total and erpc_cors_disallowed_origin_total share the same project mislabeling as erpc_cors_requests_total. All three CORS counters set project to r.URL.Path (e.g. /myproject/evm/1), not the project ID. (erpc/http_server.go:L1069)

24. erpc_upstream_request_duration_seconds full schema includes user but erpc_upstream_response_size_bytes does not. The response-size histogram is intentionally tight (project, network, category, finality only) — per-user breakdown of response size is not actionable. Do not add user to histogramDropLabels if per-user latency analysis on upstream_request_duration_seconds is required. (telemetry/metrics.go:L914-920)

25. The bundled Prometheus scrape config contains a Railway placeholder. monitoring/prometheus/prometheus.yml includes a REPLACE_SERVICE_ENDPOINT_HERE:REPLACE_SERVICE_PORT_HERE target that causes parse errors in strict Prometheus configs. Remove or replace it before using the bundled config in production.

26. erpc_grpc_bds_hard_timeout_total threshold is hard-coded at 20 seconds. The bdsHardCallTimeout constant at clients/grpc_bds_resilience.go:L34 is not configurable without recompile. A non-zero rate indicates H2 stream wedging; the watchdog then force-replaces the connection (erpc_grpc_bds_conn_replacements_total).

27. On SVM networks, finality="finalized" catch-up retries are normal, not waste. The EVM guidance — that catch-up on canonical data indicates a misconfiguration — does not transfer. Solana consensus finality (a rooted slot) and RPC indexing are independent, so getBlock on a just-rooted slot legitimately returns -32004 BlockNotAvailable or -32014 BlockStatusNotAvailableYet until the provider's indexer catches up. Alert on the trend, not on non-zero. See SVM slot tracking & health.

28. erpc_upstream_*_block_number and *_lag gauges carry slots on SVM networks. The names are architecture-neutral and were coined for EVM. A dashboard that labels them "blocks" will mislead on Solana, where a skipped slot advances the slot counter but not the block height.

29. There is no metric for SVM shred-insert (ingestion) lag. It gates cordoning and appears in the cordon WARN log (shredInsertSlotLag), but no gauge exposes it. Monitor erpc_upstream_cordoned and read the reason label instead.

Metrics server log lines (useful when diagnosing startup failures):

• "starting metrics server on port: %d" — Info, erpc/init.go:L144
• "error starting metrics server: %s" — Error, erpc/init.go:L156
• "shutting down metrics server..." — Info, erpc/init.go:L161
• "metrics server forced to shutdown: %s" — Error, erpc/init.go:L164
• "metrics server stopped" — Info, erpc/init.go:L166
• "failed to set histogram buckets, using defaults" — Warn, erpc/init.go:L56

Observability

The metrics endpoint: http://<host>:<metrics.port>/ (any path) — promhttp.Handler() is the root so every HTTP path on the port returns the full Prometheus text exposition. Canonical Prometheus convention scrapes /metrics but any path works. Plain HTTP only, no auth.

Stock collectors (from prometheus/client_golang default registry, present unless ERPC_NOMETRICS=1):

• go_* — Go runtime memory/GC/goroutine stats
• process_* — OS process CPU/memory/fd stats
• promhttp_metric_handler_* — metrics handler self-scrape stats

Bundled alerting rules (monitoring/prometheus/alert.rules):

Full 122-metric catalog at /reference/metrics.

Top-20 operational metrics:

Source code entry points

• telemetry/metrics.go:L12-L729 (opens in a new tab) — all 122 metric definitions (counters, gauges, promauto histograms, LabeledHistogram wrappers), bucket constants, DefaultHistogramBuckets, buildFilterAwareHistograms, registerOrReuse, ParseHistogramBuckets
• erpc/init.go:L47-L57 (opens in a new tab) — metric initialization sequence: SetHistogramLabelFilter → SetHistogramBuckets; network alias resolver install
• erpc/init.go:L137-L170 (opens in a new tab) — metrics HTTP server construction: promhttp.Handler() on :<port>, ReadHeaderTimeout 10s, graceful shutdown 5s
• telemetry/labeled_histogram.go:L58-L190 (opens in a new tab) — LabeledHistogram struct, WithLabelValues/DeleteLabelValues/ActiveLabelValues, label filter projection
• telemetry/handles.go:L86-L105 (opens in a new tab) — CounterHandle/GaugeHandle/ObserverHandle label-bound child caches; ResetHandleCache
• health/tracker.go:L481-L667 (opens in a new tab) — DefaultIdleEvictionAfter; rotateMetricsLoop, sweepIdle, sweepIdleObservers — idle-series eviction for upstream_request_duration_seconds and rate_limits_total
• health/tracker.go:L1381-L1458 (opens in a new tab) — erpc_network_dynamic_block_time_milliseconds EMA algorithm (α=0.1, min 3 samples, bounds 10ms–120s)
• common/config.go:L2543-L2564 (opens in a new tab) — MetricsConfig struct definition
• common/defaults.go:L749-L767 (opens in a new tab) — MetricsConfig.SetDefaults
• common/errors.go:L17-L114 (opens in a new tab) — errorLabelMode global; SetErrorLabelMode; ErrorSummary (produces error label values for all error-labeled metrics)
• cmd/erpc/initflags.go:L22-L28 (opens in a new tab) — ERPC_NOMETRICS=1 env var handling
• erpc/block_heatmap.go:L14-L175 (opens in a new tab) — ComputeBlockHeatmapBucket tip-relative bucket label algorithm for erpc_network_evm_block_range_requested_total
• clients/grpc_bds_resilience.go:L30-L34 (opens in a new tab) — bdsHardCallTimeout = 20s hard-coded constant for erpc_grpc_bds_hard_timeout_total

Related pages

• Metrics reference — full 122-metric catalog with all labels and fire conditions.
• Rate limiters — erpc_rate_limits_total and erpc_rate_limiter_* metrics originate here.
• Selection policies — erpc_selection_* metrics live here.
• Hedge — erpc_network_hedged_request_total and erpc_network_hedge_winner_total.
• Consensus — erpc_consensus_* metric family.
• Survive provider outages — operational scenario where these metrics matter most.