Metrics reference

How it works

Initialization order. All 79 counters and 23 gauges register eagerly at package-init via promauto (telemetry/metrics.go:12-729). The 17 LabeledHistogram globals are initialized as empty wrappers so early-startup or test code can observe without NPE. erpc.Init then calls telemetry.SetHistogramLabelFilter (installs drop/override config) immediately followed by telemetry.SetHistogramBuckets (rebuilds every LabeledHistogram under the active filter and registers them). ResetHandleCache runs inside SetHistogramBuckets to invalidate cached label-bound handles. [erpc/init.go:L47-57]

Label filtering (LabeledHistogram). Each LabeledHistogram stores the full canonical label schema alongside the projected subset that survives the active HistogramLabelFilter. Call sites always pass the full schema in schema order; the wrapper silently projects before forwarding to the underlying prometheus.HistogramVec. A length mismatch panics immediately. ActiveLabelValues returns the projected subset so handle-caches key on the effective (post-filter) labels, preventing duplicate series. [telemetry/labeled_histogram.go:L121-137]

Handle caching. CounterHandle, GaugeHandle, and ObserverHandle in telemetry/handles.go cache children in sync.Maps keyed by {Vec pointer, '\x1f'-joined label values}. This avoids per-observation map lookup and mutex contention inside the Prometheus library. [telemetry/handles.go:L59-105]

Idle-series eviction. The health tracker's rotateMetricsLoop fires a sweep every 10 rotation ticks (default 30 s/tick = every 5 minutes). The sweep deletes upsMetrics and ntwMetrics entries not accessed in the last 30 minutes (DefaultIdleEvictionAfter), then calls DeleteLabelValues on erpc_upstream_request_duration_seconds and erpc_rate_limits_total. Cordoned entries and wildcard ("*") rollups are never evicted. All other high-cardinality series are NOT covered — cardinality is bounded only by distinct label tuples seen over the process lifetime. [health/tracker.go:L557-667]

Metrics HTTP server. erpc.Init starts a minimal http.Server bound to :%d (port only, all interfaces) serving promhttp.Handler() at the root path. This means every URL path on the metrics port returns identical Prometheus text output — GET /, GET /metrics, GET /health all work. No TLS, no auth, no gzip. Shutdown is graceful with a 5-second budget. [erpc/init.go:L145-158]

ERPC_NOMETRICS=1 env var. When set before process start, initflags.go:init() replaces prometheus.DefaultRegisterer and DefaultGatherer with a fresh empty registry. All subsequent promauto calls succeed but produce no-ops. The default go_*/process_* collectors are also dropped. [cmd/erpc/initflags.go:L22-28]

errorLabelMode global. common.ErrorSummary produces the error label value on every error-labeled metric. The in-code default before erpc.Init runs is verbose; erpc.Init flips it to compact (the config default). In compact mode it emits short stable codes: a plain StandardError → its Base().Code string (e.g. "ErrEndpointCapacityExceeded"); for ErrFailsafeRetryExceeded, ErrUpstreamRequest, or ErrUpstreamRequestSkipped it appends the cause code (e.g. "ErrUpstreamRequest/ErrEndpointTransportFailure"); for an ErrJsonRpcExceptionInternal cause it further appends the numeric code (e.g. "ErrUpstreamRequest/ErrJsonRpcExceptionInternal/-32000"). In verbose mode it passes the full message chain through cleanUpMessage (strips newlines and long hex strings), potentially including block numbers or IP addresses — each unique value creates a new Prometheus series. [common/errors.go:L17-114]

networkAlias resolver. At startup, erpc.Init installs a NetworkAliasResolver callback mapping raw EVM chain IDs to human-readable aliases so components using numeric chain IDs (e.g. gRPC cache connector) emit matching network label values. [erpc/init.go:L62-77]

Config schema

All fields are under metrics. in the root YAML config. Source: common/config.go:L2543-2564 (struct), common/defaults.go:L749-767 (defaults), common/validation.go:L131-161 (validation).

Hardcoded server constants (not configurable):

• Bind address: ":%d" — all interfaces, port only (erpc/init.go:L149)
• Handler: promhttp.Handler() registered as root / — every path returns full metrics output
• Protocol: plain HTTP only (no TLS, no auth, no gzip)
• ReadHeaderTimeout: 10 seconds
• Graceful shutdown budget: 5 seconds

Histograms affected by metrics.histogramBuckets: erpc_upstream_request_duration_seconds, erpc_network_request_duration_seconds, erpc_consensus_duration_seconds, erpc_cache_set_success_duration_seconds, erpc_cache_set_error_duration_seconds, erpc_cache_get_success_hit_duration_seconds, erpc_cache_get_success_miss_duration_seconds, erpc_cache_get_error_duration_seconds.

Histograms NOT affected (hard-coded bucket sets): all remaining histograms use dedicated inline bucket arrays and cannot be changed without recompiling.

Worked examples

1. Drop user globally but keep it on one histogram for user-level latency analysis. Useful when you need per-user latency breakdowns on network_request_duration_seconds but want to limit cardinality everywhere else:

metrics:
  histogramDropLabels:
    - user
    - agent_name
  histogramLabelOverrides:
    network_request_duration_seconds:
      - user

2. Tighter histogram buckets for a low-latency deployment (sub-100ms upstreams). The default [0.05, 0.5, 5, 30] buckets produce poor p95/p99 resolution when upstreams respond in under 50ms. Replacing them gives useful percentile data at real latencies:

metrics:
  histogramBuckets: "0.005,0.01,0.025,0.05,0.1,0.25,0.5,1,3"

3. Disabling metrics during integration tests. erpc.Init defaults metrics.enabled to nil (disabled) under go test, but if you call Init with a partial config that doesn't go through SetDefaults, the server may try to start. Explicit override:

metrics:
  enabled: false

4. Switching to verbose error labels for a debugging session. When you need to trace a specific block-number error pattern in Grafana and short codes aren't enough — but remember to revert after, as each unique block number creates a new Prometheus series:

metrics:
  errorLabelMode: verbose

Behavioral invariants

• All 79 counters and 23 gauges are registered via promauto at package init — they exist in /metrics from process start regardless of config. [telemetry/metrics.go:L12-729]
• LabeledHistogram instances are registered in erpc.Init after SetHistogramLabelFilter → SetHistogramBuckets. Calling SetHistogramBuckets a second time with a different filter panics. [telemetry/metrics.go:L942-950]
• ERPC_NOMETRICS=1 swaps prometheus.DefaultRegisterer/DefaultGatherer for a fresh empty registry before any promauto init fires — all subsequent registrations succeed but produce no-ops; the stock go_*/process_* collectors are also dropped. [cmd/erpc/initflags.go:L22-28]
• Idle sweep runs every 5 minutes (default); series not accessed in 30 minutes are evicted from upstream_request_duration_seconds and rate_limits_total via DeleteLabelValues. [health/tracker.go:L557-667]
• The error label value is produced by common.ErrorSummary. Compact mode yields a bounded set of code-path identifiers; verbose mode can produce one series per unique block number or IP. [common/errors.go:L46-66]
• erpc_rate_limiter_budget_decision_total is registered but has zero production call sites — always zero. [telemetry/metrics.go:L515-519]
• erpc_network_hedge_delay_seconds is registered but never observed in production — always empty. [telemetry/metrics.go:L813]

Best practices

• Use errorLabelMode: compact (the default) in production. Verbose mode can create one Prometheus series per unique block number or IP address in error messages — this is an unbounded cardinality hazard.
• Add ["user", "agent_name"] to histogramDropLabels when serving untrusted callers or managing many API keys. These labels are the leading cardinality source across all LabeledHistogram instances. Counter dimensions are preserved.
• Do not rely on hostV4/hostV6 to restrict scrape access — the server binds all interfaces regardless of those fields. Use firewall rules or network policy to limit who can reach port 4001.
• Alert on erpc_network_failed_request_total{severity="critical"} and page on severity="warning". Severity is already classified by common.ClassifySeverity — no need to build your own error-code regexes.
• Use erpc_rate_limits_total for rate-limit alerting. The bundled alert rule referencing erpc_upstream_request_self_rate_limited_total uses a stale metric name that no longer exists.
• Monitor erpc_upstream_cordoned as an early-warning signal for upstream health. A non-zero value means eRPC has autonomously excluded that upstream; combine with erpc_upstream_cordon_event_total to track how often it happens.
• For erpc_network_evm_block_range_requested_total, be aware the bucket label is unbounded. Consider a Prometheus metric_relabel_configs drop rule for the bucket label if long-running chains accumulate too many series.

Edge cases & gotchas

1. hostV4/hostV6 are defined but unused. The bind address is always ":%d" (all interfaces). Firewall rules are the only way to restrict access to the metrics port. (erpc/init.go:L149)
2. metrics.enabled is nil (disabled) under go test. Integration tests that call erpc.Init will not spin up a metrics server unless Enabled: true is set explicitly. (common/defaults.go:L750-752)
3. Changing errorLabelMode does not retroactively relabel existing series. Counter Vec children are created lazily on first WithLabelValues. Any series created before erpc.Init sets the mode carry verbose labels; subsequent series carry compact labels. Both sets coexist in Prometheus until the verbose series naturally expire.
4. histogramDropLabels changes after first SetHistogramBuckets panic. Re-calling with a changed filter is not supported. The config-analyzer calls SetHistogramBuckets for validation; in practice it runs in a separate code path before Init. (telemetry/metrics.go:L942-950)
5. Idle sweep evicts only upstream_request_duration_seconds and rate_limits_total. All other high-cardinality histograms and counters (network_request_duration_seconds, upstream_request_total, etc.) are never DeleteLabelValues'd — they grow monotonically. For method-flood scenarios add user and agent_name to histogramDropLabels.
6. erpc_cors_requests_total project label receives r.URL.Path, not the project ID. Series will have values like /myproject/evm/1. Dashboard queries on CORS metrics must match the path-style value, not a bare project ID. (erpc/http_server.go:L1020)
7. erpc_ristretto_cache_current_cost requires memory.emitMetrics: true. Without this flag the gauge is registered and appears in /metrics but always returns 0. It is the only metric in the system requiring an explicit opt-in to be meaningful. (data/memory.go:L71)
8. erpc_upstream_cordoned category label is the cordon scope (method string or "*"), not a request-category label. Do not join this category with category from request-accounting metrics. (health/tracker.go:L458-465)
9. erpc_upstream_cordoned vendor label is "n/a" when no vendor is configured. Dashboard JOINs between cordon metrics and upstream request metrics must handle the "n/a" value for unvendored upstreams. (upstream/upstream.go:L275-286)
10. erpc_auth_failed_total always has strategy="database". Only the database auth strategy emits this counter. For other strategies, monitor failures via erpc_network_failed_request_total{error=~"ErrAuthUnauthorized.*"}. (auth/strategy_database.go:L467-472)
11. erpc_upstream_attempt_outcome_total is_hedge/is_retry are literal "true"/"false" strings. Use {is_hedge="true"} in PromQL, not {is_hedge="1"}. (upstream/upstream.go:L80-85)
12. erpc_network_evm_block_range_requested_total bucket label is unbounded. When tip is known, ComputeBlockHeatmapBucket produces human-readable relative labels ("TIP", "L100k", "100k-200k", etc.). This counter is not covered by the idle sweep — cardinality grows permanently. (erpc/block_heatmap.go:L87-175)
13. erpc_selection_* metrics carry a method label. Untrusted callers injecting hundreds of distinct method names create unbounded selection-gauge series. Consider a Prometheus relabeling drop rule for the method label in untrusted deployments.
14. erpc_upstream_block_head_large_rollback is silent for rollbacks ≤ 1024 blocks. DefaultToleratedBlockHeadRollback = 1024 is the threshold. A gauge value of 0 is normal; non-zero warrants investigation. (architecture/evm/evm_state_poller.go:L27)
15. Cordoned upstreams are never swept. A cordoned tracker entry persists in memory — and the erpc_upstream_cordoned{...} = 1 gauge persists in /metrics — until the upstream is uncordoned and subsequently goes idle for idleEvictionAfter. (health/tracker.go:L603-608)
16. erpc_network_dynamic_block_time_milliseconds is bounded 10ms–120s. Fast chains (block time below 10ms) or stuck chains (no blocks for over 2 minutes) report the floor or ceiling value, not the true EMA. (health/tracker.go:L1456)
17. Two bundled alert rules reference stale metric names. erpc_upstream_request_self_rate_limited_total and erpc_network_request_self_rate_limited_total no longer exist — use erpc_rate_limits_total as the replacement. (monitoring/prometheus/alert.rules:L1)
18. registerOrReuse silently returns an existing LabeledHistogram on identical re-registration. Calling SetHistogramBuckets twice with the same bucket string and filter is idempotent; the second call is a no-op against the Prometheus registry. (telemetry/metrics.go:L978-995)
19. The bundled Prometheus scrape target includes REPLACE_SERVICE_ENDPOINT_HERE and REPLACE_SERVICE_PORT_HERE placeholders. These are Railway deployment stubs that cause a parse error in strict Prometheus configs. Remove or replace both before using monitoring/prometheus/prometheus.yml in production.
20. ParseHistogramBuckets silently sorts the input. If metrics.histogramBuckets is provided out of order (e.g. "5,0.5,0.05,30") the values are sorted automatically and no warning is emitted. The resulting bucket set is valid, but operators relying on a specific insertion order will not get what they typed. [telemetry/metrics.go:L997-1015]
21. erpc_upstream_response_size_bytes is intentionally tight-cardinality. Its label set is project, network, category, finality — no user or upstream — because per-user response-size breakdown is not operationally actionable. If you need per-user latency analysis, retain user on erpc_upstream_request_duration_seconds via histogramLabelOverrides instead of dropping it globally. [telemetry/metrics.go:L924]
22. erpc_upstream_request_duration_seconds has two high-cardinality labels: user and composite. The composite label carries the composite-request type (e.g. "logs-split-on-error", "query-logs-shim") and, combined with user, can produce many series in high-traffic deployments. Both are candidates for histogramDropLabels; the idle sweep (DefaultIdleEvictionAfter = 30 min) provides a safety valve. [health/tracker.go:L333-358]
23. ErrEndpointServerSideException HTTP status code is not exposed as a Prometheus label. When an upstream returns an HTTP 5xx that does not match a known pattern, eRPC wraps it in ErrEndpointServerSideException and echoes the upstream's status code back to the caller via ErrorStatusCode(). The raw numeric code surfaces in erpc_upstream_request_errors_total{error="ErrEndpointServerSideException"} in compact mode (no code number) or in the full message in verbose mode, but no label carries it. For debugging, use OTel traces: SetTraceSpanError attaches attribute.String("error.code", stdErr.CodeChain()) and the serialised StandardError JSON to the span — though originalStatusCode is a private field and is not in the JSON output. The only way to retrieve the exact upstream HTTP status is from structured logs (logged at DEBUG level before wrapping). [common/errors.go:L1951-1979]

Full metrics catalog

All metric names carry the erpc_ prefix. Full definitions: telemetry/metrics.go:L12-932.

Area 1: Upstream request accounting

Area 2: Network request accounting

Area 3: Block number and tip tracking

Area 4: Cache connector freshness

Area 5: Cache operations (JSON-RPC cache)

Area 6: Rate limiting

Area 7: Upstream health (cordon, circuit-breaker, probing)

Area 8: Selection policy

Area 9: Consensus

Area 10: Hedging

Area 11: EVM method-specific

Area 12: gRPC BDS resilience

Area 13: CORS

Area 14: Shadow testing

Area 15: Auth

Area 16: Panics

Histogram bucket sets

Only DefaultHistogramBuckets are configurable at runtime. All other bucket sets require a recompile.

Observability

The metrics endpoint: http://<host>:<metrics.port>/ (any path) — promhttp.Handler() is the root handler so every HTTP path on port 4001 returns the full Prometheus text exposition. Plain HTTP only, no auth, no path routing. Reference scrape config at monitoring/prometheus/prometheus.yml (opens in a new tab) (scrape interval 10 s, default target host.docker.internal:4001).

Stock collectors (present unless ERPC_NOMETRICS=1): go_* — Go runtime stats; process_* — OS process stats; promhttp_metric_handler_* — handler self-scrape stats.

Bundled alerting rules (monitoring/prometheus/alert.rules):

Metrics server log lines (all emitted by erpc/init.go):

Source code entry points

• telemetry/metrics.go:L12-L729 (opens in a new tab) — all 79 counters and 23 gauges; bucket constants; DefaultHistogramBuckets
• telemetry/metrics.go:L755-L932 (opens in a new tab) — buildFilterAwareHistograms; all 17 LabeledHistogram definitions; SetHistogramBuckets; ParseHistogramBuckets; registerOrReuse
• telemetry/labeled_histogram.go:L16-L187 (opens in a new tab) — HistogramLabelFilter; SetHistogramLabelFilter; LabeledHistogram with WithLabelValues, DeleteLabelValues, ActiveLabelValues
• telemetry/handles.go:L59-L105 (opens in a new tab) — CounterHandle/GaugeHandle/ObserverHandle caches; ResetHandleCache
• erpc/init.go:L47-L170 (opens in a new tab) — initialization sequence: SetHistogramLabelFilter → SetHistogramBuckets → network alias resolver → metrics HTTP server lifecycle
• common/config.go:L2543-L2564 (opens in a new tab) — MetricsConfig struct
• common/defaults.go:L749-L767 (opens in a new tab) — MetricsConfig.SetDefaults
• common/errors.go:L17-L114 (opens in a new tab) — errorLabelMode global; SetErrorLabelMode; ErrorSummary
• health/tracker.go:L481-L667 (opens in a new tab) — DefaultIdleEvictionAfter; rotateMetricsLoop; sweepIdle; sweepIdleObservers
• cmd/erpc/initflags.go:L22-L28 (opens in a new tab) — ERPC_NOMETRICS=1 registry replacement
• monitoring/catch-up-metrics.md (opens in a new tab) — operator guide for reading erpc_network_retry_attempt_total + erpc_network_data_unavailable_wait_seconds together; covers CatchUpWaitHistogramBuckets design rationale, finality-label split, Little's Law pressure interpretation, and tuning knobs

Related pages

• Rate limiters — the erpc_rate_limits_total counter tracks every budget decision.
• Failsafe: Hedge — erpc_network_hedged_request_total and erpc_network_hedge_winner_total are the hedge observability surface.
• Failsafe: Retry — erpc_network_retry_attempt_total with its reason label tracks retry behavior.
• Selection policies — the erpc_selection_* family tracks tick-by-tick upstream ranking.
• Cache — erpc_cache_get_* and erpc_cache_set_* families cover all cache connector activity.
• Auth — erpc_auth_failed_total for database-strategy failures.