# Metrics reference

> Source: https://docs.erpc.cloud/reference/metrics
> Every observable signal eRPC emits — 141 Prometheus metrics across upstreams, cache, rate limiting, consensus, hedging, and more — ready to wire into your dashboards and alerts.
> Format: machine-readable markdown export of the docs page above.
> All collapsible AI sections are inlined and fully expanded.

# Metrics reference

eRPC exposes a Prometheus `/metrics` endpoint on port 4001. Every subsystem emits counters, gauges, and histograms — upstream health, cache efficiency, rate limiting, consensus, hedging, selection scoring, and more. Drop any scraper on port 4001 and the full picture of your RPC fleet is immediately visible.

- 141 metric definitions under the `erpc_` namespace
- One knob shapes the whole page: `metrics.customizations` drops families, projects labels, and overrides histogram buckets — a dropped family costs no series at all
- High-cardinality labels (`user`, `agent_name`) are droppable fleet-wide and re-addable per metric
- Stale series auto-evict every five minutes so cardinality stays bounded

## Agent reference

Copy one of these prompts into your AI agent session (Claude Code, Cursor, …) — each one
points the agent at this page's machine-readable reference so it can do the work correctly:

**Prompt Example #1: wire up Grafana dashboards for my eRPC deployment**

```text
I want to build Grafana dashboards for my eRPC deployment — upstream health, cache
hit rate, request latency percentiles, and rate-limit events. Using the full metrics
catalog, generate PromQL queries for each panel and point out any cardinality footguns
I should avoid (like the block-range heatmap or CORS project label). Work with my existing eRPC config. Reference:
https://docs.erpc.cloud/reference/metrics.llms.txt
```

**Prompt Example #2: reduce Prometheus cardinality without losing signal**

```text
My Prometheus is accumulating too many time series from eRPC histograms. Audit my
my eRPC config metrics config and recommend which metrics.customizations entries to add —
which labels to drop fleet-wide, which metrics to keep per-user latency on, and
how to tune buckets for my sub-100ms upstream fleet. Reference:
https://docs.erpc.cloud/reference/metrics.llms.txt
```

**Prompt Example #3: alert on upstream health and cordon events**

```text
I need PagerDuty-ready alerts for upstream health in my eRPC setup: cordoned
upstreams, circuit-breaker state changes, and BDS hard-timeout spikes. Write PromQL
alert expressions referencing the correct metric names (watch out for stale names in
the bundled alert.rules file). Work with my existing eRPC config. Reference:
https://docs.erpc.cloud/reference/metrics.llms.txt
```

**Prompt Example #4: debug a missing or always-zero metric**

```text
One of my eRPC metrics is always 0 and I can't tell if it's a config issue or a
dormant metric. Check the metrics reference for known dormant metrics and required
opt-in flags (like memory.emitMetrics), and tell me whether my eRPC config is missing
any config that would make the metric meaningful. Reference:
https://docs.erpc.cloud/reference/metrics.llms.txt
```

---

### Metrics — full agent reference

### How it works

**Initialization order.** Every metric family is *defined* at package init and *registered* later, by a single `telemetry.Configure` call from `erpc.Init`. Definitions go through the `Define*` factories in [`telemetry/manager.go`](https://github.com/erpc/erpc/blob/main/telemetry/manager.go) — `DefineCounter`, `DefineGauge`, `DefineLabeledCounter`, `DefineLabeledHistogram` — which record the family (with its kind) in the manager's index and return a usable, unregistered collector. Call sites hold that pointer for the process lifetime and never check whether it is registered.

`Configure` then, in order: compiles `metrics.customizations` and the deprecated label knobs into a single `MetricPolicy` (returning an error and registering nothing if an entry is malformed), resolves `histogramBuckets`, publishes the policy, and walks the definition index. For each family it skips the ones the policy drops, rebuilds the rest under the now-published policy — its label projection and, for a histogram, its per-family buckets — registers them with `prometheus.DefaultRegisterer`, and finally calls `ResetHandleCache` to invalidate handles pointing at the pre-rebuild Vecs. [[`telemetry/manager.go:L196-286`](https://github.com/erpc/erpc/blob/main/telemetry/manager.go#L196-L286)]

The define/register split is forced by Prometheus: a registry freezes a family's label-set hash the first time it is registered and keeps it for the registry's lifetime (`dimHashesByName` survives `Unregister`). Anything that changes a label set — a `labels` rule, the deprecated `histogramDropLabels`/`counterDropLabels` — therefore has to be applied *before* the first registration. Dropping a family needs the same window for a different reason: the only way to keep a family off `/metrics` entirely is to never register it.

**The policy engine.** [`telemetry/policy.go`](https://github.com/erpc/erpc/blob/main/telemetry/policy.go) owns every exposure, label and bucket decision, so there is exactly one precedence implementation. It compiles each customization into a rule carrying a compiled subject pattern (exact, `prefix*`, or `*`), an optional exposure action, its label rules and its buckets; the deprecated knobs desugar onto the same rule type with a kind mask (`counterDropLabels` becomes a counters-only `*` rule). Rules are sorted by specificity — exact beats prefix, longer prefix beats shorter, ties break to the later entry, and a desugared legacy knob sits below an equally specific customization so migrating a field takes effect. Lookups (`Exposed`, `labelIndices`, `BucketsFor`) walk the sorted rules and take the most specific match.

**`SetHistogramBuckets`.** A narrower entry point that resolves the bucket string and registers the histogram families only. Two callers use it: config validation, which wants the parse error and nothing else, and test setup, which needs histograms scrapeable while leaving counter label sets still open. Production goes through `Configure`.

**Label projection (`LabeledCounter`, `LabeledHistogram`).** Each wrapper stores the full canonical label schema alongside the projected subset the policy keeps for that family. Call sites always pass the full schema in schema order; the wrapper silently projects before forwarding to the underlying `prometheus.CounterVec`/`HistogramVec`. A length mismatch panics immediately. `ActiveLabelValues` returns the projected subset so handle-caches key on the effective (post-projection) labels, preventing duplicate series. Gauges are never projected — collapsing gauge series would report whichever writer wrote last rather than a coarser number. [[`telemetry/labeled_histogram.go:L117-130`](https://github.com/erpc/erpc/blob/main/telemetry/labeled_histogram.go#L117-L130)]

**Handle caching.** `CounterHandle`, `GaugeHandle`, and `ObserverHandle` in `telemetry/handles.go` cache children in `sync.Map`s keyed by `{Vec pointer, '\x1f'-joined label values}`. This avoids per-observation map lookup and mutex contention inside the Prometheus library. [[`telemetry/handles.go:L59-105`](https://github.com/erpc/erpc/blob/main/telemetry/handles.go#L59-L105)]

**Idle-series eviction.** The health tracker's `rotateMetricsLoop` fires a sweep every 10 rotation ticks (default 30 s/tick = every 5 minutes). The sweep deletes `upsMetrics` and `ntwMetrics` entries not accessed in the last 30 minutes (`DefaultIdleEvictionAfter`), then calls `DeleteLabelValues` on `erpc_upstream_request_duration_seconds` and `erpc_rate_limits_total`. Cordoned entries and wildcard (`"*"`) rollups are never evicted. All other high-cardinality series are NOT covered — cardinality is bounded only by distinct label tuples seen over the process lifetime. [[`health/tracker.go:L557-667`](https://github.com/erpc/erpc/blob/main/health/tracker.go#L557-L667)]

**Metrics HTTP server.** `erpc.Init` starts a minimal `http.Server` bound to `:%d` (port only, all interfaces) serving the Prometheus handler at the root path. This means every URL path on the metrics port returns identical Prometheus text output — `GET /`, `GET /metrics`, `GET /health` all work. No TLS, no auth, no gzip. Shutdown is graceful with a 5-second budget. The gatherer is wrapped by `telemetry.Gatherer`, which applies the policy's drop rules to families the manager does not register itself (the stock `go_*`/`process_*`/`promhttp_*` collectors); with no customizations configured it hands the default gatherer straight through. [[`erpc/init.go:L184-215`](https://github.com/erpc/erpc/blob/main/erpc/init.go#L184-L215)]

**`ERPC_NOMETRICS=1` env var.** When set before process start, `initflags.go:init()` replaces `prometheus.DefaultRegisterer` and `DefaultGatherer` with a fresh empty registry. All subsequent registrations succeed but produce no-ops. The default `go_*`/`process_*` collectors are also dropped. [[`cmd/erpc/initflags.go:L22-28`](https://github.com/erpc/erpc/blob/main/cmd/erpc/initflags.go#L22-L28)]

**`errorLabelMode` global.** `common.ErrorSummary` produces the `error` label value on every error-labeled metric. The in-code default before `erpc.Init` runs is `verbose`; `erpc.Init` flips it to `compact` (the config default). In `compact` mode it emits short stable codes: a plain `StandardError` → its `Base().Code` string (e.g. `"ErrEndpointCapacityExceeded"`); for `ErrFailsafeRetryExceeded`, `ErrUpstreamRequest`, or `ErrUpstreamRequestSkipped` it appends the cause code (e.g. `"ErrUpstreamRequest/ErrEndpointTransportFailure"`); for an `ErrJsonRpcExceptionInternal` cause it further appends the numeric code (e.g. `"ErrUpstreamRequest/ErrJsonRpcExceptionInternal/-32000"`). In `verbose` mode it passes the full message chain through `cleanUpMessage` (strips newlines and long hex strings), potentially including block numbers or IP addresses — each unique value creates a new Prometheus series. [[`common/errors.go:L17-114`](https://github.com/erpc/erpc/blob/main/common/errors.go#L17-L114)]

**`networkAlias` resolver.** At startup, `erpc.Init` installs a `NetworkAliasResolver` callback mapping raw EVM chain IDs to human-readable aliases so components using numeric chain IDs (e.g. gRPC cache connector) emit matching `network` label values. [[`erpc/init.go:L95-110`](https://github.com/erpc/erpc/blob/main/erpc/init.go#L95-L110)]

---

### Config schema

All fields are under `metrics.` in the root YAML config. Source: [`common/config.go:L2899-3016`](https://github.com/erpc/erpc/blob/main/common/config.go#L2899-L3016) (struct), [`common/defaults.go:L908-931`](https://github.com/erpc/erpc/blob/main/common/defaults.go#L908-L931) (defaults), [`common/validation.go:L139-171`](https://github.com/erpc/erpc/blob/main/common/validation.go#L139-L171) (validation).

| Field | Type | Default | Behavior / footguns |
|---|---|---|---|
| `metrics.enabled` | `*bool` | `true` in production; `nil` (disabled) under `go test` | Whether to start the `/metrics` HTTP server. Metrics are still registered and counted when false — they just are not scraped. Source: [`common/defaults.go:L909-911`](https://github.com/erpc/erpc/blob/main/common/defaults.go#L909-L911) |
| `metrics.port` | `*int` | `4001` | TCP port for the metrics server. Required when `enabled=true`; `erpc.Init` aborts if nil. Source: [`common/defaults.go:L918-920`](https://github.com/erpc/erpc/blob/main/common/defaults.go#L918-L920) |
| `metrics.hostV4` | `*string` | `"0.0.0.0"` | Defined and defaulted but **not used** — the server binds `":%d"` (all interfaces). Setting this to `127.0.0.1` does NOT restrict to loopback. Use firewall rules instead. Source: [`erpc/init.go:L189`](https://github.com/erpc/erpc/blob/main/erpc/init.go#L189) |
| `metrics.hostV6` | `*string` | `"[::]"` | Same caveat as `hostV4` — defined but not used in the bind address. Source: [`common/defaults.go:L915-917`](https://github.com/erpc/erpc/blob/main/common/defaults.go#L915-L917) |
| `metrics.listenV4` | `*bool` | `nil` | Defined in struct but never read in production code. Dead config field. |
| `metrics.listenV6` | `*bool` | `nil` | Defined in struct but never read. Dead config field. |
| `metrics.errorLabelMode` | `string` | `"compact"` | Controls the `error` label on all error-labeled metrics. `"compact"` → short stable codes (bounded cardinality, recommended). `"verbose"` → full human messages including block numbers and IP addresses (unbounded cardinality risk). Must be `""`, `"compact"`, or `"verbose"`. Source: [`common/defaults.go:L921-923`](https://github.com/erpc/erpc/blob/main/common/defaults.go#L921-L923) |
| `metrics.histogramBuckets` | `string` | `""` → `[0.05, 0.5, 5, 30]` | Comma-separated float64 bucket boundaries for every histogram that does not declare its own buckets. Empty → use defaults. Parsed and sorted by `ParseHistogramBuckets`. Invalid float → warning + fallback to defaults. Source: [`telemetry/metrics.go:L943-948`](https://github.com/erpc/erpc/blob/main/telemetry/metrics.go#L943-L948) |
| `metrics.customizations` | `[]object` | `nil` (nothing customized) | The one knob for shaping `/metrics`. Each entry has a `subject`, an optional `action`, an optional `labels` list, and optional `buckets`. Startup-only — every field is resolved before the first registration, so a change needs a restart. Source: [`telemetry/policy.go`](https://github.com/erpc/erpc/blob/main/telemetry/policy.go) |
| `metrics.customizations[].subject` | `string` | required | Selects metric families: an exact name (`"upstream_request_total"`), a prefix ending in `*` (`"consensus_*"`), or `*` for every family. The `erpc_` prefix is optional. Stock collectors are named in full (`"go_goroutines"`, `"process_cpu_seconds_total"`, `"promhttp_metric_handler_requests_total"`) and obey the same rules, so `subject: "*"` with `action: drop` drops them too. `*` is only legal as the final character, and the same subject may not appear twice. |
| `metrics.customizations[].action` | `"keep"` \| `"drop"` | none (exposure untouched) | `drop` removes the matched families from `/metrics`; `keep` spares them from a broader `drop`. Omit it to customize labels or buckets without touching exposure. A dropped eRPC family is never registered, so it costs no series, no collection time and no scrape bytes; a dropped stock collector is filtered out of the scrape response instead, which shrinks the page but not collection. |
| `metrics.customizations[].labels` | `[]object` | `nil` (full label set) | A list of `{subject, action}` entries projecting the matched families' label sets, with the same precedence rules applied to label names — `agent_*: drop` then `agent_name: keep` drops the group and spares the one label. Counters and histograms are projected; gauges are not. |
| `metrics.customizations[].buckets` | `[]float64` | `nil` | Replaces the bucket boundaries of the matched histograms, overriding both `metrics.histogramBuckets` and what the metric declares in code. Must be strictly increasing. Ignored (with a startup warning) if an exact subject names something that is not a histogram. |
| `metrics.histogramDropLabels` | `[]string` | `nil` | **Deprecated** — use a `customizations` entry with `subject: "*"` and a `labels` list. Still honored: desugared into an every-histogram label drop on the same rule engine. Source: [`common/config.go:L2928-2944`](https://github.com/erpc/erpc/blob/main/common/config.go#L2928-L2944) |
| `metrics.histogramLabelOverrides` | `map[string][]string` | `nil` | **Deprecated** — use a `customizations` entry with the exact `subject` and a `labels` list keeping what that metric needs. Source: [`common/config.go:L2928-2944`](https://github.com/erpc/erpc/blob/main/common/config.go#L2928-L2944) |
| `metrics.counterDropLabels` | `[]string` | `nil` | **Deprecated** — same replacement as `histogramDropLabels`. Desugared into an every-counter label drop. |
| `metrics.counterLabelOverrides` | `map[string][]string` | `nil` | **Deprecated** — same replacement as `histogramLabelOverrides`. |

**Precedence.** Overlapping subjects resolve by *specificity*, not by list order: an exact
family name beats a prefix, a longer prefix beats a shorter one, and two equally specific
subjects break to the one written later. So `drop consensus_*` plus
`keep consensus_duration_seconds` means the same thing whichever order it appears in, and
appending a carve-out never requires reordering the list. The deprecated label knobs are
desugared into the same rules but sit *below* an equally specific `customizations` entry,
so migrating a field off them takes effect immediately.

**Hardcoded server constants (not configurable):**

- Bind address: `":%d"` — all interfaces, port only ([`erpc/init.go:L189`](https://github.com/erpc/erpc/blob/main/erpc/init.go#L189))
- Handler: the Prometheus handler registered as root `/`, over a gatherer wrapped by `telemetry.Gatherer` — every path returns full metrics output
- Protocol: plain HTTP only (no TLS, no auth, no gzip)
- `ReadHeaderTimeout`: 10 seconds
- Graceful shutdown budget: 5 seconds

**Histograms affected by `metrics.histogramBuckets`:** `erpc_upstream_request_duration_seconds`, `erpc_network_request_duration_seconds`, `erpc_consensus_duration_seconds`, `erpc_cache_set_success_duration_seconds`, `erpc_cache_set_error_duration_seconds`, `erpc_cache_get_success_hit_duration_seconds`, `erpc_cache_get_success_miss_duration_seconds`, `erpc_cache_get_error_duration_seconds`.

**Histograms NOT affected by `metrics.histogramBuckets`:** all remaining histograms declare dedicated inline bucket arrays, and the global list deliberately leaves those alone. They are still reachable via a `customizations` entry with `buckets`, which overrides the compiled-in array.

---

### Worked examples

**1. Drop `user` globally but keep it on one histogram for user-level latency analysis.** Useful when you need per-user latency breakdowns on `network_request_duration_seconds` but want to limit cardinality everywhere else:

```yaml
metrics:
  customizations:
    # Fleet-wide: no per-user, no per-agent dimensions.
    - subject: "*"
      labels:
        - subject: user
          action: drop
        - subject: agent_name
          action: drop
    # …except on this one histogram, which a latency dashboard reads per user.
    - subject: network_request_duration_seconds
      labels:
        - subject: user
          action: keep
```

**2. Tighter histogram buckets for a low-latency deployment (sub-100ms upstreams).** The default `[0.05, 0.5, 5, 30]` buckets produce poor p95/p99 resolution when upstreams respond in under 50ms. Replacing them gives useful percentile data at real latencies:

```yaml
metrics:
  histogramBuckets: "0.005,0.01,0.025,0.05,0.1,0.25,0.5,1,3"
```

**3. Disabling metrics during integration tests.** `erpc.Init` defaults `metrics.enabled` to `nil` (disabled) under `go test`, but if you call `Init` with a partial config that doesn't go through `SetDefaults`, the server may try to start. Explicit override:

```yaml
metrics:
  enabled: false
```

**4. Keeping only the families a dashboard actually reads.** The strongest cardinality lever available: a dropped family is never registered, so it costs no series, no collection time, and no scrape bytes. An allowlist is written explicitly — drop everything, then keep what you want — so nothing depends on which other keys are present. `subject: "*"` also covers the stock `go_*`/`process_*` collectors, so name them if you want to keep them:

```yaml
metrics:
  customizations:
    - subject: "*"
      action: drop
    - subject: upstream_request_total
      action: keep
    - subject: upstream_request_errors_total
      action: keep
    - subject: network_request_duration_seconds
      action: keep
    - subject: "go_*"
      action: keep
    - subject: "process_*"
      action: keep
```

**5. Keeping a subsystem but dropping its noisiest family.** The exact subject is more specific than the prefix, so the carve-out does not require enumerating everything else — and the order of the two entries does not matter:

```yaml
metrics:
  customizations:
    - subject: "consensus_*"
      action: keep
    - subject: consensus_responses_collected
      action: drop
```

**6. Per-metric histogram buckets.** `metrics.histogramBuckets` is one global list, which is the wrong resolution when two histograms measure different things. `buckets` on a customization overrides it — and the metric's own compiled-in buckets — for just the families the subject matches:

```yaml
metrics:
  histogramBuckets: "0.05,0.5,5,30"
  customizations:
    # Sub-100ms upstreams need resolution the global list does not have.
    - subject: upstream_request_duration_seconds
      buckets: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 1]
    # This one counts blocks, not seconds.
    - subject: network_evm_get_logs_range_requested
      buckets: [10, 100, 1000, 10000, 100000]
```

**7. Switching to verbose error labels for a debugging session.** When you need to trace a specific block-number error pattern in Grafana and short codes aren't enough — but remember to revert after, as each unique block number creates a new Prometheus series:

```yaml
metrics:
  errorLabelMode: verbose
```

---

### Behavioral invariants

- No metric family is registered at package init. Every family — counters, gauges, histograms — becomes scrapeable only once `telemetry.Configure` runs from `erpc.Init`. A process that never calls `Init` (or calls it with a config whose `metrics` section is absent) exposes the histogram families only. [[`telemetry/manager.go`](https://github.com/erpc/erpc/blob/main/telemetry/manager.go)]
- Exposure is a startup decision. A family dropped by a customization is never registered, so it cannot be turned back on without a restart — and a family that *was* registered cannot be removed by a later `Configure`, because Prometheus keeps the frozen label-set hash for the registry's lifetime.
- Metrics still count when `metrics.enabled` is false: `enabled` only controls whether the HTTP server starts. A `drop` customization is what actually prevents a family from existing.
- A customization with no `action` never affects exposure. It customizes labels or buckets only, and it does not turn on the scrape-path filter that hides stock collectors.
- A repeat `Configure` on the same registry leaves already-registered families untouched — their label sets are frozen, and accumulated values are preserved. Changing `customizations` or `histogramBuckets` requires a restart.
- `ERPC_NOMETRICS=1` swaps `prometheus.DefaultRegisterer`/`DefaultGatherer` for a fresh empty registry before `Configure` runs — all subsequent registrations succeed but produce no-ops; the stock `go_*`/`process_*` collectors are also dropped. [[`cmd/erpc/initflags.go:L22-28`](https://github.com/erpc/erpc/blob/main/cmd/erpc/initflags.go#L22-L28)]
- Idle sweep runs every 5 minutes (default); series not accessed in 30 minutes are evicted from `upstream_request_duration_seconds` and `rate_limits_total` via `DeleteLabelValues`. [[`health/tracker.go:L557-667`](https://github.com/erpc/erpc/blob/main/health/tracker.go#L557-L667)]
- The `error` label value is produced by `common.ErrorSummary`. Compact mode yields a bounded set of code-path identifiers; verbose mode can produce one series per unique block number or IP. [[`common/errors.go:L46-66`](https://github.com/erpc/erpc/blob/main/common/errors.go#L46-L66)]
- `erpc_rate_limiter_budget_decision_total` is registered but has zero production call sites — always zero. [[`telemetry/metrics.go:L515-519`](https://github.com/erpc/erpc/blob/main/telemetry/metrics.go#L515-L519)]
- `erpc_network_hedge_delay_seconds` is registered but never observed in production — always empty. [[`telemetry/metrics.go:L813`](https://github.com/erpc/erpc/blob/main/telemetry/metrics.go#L813)]

---

### Best practices

- Use `errorLabelMode: compact` (the default) in production. Verbose mode can create one Prometheus series per unique block number or IP address in error messages — this is an unbounded cardinality hazard.
- Drop `user` and `agent_name` with a `subject: "*"` customization when serving untrusted callers or managing many API keys. These labels are the leading cardinality source across the projected families. Re-add them with an exact subject on the one or two metrics a dashboard or billing pipeline actually reads.
- Do not rely on `hostV4`/`hostV6` to restrict scrape access — the server binds all interfaces regardless of those fields. Use firewall rules or network policy to limit who can reach port 4001.
- Alert on `erpc_network_failed_request_total{severity="critical"}` and page on `severity="warning"`. Severity is already classified by `common.ClassifySeverity` — no need to build your own error-code regexes.
- Use `erpc_rate_limits_total` for rate-limit alerting. The bundled alert rule referencing `erpc_upstream_request_self_rate_limited_total` uses a stale metric name that no longer exists.
- Monitor `erpc_upstream_cordoned` as an early-warning signal for upstream health. A non-zero value means eRPC has autonomously excluded that upstream; combine with `erpc_upstream_cordon_event_total` to track how often it happens.
- For `erpc_network_evm_block_range_requested_total`, be aware the `bucket` label is unbounded. Consider a Prometheus `metric_relabel_configs` drop rule for the `bucket` label if long-running chains accumulate too many series.

---

### Edge cases & gotchas

1. **`hostV4`/`hostV6` are defined but unused.** The bind address is always `":%d"` (all interfaces). Firewall rules are the only way to restrict access to the metrics port. ([`erpc/init.go:L189`](https://github.com/erpc/erpc/blob/main/erpc/init.go#L189))
2. **`metrics.enabled` is `nil` (disabled) under `go test`.** Integration tests that call `erpc.Init` will not spin up a metrics server unless `Enabled: true` is set explicitly. ([`common/defaults.go:L909-911`](https://github.com/erpc/erpc/blob/main/common/defaults.go#L909-L911))
3. **Changing `errorLabelMode` does not retroactively relabel existing series.** Counter Vec children are created lazily on first `WithLabelValues`. Any series created before `erpc.Init` sets the mode carry verbose labels; subsequent series carry compact labels. Both sets coexist in Prometheus until the verbose series naturally expire.
4. **Label-set, bucket and exposure changes need a restart.** Prometheus keeps a family's label-set hash for the registry's lifetime, so a second `Configure` leaves already-registered families exactly as they are — a changed `customizations` list or `histogramBuckets` is silently not applied to them. ([`telemetry/manager.go`](https://github.com/erpc/erpc/blob/main/telemetry/manager.go))
5. **A `subject` that matches nothing is only a warning.** A typo'd family name is not a validation error — `Init` logs the unmatched subjects, quoting the text you wrote, and starts anyway. Check the startup log for `subjects match no known metric family` after editing the list. Subjects naming a stock collector are excluded from the check, since those are registered outside the manager and are never in `KnownFamilies`.
6. **A rule aimed at a family that cannot honor it is also only a warning.** `buckets` on a counter, or `labels` on a plain counter or gauge whose label set is fixed in code, is silently ignored; `Init` warns only when an *exact* subject names such a family, so a sweeping `subject: "*"` does not produce one warning per family. Look for `rules do not apply to the family they name`.
7. **Third-party collectors outside `go_*`/`process_*`/`promhttp_*` cannot be addressed.** Any other name is normalized by prepending `erpc_`, so a collector registered by a dependency under some other prefix is unreachable from `customizations` — `subject: "*", action: drop` still removes it (via the scrape filter), but no `keep` can name it back. ([`telemetry/policy.go`](https://github.com/erpc/erpc/blob/main/telemetry/policy.go))
8. **Idle sweep evicts only `upstream_request_duration_seconds` and `rate_limits_total`.** All other high-cardinality histograms and counters (`network_request_duration_seconds`, `upstream_request_total`, etc.) are never `DeleteLabelValues`'d — they grow monotonically. For method-flood scenarios drop `user` and `agent_name` with a `subject: "*"` customization.
9. **`erpc_cors_requests_total` `project` label receives `r.URL.Path`, not the project ID.** Series will have values like `/myproject/evm/1`. Dashboard queries on CORS metrics must match the path-style value, not a bare project ID. ([`erpc/http_server.go:L1020`](https://github.com/erpc/erpc/blob/main/erpc/http_server.go#L1020))
10. **`erpc_ristretto_cache_current_cost` requires `memory.emitMetrics: true`.** Without this flag the gauge is registered and appears in `/metrics` but always returns 0. It is the only metric in the system requiring an explicit opt-in to be meaningful. ([`data/memory.go:L71`](https://github.com/erpc/erpc/blob/main/data/memory.go#L71))
11. **`erpc_upstream_cordoned` `category` label is the cordon scope (method string or `"*"`), not a request-category label.** Do not join this `category` with `category` from request-accounting metrics. ([`health/tracker.go:L458-465`](https://github.com/erpc/erpc/blob/main/health/tracker.go#L458-L465))
12. **`erpc_upstream_cordoned` `vendor` label is `"n/a"` when no vendor is configured.** Dashboard JOINs between cordon metrics and upstream request metrics must handle the `"n/a"` value for unvendored upstreams. ([`upstream/upstream.go:L275-286`](https://github.com/erpc/erpc/blob/main/upstream/upstream.go#L275-L286))
13. **`erpc_auth_failed_total` increments when `ErrAuthUnauthorized` is sent to the client.** Closed `reason` per strategy via `NewErrAuthUnauthorized`; intermediate multi-strategy rejects are not counted. `agent_name` is always `"unknown"`. ([`auth/auth_metrics.go`](https://github.com/erpc/erpc/blob/main/auth/auth_metrics.go))
14. **`erpc_upstream_attempt_outcome_total` `is_hedge`/`is_retry` are literal `"true"`/`"false"` strings.** Use `{is_hedge="true"}` in PromQL, not `{is_hedge="1"}`. ([`upstream/upstream.go:L80-85`](https://github.com/erpc/erpc/blob/main/upstream/upstream.go#L80-L85))
15. **`erpc_network_evm_block_range_requested_total` `bucket` label is unbounded.** When tip is known, `ComputeBlockHeatmapBucket` produces human-readable relative labels (`"TIP"`, `"L100k"`, `"100k-200k"`, etc.). This counter is not covered by the idle sweep — cardinality grows permanently. ([`erpc/block_heatmap.go:L87-175`](https://github.com/erpc/erpc/blob/main/erpc/block_heatmap.go#L87-L175))
16. **`erpc_selection_*` metrics carry a `method` label.** Untrusted callers injecting hundreds of distinct method names create unbounded selection-gauge series. Consider a Prometheus relabeling drop rule for the `method` label in untrusted deployments.
17. **`erpc_upstream_block_head_large_rollback` is silent for rollbacks ≤ 1024 blocks, and sticky above them.** `DefaultToleratedBlockHeadRollback = 1024` is the threshold. The gauge is `Set` to the rollback *size* and is never reset, so `> 0` means "this process saw a rollback at some point", not "a rollback is happening now", and `rate()`/`increase()` are meaningless on it. Alert on `changes(...[window])`, split by the `finality` label. ([`architecture/evm/evm_state_poller.go:L27`](https://github.com/erpc/erpc/blob/main/architecture/evm/evm_state_poller.go#L27))
18. **Cordoned upstreams are never swept.** A cordoned tracker entry persists in memory — and the `erpc_upstream_cordoned{...} = 1` gauge persists in `/metrics` — until the upstream is uncordoned and subsequently goes idle for `idleEvictionAfter`. ([`health/tracker.go:L603-608`](https://github.com/erpc/erpc/blob/main/health/tracker.go#L603-L608))
19. **`erpc_network_dynamic_block_time_milliseconds` is bounded 10ms–120s.** Fast chains (block time below 10ms) or stuck chains (no blocks for over 2 minutes) report the floor or ceiling value, not the true EMA. ([`health/tracker.go:L1456`](https://github.com/erpc/erpc/blob/main/health/tracker.go#L1456))
20. **Two bundled alert rules reference stale metric names.** `erpc_upstream_request_self_rate_limited_total` and `erpc_network_request_self_rate_limited_total` no longer exist — use `erpc_rate_limits_total` as the replacement. ([`monitoring/prometheus/alert.rules:L1`](https://github.com/erpc/erpc/blob/main/monitoring/prometheus/alert.rules#L1))
21. **The bundled Prometheus scrape target includes `REPLACE_SERVICE_ENDPOINT_HERE` and `REPLACE_SERVICE_PORT_HERE` placeholders.** These are Railway deployment stubs that cause a parse error in strict Prometheus configs. Remove or replace both before using `monitoring/prometheus/prometheus.yml` in production.
22. **`ParseHistogramBuckets` silently sorts the input.** If `metrics.histogramBuckets` is provided out of order (e.g. `"5,0.5,0.05,30"`) the values are sorted automatically and no warning is emitted. The resulting bucket set is valid, but operators relying on a specific insertion order will not get what they typed. [[`telemetry/metrics.go:L1133-1151`](https://github.com/erpc/erpc/blob/main/telemetry/metrics.go#L1133-L1151)]
23. **`erpc_upstream_response_size_bytes` is intentionally tight-cardinality.** Its label set is `project, network, category, finality` — no `user` or `upstream` — because per-user response-size breakdown is not operationally actionable. If you need per-user latency analysis, retain `user` on `erpc_upstream_request_duration_seconds` with an exact-subject `labels: [{subject: user, action: keep}]` customization instead of not dropping it globally. [[`telemetry/metrics.go:L1125-1130`](https://github.com/erpc/erpc/blob/main/telemetry/metrics.go#L1125-L1130)]
24. **`erpc_upstream_request_duration_seconds` has two high-cardinality labels: `user` and `composite`.** The `composite` label carries the composite-request type (e.g. `"logs-split-on-error"`, `"query-logs-shim"`) and, combined with `user`, can produce many series in high-traffic deployments. Both are candidates for a `labels: [{subject: ..., action: drop}]` customization; the idle sweep (`DefaultIdleEvictionAfter = 30 min`) provides a safety valve. [[`health/tracker.go:L333-358`](https://github.com/erpc/erpc/blob/main/health/tracker.go#L333-L358)]
25. **`ErrEndpointServerSideException` HTTP status code is not exposed as a Prometheus label.** When an upstream returns an HTTP 5xx that does not match a known pattern, eRPC wraps it in `ErrEndpointServerSideException` and echoes the upstream's status code back to the caller via `ErrorStatusCode()`. The raw numeric code surfaces in `erpc_upstream_request_errors_total{error="ErrEndpointServerSideException"}` in compact mode (no code number) or in the full message in verbose mode, but no label carries it. For debugging, use OTel traces: `SetTraceSpanError` attaches `attribute.String("error.code", stdErr.CodeChain())` and the serialised `StandardError` JSON to the span — though `originalStatusCode` is a private field and is not in the JSON output. The only way to retrieve the exact upstream HTTP status is from structured logs (logged at DEBUG level before wrapping). [[`common/errors.go:L1951-1979`](https://github.com/erpc/erpc/blob/main/common/errors.go#L1951-L1979)]

---

### Full metrics catalog

All metric names carry the `erpc_` prefix. Full definitions: [`telemetry/metrics.go:L11-1131`](https://github.com/erpc/erpc/blob/main/telemetry/metrics.go#L11-L1131).

#### Area 1: Upstream request accounting

| Metric | Type | Labels | When it fires |
|---|---|---|---|
| `erpc_upstream_request_total` | counter | project, vendor, network, upstream, category, attempt, composite, finality, user, agent_name | Each actual attempt sent to an upstream. `composite` ∈ `"none"`, `"logs-split-on-error"`, `"logs-split-proactive"`, `"trace-filter-split-on-error"`, `"trace-filter-split-proactive"`, `"query-blocks-shim"`, `"query-transactions-shim"`, `"query-logs-shim"`, `"query-traces-shim"`, `"query-transfers-shim"`. |
| `erpc_upstream_request_errors_total` | counter | project, vendor, network, upstream, category, error, severity, composite, finality, user, agent_name | Upstream attempt returned an error. `severity` ∈ `"critical"`, `"warning"`, `"info"` via `common.ClassifySeverity`. Also incremented with severity=info for block-availability-gated skips. |
| `erpc_upstream_request_skipped_total` | counter | project, vendor, network, upstream, category, finality, user, agent_name | Upstream pre-forward checks decided to skip. |
| `erpc_upstream_request_missing_data_error_total` | counter | project, vendor, network, upstream, category, finality, user, agent_name | Upstream returned missing-data / not-synced error. |
| `erpc_upstream_request_empty_response_total` | counter | project, vendor, network, upstream, category, finality, user, agent_name | Upstream returned an emptyish response. |
| `erpc_upstream_wrong_empty_response_total` | counter | project, vendor, network, upstream, category, finality, user, agent_name | Upstream returned empty while consensus determined others had data. |
| `erpc_upstream_selection_total` | counter | project, network, upstream, category, reason, finality | Upstream picked for an attempt. `reason` ∈ `"primary"`, `"retry"`, `"hedge"`, `"consensus_slot"`, `"sweep"`. `"sweep"` = the upstream was picked as part of `runUpstreamSweep` — the try-all-upstreams iteration used in the non-consensus path when the primary fails and retries loop through remaining upstreams in sequence. |
| `erpc_upstream_credit_units_total` | counter | project, network, upstream, vendor, category, finality | Vendor credit units accrued by upstream attempts (Alchemy CUs, QuickNode credits, dRPC CUs). Summed across retries/hedges/consensus slots; cache hits and never-dialed attempts cost 0. Values are each vendor's own units — not money, not comparable across vendors. **No `user` label** (cardinality); per-user cost is in the `X-ERPC-Credits` response header and traces. Vendorless upstreams are excluded. |
| `erpc_upstream_attempt_outcome_total` | counter | project, network, upstream, category, outcome, is_hedge, is_retry, finality | Terminal attempt outcome. `outcome` ∈ `"success"`, `"empty"`, `"transport_error"`, `"server_error"`, `"client_error"`, `"rate_limited"`, `"missing_data"`, `"exec_revert"`, `"block_unavailable"`, `"breaker_open"`, `"cancelled"`, `"timeout"`, `"skipped"`. `is_hedge`/`is_retry` are literal `"true"`/`"false"`. |
| `erpc_upstream_request_duration_seconds` | LabeledHistogram | project, vendor, network, upstream, category, composite, finality, user | Duration of each upstream attempt. Idle series swept every ~30 min. `user` is a cardinality candidate for a label `drop` customization. |
| `erpc_upstream_response_size_bytes` | LabeledHistogram | project, network, category, finality | Decoded post-gzip result-body byte count. Tight label set by design (no `user`). Buckets: 4096, 65536, 1048576, 16777216, 104857600. |

#### Area 2: Network request accounting

| Metric | Type | Labels | When it fires |
|---|---|---|---|
| `erpc_network_request_received_total` | counter | project, network, category, finality, user, agent_name | Request received for a network. |
| `erpc_network_failed_request_total` | counter | project, network, category, attempt, error, severity, finality, user, agent_name | Request failed at network/project level. `severity` ∈ `"critical"`, `"warning"`, `"info"`. Page on critical, ticket on warning, ignore info. |
| `erpc_network_successful_request_total` | counter | project, network, vendor, upstream, category, attempt, finality, emptyish, user, agent_name | Request succeeded. `emptyish` ∈ `"true"`/`"false"` — true means the response was empty-ish (null, empty array). |
| `erpc_network_multiplexed_request_total` | counter | project, network, category, finality, user, agent_name | Request de-duplicated into an identical in-flight request. |
| `erpc_network_fallback_escape_total` | counter | project, network, category | The per-request fallback escape (`failover.onDefaultsExhausted`) retried a request on `tier:fallback` upstreams. Expected zero in steady state. |
| `erpc_upstream_websocket_connected` | gauge | project, vendor, network, upstream | `1` while the upstream's WebSocket connection is established, `0` while it reconnects. |
| `erpc_upstream_websocket_heads_stall_reconnects_total` | counter | project, network, upstream | Reconnects forced because the upstream's newHeads subscription went silent while its polled head kept advancing. Any increase means a node dropped a live subscription without closing it. |
| `erpc_websocket_subscription_notifications_dropped_total` | counter | project, network, kind | Client subscription notifications dropped because the subscription's buffer (`server.webSocket.subscriptionBufferSize`) was full. For `kind="logs"` the connection is also closed with `1013`. |
| `erpc_websocket_connections_active` | gauge | project | Open client WebSocket connections. |
| `erpc_websocket_subscriptions_active` | gauge | project, network, kind | Open client `eth_subscribe` subscriptions; drops on `eth_unsubscribe` and when the connection closes. |
| `erpc_websocket_notifications_sent_total` | counter | project, network, kind | Subscription notifications written to clients. |
| `erpc_websocket_notification_bytes_total` | counter | project, network, kind | JSON bytes of those notifications, without cost frame headers. |
| `erpc_websocket_send_queue_length` | histogram | project, network, kind | Notifications still queued on a client subscription when its writer takes the next one. Each subscription has its own queue of `server.webSocket.subscriptionBufferSize`. |
| `erpc_upstream_subscriptions_active` | gauge | project, network, upstream, kind | `eth_subscribe` subscriptions live on upstream WebSocket connections: `newHeads` on each upstream, plus one per distinct `logs` / `newPendingTransactions` filter, shared by every client. |
| `erpc_upstream_subscription_events_total` | counter | project, network, upstream, kind | Subscription notifications received from upstreams, before dedup. |
| `erpc_upstream_subscription_event_bytes_total` | counter | project, network, upstream, kind | Bytes of those notifications. |
| `erpc_subscription_dedup_dropped_total` | counter | project, network, kind | Upstream events dropped as duplicates of one already delivered (the same head or log from several upstreams). |
| `erpc_subscription_dedup_entries` | gauge | project, network | Keys held by the network's `logs` / `newPendingTransactions` dedup windows; each filter holds at most `indexer.dedupWindowSize` (default 8192). |
| `erpc_subscription_withheld_heads_total` | counter | project, network | Upstream heads not delivered to clients because the network did not accept them, e.g. a `tier:fallback` head while a primary streams heads, or a head the upstream's poller has not accepted yet. |
| `erpc_subscription_undelivered_total` | counter | project, network, kind | Upstream events that passed dedup but matched no client subscription. Sustained growth on `logs` / `newPendingTransactions` means upstream subscriptions nobody consumes; on `newHeads` it only means no client subscribes to heads. |
| `erpc_network_subscription_last_head_timestamp_seconds` | gauge | project, network | Unix time the subscription indexer last accepted a new head. Alert on `time() - this` well past the block time while clients are subscribed: the `newHeads` stream has stalled. |
| `erpc_websocket_rejected_total` | counter | project, reason | Client WebSocket connection or subscription refused by an instance-wide cap. `reason` ∈ `"conn_global"` (`server.webSocket.maxConnections`), `"conn_per_ip"` (`maxConnectionsPerIP`), `"sub_global"` (`maxSubscriptions`). |
| `erpc_network_static_response_served_total` | counter | project, network, category | Served from a configured static response; no upstream touched. |
| `erpc_network_timeout_fired_total` | counter | project, network, category, finality, scope | Timeout policy killed a request. `scope` ∈ `"network"`, `"upstream"`. Suppressed when retry-exhausted error wins. |
| `erpc_network_retry_attempt_total` | counter | project, network, category, reason, finality | Network-scope retry. `reason` ∈ `"empty_result"`, `"pending_tx"`, `"retryable_error"`, `"block_unavailable"`, `"missing_data"`. |
| `erpc_network_request_duration_seconds` | LabeledHistogram | project, network, vendor, upstream, category, finality, user | End-to-end network request duration. `vendor`/`upstream` = `"<error>"` on failure. |
| `erpc_network_data_unavailable_wait_seconds` | LabeledHistogram | project, network, category, reason, finality | Wall-clock catch-up delay before data-not-yet-available retry. Buckets: 0.1, 0.25, 0.5, 1, 2, 4, 8, 16, 32, 64 s. |
| `erpc_network_timeout_duration_seconds` | LabeledHistogram | project, network, category, finality | Computed dynamic timeout per request. Buckets: 0.05, 0.1, 0.3, 0.5, 1, 3, 5, 10, 30 s. |

#### Area 3: Block number and tip tracking

> **On SVM networks these carry slots, not block heights.** The names were coined for EVM and are reused verbatim by the SVM state poller, so `erpc_upstream_latest_block_number` is the upstream's *processed slot*, `erpc_upstream_finalized_block_number` its *finalized (rooted) slot*, and the `*_lag` gauges count slots. That distinction is not cosmetic: a skipped Solana slot advances the slot counter but not the block height, so slot and height are different counters. The rollback threshold on SVM is `DefaultToleratedSlotRollback` = 1024 slots. `erpc_upstream_stale_latest_block_total` fires when a `getSlot` answer is corrected to the highest known slot for its commitment. The poller emits no `*_polled_total`, no `erpc_network_dynamic_block_time_milliseconds`, no served-tip gauges, and no metric at all for shred-insert (ingestion) lag — see [SVM slot tracking & health](/reference/svm/slot-tracking.llms.txt).

| Metric | Type | Labels | When it fires |
|---|---|---|---|
| `erpc_upstream_latest_block_number` | gauge | project, vendor, network, upstream | Upstream's latest block advanced. |
| `erpc_upstream_finalized_block_number` | gauge | project, vendor, network, upstream | Upstream's finalized block advanced. |
| `erpc_upstream_block_head_lag` | gauge | project, vendor, network, upstream | Blocks behind the corroborated network head (second-highest non-benched upstream). |
| `erpc_upstream_finalization_lag` | gauge | project, vendor, network, upstream | Finalized blocks behind the corroborated network finalized head. |
| `erpc_upstream_block_head_large_rollback` | gauge | project, vendor, network, upstream, finality | Block head rolled back by more than `DefaultToleratedBlockHeadRollback` = 1024 blocks. Gauge value = absolute delta of the rollback. `finality` is the head axis that rolled back (`latest` or `finalized`), **not** a `DataFinalityState`. Sticky: never reset, so a non-zero value means "this process saw a rollback at some point", not "a rollback is happening now". |
| `erpc_upstream_latest_block_polled_total` | counter | project, vendor, network, upstream | State poller proactively polled latest block. |
| `erpc_upstream_finalized_block_polled_total` | counter | project, vendor, network, upstream | State poller proactively polled finalized block. |
| `erpc_upstream_stale_latest_block_total` | counter | project, vendor, network, upstream, category | Upstream returned a stale latest block vs. others. |
| `erpc_upstream_stale_finalized_block_total` | counter | project, vendor, network, upstream | Upstream returned a stale finalized block vs. others. |
| `erpc_upstream_stale_upper_bound_total` | counter | project, vendor, network, upstream, category, confidence | Request skipped: upstream latest block &lt; requested upper bound. `confidence` ∈ `"blockHead"`, `"finalizedBlock"`. |
| `erpc_upstream_stale_lower_bound_total` | counter | project, vendor, network, upstream, category, confidence | Request skipped: requested lower bound below upstream's available range. Same `confidence` values. |
| `erpc_network_latest_block_timestamp_distance_seconds` | gauge | project, network, origin | `now − latest block timestamp`. `origin` ∈ `"evm_state_poller"`, `"network_response"`. |
| `erpc_network_dynamic_block_time_milliseconds` | gauge | project, network | EMA block-time estimate (α=0.1, min 3 samples). Returns 0 until 3 samples. Bounded 10ms–120s. |
| `erpc_network_served_tip_block_number` | gauge | project, network, lane, axis | The tip actually served per `axis` (`latest`/`finalized`): the [served-tip pick](/reference/evm/block-tracking.llms.txt#served-tip) after the regression guard, the WebSocket delivered-head floor (network-wide `latest` only) and the guarantee floors. Set on every evaluation with a known tip, with or without a `servedTip` block. `lane="all"` = network-wide, evaluated at least every second; a named lane = a [use-upstream group](/reference/lanes.llms.txt), evaluated only by its requests; a request without a lane writes nothing. Keeps its last value while no upstream has a head. |
| `erpc_network_served_tip_lag_blocks` | gauge | project, network, lane, axis | `Freshest − served`, clamped at 0. `Freshest` is the second-highest head on the ballot (the only one when one votes), so a lone far-ahead head never inflates it. Reads 0 at quorum ≤ 2 unless a guarantee floor sits below the pick, and 0 while the guard or the WebSocket floor lifts the tip. Keeps its last value while no upstream has a head. |
| `erpc_network_served_tip_advance_age_seconds` | gauge | project, network, lane, axis | Seconds since this process saw the served value change, written at each evaluation; absent until the first change. Keeps rising while no upstream has a head: the stuck-tip signal. |
| `erpc_network_served_tip_regression_total` | counter | project, network, lane, axis, outcome | [Regression-guard](/reference/evm/block-tracking.llms.txt#regression-guard) transitions, not evaluations. `outcome` ∈ `"held"` (serving the last corroborated pick), `"failed_open"` (the hold outlasted 1 minute). Any increase is alert-worthy. |
| `erpc_upstream_head_benched_total` | counter | project, vendor, network, upstream | Times this process [benched](/reference/evm/block-tracking.llms.txt#benching) the upstream's head out of the served-tip ballots and the lag reference: it stayed behind peers that kept advancing for its stall window. |

#### Area 4: Cache connector freshness

| Metric | Type | Labels | When it fires |
|---|---|---|---|
| `erpc_cache_connector_earliest_block_number` | gauge | connector, network | gRPC cache connector availability poll — earliest available block. |
| `erpc_cache_connector_latest_block_number` | gauge | connector, network | Same poll — latest block. |
| `erpc_cache_connector_finalized_block_number` | gauge | connector, network | Same poll — finalized block. |
| `erpc_cache_connector_earliest_block_timestamp_seconds` | gauge | connector, network | Same poll — earliest block unix timestamp. |
| `erpc_cache_connector_latest_block_timestamp_seconds` | gauge | connector, network | Same poll — latest block unix timestamp. |
| `erpc_cache_connector_finalized_block_timestamp_seconds` | gauge | connector, network | Same poll — finalized block unix timestamp. |

#### Area 5: Cache operations (JSON-RPC cache)

| Metric | Type | Labels | When it fires |
|---|---|---|---|
| `erpc_cache_get_success_hit_total` | counter | project, network, category, connector, policy, ttl | Cache get hit. |
| `erpc_cache_get_success_miss_total` | counter | project, network, category, connector, policy, ttl | Cache get miss. |
| `erpc_cache_get_error_total` | counter | project, network, category, connector, policy, ttl, error | Cache get errored. |
| `erpc_cache_get_skipped_total` | counter | project, network, category | Cache get skipped — no matching policy. |
| `erpc_cache_get_age_guard_reject_total` | counter | project, network, method, connector, policy, ttl | Cached item rejected: block-timestamp age exceeded policy TTL. |
| `erpc_cache_set_success_total` | counter | project, network, category, connector, policy, ttl | Cache set succeeded. |
| `erpc_cache_set_error_total` | counter | project, network, category, connector, policy, ttl, error | Cache set errored. |
| `erpc_cache_set_skipped_total` | counter | project, network, category, connector, policy, ttl | Cache set skipped by policy. |
| `erpc_cache_set_original_bytes_total` | counter | project, network, category, connector, policy, ttl | Uncompressed bytes written on cache set. |
| `erpc_cache_set_compressed_bytes_total` | counter | project, network, category, connector, policy, ttl | Compressed bytes written on cache set. |
| `erpc_cache_get_success_hit_duration_seconds` | LabeledHistogram | project, network, category, connector, policy, ttl | Cache get hit duration. |
| `erpc_cache_get_success_miss_duration_seconds` | LabeledHistogram | project, network, category, connector, policy, ttl | Cache get miss duration. |
| `erpc_cache_get_error_duration_seconds` | LabeledHistogram | project, network, category, connector, policy, ttl, error | Cache get error duration. |
| `erpc_cache_set_success_duration_seconds` | LabeledHistogram | project, network, category, connector, policy, ttl | Cache set success duration. |
| `erpc_cache_set_error_duration_seconds` | LabeledHistogram | project, network, category, connector, policy, ttl, error | Cache set error duration. |
| `erpc_ristretto_cache_current_cost` | gauge | connector | Ristretto (memory connector) current total cost. **Requires `memory.emitMetrics: true`** — always 0 otherwise. |
| `erpc_ristretto_cache_sets_failed_total` | counter | connector | Ristretto set dropped / rejected. |

#### Area 6: Rate limiting

| Metric | Type | Labels | When it fires |
|---|---|---|---|
| `erpc_rate_limits_total` | counter | project, network, vendor, upstream, category, finality, user, agent_name, budget, scope, auth, origin | Unified rate-limit event. `scope="remote", origin="upstream", budget="<remote>"` = upstream 429 passthrough. For local budget denials, `scope` = `ScopeString()` of the rule: a comma-joined set of enabled scope flags (e.g. `"user"`, `"network"`, `"ip"`, or combinations like `"user,network"`); `origin` = empty string. Idle series deleted by health-tracker sweep. |
| `erpc_rate_limiter_budget_max_count` | gauge | budget, method, scope | Budget's allowed req/s, set on rule creation or auto-tuner update. `scope` = `ScopeString()` comma-joined flags (e.g. `"user,network"`). |
| `erpc_rate_limiter_budget_decision_total` | counter | project, network, category, finality, user, agent_name, budget, method, scope, decision | **DEPRECATED / DORMANT** — registered but zero production call sites. Always 0. |
| `erpc_rate_limiter_failopen_total` | counter | project, network, user, agent_name, budget, category, reason | Rate limiter failed open. `reason` ∈ `"admission_full"`, `"limit_timeout"`. |
| `erpc_rate_limiter_remote_inflight` | gauge | budget | Concurrent in-flight remote (e.g. Redis) DoLimit calls per budget. |
| `erpc_rate_limiter_remote_admission_shedded_total` | counter | budget | Remote check fail-opened because admission semaphore full (load shed). |
| `erpc_rate_limiter_remote_duration_seconds` | LabeledHistogram | budget, result | Remote rate-limit check duration. `result` ∈ `"ok"`, `"over_limit"`, `"fail_open"`. Buckets: 0.001–5 s. |

#### Area 7: Upstream health (cordon, circuit-breaker, probing)

| Metric | Type | Labels | When it fires |
|---|---|---|---|
| `erpc_upstream_cordoned` | gauge | project, vendor, network, upstream, category, reason | 1 on cordon, 0 on uncordon. `category` = method string or `"*"` (wholesale). NOT the standard request-category label. `vendor` = `"n/a"` when unvendored. |
| `erpc_upstream_cordon_event_total` | counter | project, network, upstream, action | Admin cordon/uncordon. `action` ∈ `"cordon"`, `"uncordon"`. |
| `erpc_upstream_cordon_duration_seconds` | histogram | project, network, upstream | Seconds spent cordoned, observed on each uncordon. Buckets: 1–86400 s. |
| `erpc_upstream_breaker_state_change_total` | counter | project, upstream, transition | Circuit-breaker state transition. `transition` ∈ `"closed_to_open"`, `"half_open_to_open"`, `"half_open_to_closed"`, `"open_to_half_open"`. |
| `erpc_selection_probe_requests_total` | counter | network, upstream, method | Probe-mirror request fired at an excluded upstream. |
| `erpc_selection_probe_errors_total` | counter | network, upstream, method, reason | Probe request errored. `reason` ∈ `"timeout"`, `"throttled"`, `"auth"`, `"skipped"`, `"error"`. `"skipped"` = upstream intentionally rejected; `"error"` = other failures. |
| `erpc_selection_probe_skipped_total` | counter | network, reason | Probe candidate skipped pre-fire. `reason` ∈ `"write_method"`, `"opt_out"`, `"sampled_out"`, `"max_concurrent"`, `"no_method"`. |
| `erpc_selection_probe_dropped_total` | counter | network, reason | Probe-bus publish dropped — per-network feed channel full. |

#### Area 8: Selection policy

| Metric | Type | Labels | When it fires |
|---|---|---|---|
| `erpc_selection_position` | gauge | project, network, method, upstream | Tick output: 0=primary, 1+=runner-up, −1=excluded. |
| `erpc_selection_score` | gauge | project, network, method, upstream | Per-upstream `sortByScore` score (lower=better). Absent for upstreams that bypassed scoring. |
| `erpc_selection_eligible_upstreams` | gauge | project, network, method | Count of upstreams returned by most recent tick. |
| `erpc_selection_rejection_total` | counter | project, network, method, upstream, step | Tick rejected upstream at a std-lib step. |
| `erpc_selection_exclusion_total` | counter | project, network, method, upstream, reason | Exclusion event. `reason` = leaf-predicate slug. |
| `erpc_selection_shadow_exclusion_total` | counter | project, network, method, upstream, reason | `shadowExcludeIf` would-have-excluded; upstream stays in rotation. |
| `erpc_selection_excluded_seconds` | gauge | project, network, method, upstream | Wall-clock seconds continuously excluded. 0 when in rotation. |
| `erpc_selection_readmit_total` | counter | project, network, method, upstream | Excluded → in-list transition. |
| `erpc_selection_primary_switch_total` | counter | project, network, method, from, to | Primary upstream changed between ticks. |
| `erpc_selection_sticky_hold_total` | counter | project, network, method, upstream | `stickyPrimary` held a primary that would otherwise flip. |
| `erpc_selection_eval_duration_seconds` | histogram | project, network, method | Per-tick selection-policy eval latency. Buckets: 0.0005–1 s. |
| `erpc_selection_eval_errors_total` | counter | project, network, method, kind | Eval failure. `kind` ∈ `"timeout"`, `"throw"`, `"invalid_return"`, `"fallback_default"`. |
| `erpc_selection_readmit_age_seconds` | histogram | project, network, method | `now − excludedSince` at readmit. Buckets: 1–3600 s. |

#### Area 9: Consensus

| Metric | Type | Labels | When it fires |
|---|---|---|---|
| `erpc_consensus_total` | counter | project, network, category, outcome, finality, user, agent_name | Consensus round completed. `outcome` ∈ `"success"`, `"consensus_on_error"`, `"dispute"`, `"low_participants"`, `"generic_error"`, `"caller_abandoned"`. |
| `erpc_consensus_misbehavior_detected_total` | counter | project, network, upstream, category, finality, response_type, larger_than_consensus, user, agent_name | Upstream returned different data than consensus. `response_type` ∈ `"NonEmpty"`, `"Empty"`, `"ConsensusError"`, `"InfrastructureError"`. |
| `erpc_consensus_upstream_punished_total` | counter | project, network, upstream, user, agent_name | Upstream punished after misbehavior threshold. |
| `erpc_consensus_short_circuit_total` | counter | project, network, category, reason, finality, user, agent_name | Round short-circuited. `reason` ∈ `"sendrawtx_first_success"`, `"consensus_error_threshold"`, `"unassailable_lead"`. |
| `erpc_consensus_wait_capped_total` | counter | project, network, category, trigger, finality, user, agent_name | Round resolved early by `maxWaitOnResult`/`maxWaitOnEmpty`. `trigger` ∈ `"result"`, `"empty"`. |
| `erpc_consensus_errors_total` | counter | project, network, category, error, finality, user, agent_name | Consensus-level error by type. |
| `erpc_consensus_upstream_errors_total` | counter | project, network, upstream, category, finality, response_type, error_code, user, agent_name | Participant upstream errored during consensus. `response_type` same values as misbehavior_detected. |
| `erpc_consensus_panics_total` | counter | project, network, category, finality, user, agent_name | Panic recovered inside consensus. |
| `erpc_consensus_cancellations_total` | counter | project, network, category, phase, finality, user, agent_name | Context cancelled during consensus. `phase` ∈ `"before_execution"`, `"after_execution"`, `"caller_abandoned"`. |
| `erpc_consensus_duration_seconds` | LabeledHistogram | project, network, category, outcome, finality, user, agent_name | Consensus round duration. |
| `erpc_consensus_responses_collected` | LabeledHistogram | project, network, category, vendors, short_circuited, finality, user, agent_name | Responses collected before decision. `vendors` = comma-joined sorted vendor names. `short_circuited` ∈ `"true"`/`"false"`. Linear buckets 1–10. |
| `erpc_consensus_agreement_count` | LabeledHistogram | project, network, category, finality, user, agent_name | Upstreams agreeing on most common result. Linear buckets 1–10. |

#### Area 10: Hedging

| Metric | Type | Labels | When it fires |
|---|---|---|---|
| `erpc_network_hedged_request_total` | counter | project, network, upstream, category, attempt, finality, user, agent_name | Hedged request fired. |
| `erpc_network_hedge_discards_total` | counter | project, network, upstream, category, attempt, hedge, finality, user, agent_name | Hedged request discarded (wasted work; losing leg cancelled). |
| `erpc_network_hedge_winner_total` | counter | project, network, upstream, category, finality | Hedge race won by the upstream whose response was kept. |
| `erpc_network_hedge_delay_seconds` | LabeledHistogram | project, network, category, finality | **DORMANT** — registered but no production observe site. |

#### Area 11: EVM method-specific

| Metric | Type | Labels | When it fires |
|---|---|---|---|
| `erpc_network_evm_get_logs_split_success_total` | counter | project, network, user, agent_name | A split `eth_getLogs` sub-request succeeded. |
| `erpc_network_evm_get_logs_split_failure_total` | counter | project, network, user, agent_name | A split `eth_getLogs` sub-request failed. |
| `erpc_network_evm_get_logs_forced_splits_total` | counter | project, network, dimension, user, agent_name | `eth_getLogs` forcibly split. `dimension` ∈ `"block_range"`, `"addresses"`, `"topics0"` (topics[0] OR-array only). |
| `erpc_network_evm_trace_filter_split_success_total` | counter | project, network, method, user, agent_name | Split `trace_filter`/`arbtrace_filter` sub-request succeeded. |
| `erpc_network_evm_trace_filter_split_failure_total` | counter | project, network, method, user, agent_name | Split `trace_filter`/`arbtrace_filter` sub-request failed. |
| `erpc_network_evm_trace_filter_forced_splits_total` | counter | project, network, method, dimension, user, agent_name | `trace_filter` split. `dimension` ∈ `"block_range"`, `"from_address"`, `"to_address"`. |
| `erpc_network_evm_block_range_requested_total` | counter | project, network, vendor, upstream, category, user, finality, bucket, size | Block-range heatmap. `bucket` uses tip-relative labels (`"TIP"`, `"L100k"`, `"100k-200k"`, etc.) when tip is known; falls back to static 100000-block aligned labels otherwise. **Not covered by idle sweep — cardinality is unbounded.** |
| `erpc_network_evm_get_logs_range_requested` | LabeledHistogram | project, network, category, user, finality | `eth_getLogs` requested block-range size (observed value = `toBlock − fromBlock`). Buckets: 1, 10, 100, 500, 1000, 5000, 10000, 30000. |
| `erpc_network_evm_trace_filter_range_requested` | LabeledHistogram | project, network, method, user, finality | `trace_filter`/`arbtrace_filter` requested block-range size. Same buckets. |

#### Area 12: gRPC BDS resilience

| Metric | Type | Labels | When it fires |
|---|---|---|---|
| `erpc_grpc_bds_hard_timeout_total` | counter | project, upstream, method | BDS gRPC call hit the bounded-wait hard timeout (20 s, hard-coded). Non-zero rate indicates H2 stream wedging; the watchdog then force-replaces the connection. |
| `erpc_grpc_bds_conn_replacements_total` | counter | project, upstream | BDS pool connection force-closed by stuck-call watchdog. |

#### Area 13: CORS

| Metric | Type | Labels | When it fires |
|---|---|---|---|
| `erpc_cors_requests_total` | counter | project, origin | Request carrying an `Origin` header. **Note:** `project` label = `r.URL.Path` (e.g. `/myproject/evm/1`), not the project ID. |
| `erpc_cors_preflight_requests_total` | counter | project, origin | Allowed-origin OPTIONS preflight. Same `project` mislabeling. |
| `erpc_cors_disallowed_origin_total` | counter | project, origin | Request from disallowed origin. Same `project` mislabeling. |

#### Area 14: Shadow testing

| Metric | Type | Labels | When it fires |
|---|---|---|---|
| `erpc_shadow_response_identical_total` | counter | project, vendor, network, upstream, category | Shadow upstream response identical to expected. |
| `erpc_shadow_response_mismatch_total` | counter | project, vendor, network, upstream, category, finality, emptyish, larger | Shadow response differs. `emptyish` ∈ `"true"`/`"false"` (shadow was empty-ish). `larger` ∈ `"true"`/`"false"` (shadow body larger than primary). |
| `erpc_shadow_response_error_total` | counter | project, vendor, network, upstream, category, error | Shadow upstream request errored. |

#### Area 15: Auth

| Metric | Type | Labels | When it fires |
|---|---|---|---|
| `erpc_auth_failed_total` | counter | project, network, strategy, reason, agent_name | Client-facing auth denial. Closed `reason` set; `agent_name` always `unknown`; no `user` label. |

#### Area 16: Panics

| Metric | Type | Labels | When it fires |
|---|---|---|---|
| `erpc_unexpected_panic_total` | counter | scope, extra, error | Recovered panic. `scope` ∈ `"request-handler"`, `"final-error-writer"`, `"top-level-handler"`, `"timeout-handler"`, `"validate-pattern"`, `"redis-pubsub"`, `"shared-state-registry"`, `"matcher"`. |

---

### Histogram bucket sets

| Bucket set | Values | Applies to |
|---|---|---|
| `DefaultHistogramBuckets` (overridable via `metrics.histogramBuckets`) | 0.05, 0.5, 5, 30 s | `upstream_request_duration_seconds`, `network_request_duration_seconds`, `consensus_duration_seconds`, all 5 cache duration histograms (8 total) |
| `EvmGetLogsRangeHistogramBuckets` | 1, 10, 100, 500, 1000, 5000, 10000, 30000 blocks | `network_evm_get_logs_range_requested`, `network_evm_trace_filter_range_requested` |
| `CatchUpWaitHistogramBuckets` | 0.1, 0.25, 0.5, 1, 2, 4, 8, 16, 32, 64 s | `network_data_unavailable_wait_seconds` |
| hedge delay (hard-coded) | 0.01, 0.03, 0.05, 0.2, 0.3, 0.5, 0.7, 1, 3 s | `network_hedge_delay_seconds` (dormant) |
| timeout duration (hard-coded) | 0.05, 0.1, 0.3, 0.5, 1, 3, 5, 10, 30 s | `network_timeout_duration_seconds` |
| selection eval (hard-coded) | 0.0005, 0.001, 0.002, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1 s | `selection_eval_duration_seconds` |
| readmit age (hard-coded) | 1, 5, 15, 30, 60, 120, 300, 600, 1800, 3600 s | `selection_readmit_age_seconds` |
| cordon duration (hard-coded) | 1, 10, 60, 300, 900, 1800, 3600, 7200, 21600, 86400 s | `upstream_cordon_duration_seconds` |
| rate-limiter remote (hard-coded) | 0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2, 5 s | `rate_limiter_remote_duration_seconds` |
| response size (hard-coded) | 4096, 65536, 1048576, 16777216, 104857600 bytes | `upstream_response_size_bytes` |
| consensus counts (linear) | 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 | `consensus_responses_collected`, `consensus_agreement_count` |

Only `DefaultHistogramBuckets` are configurable at runtime. All other bucket sets require a recompile.

---

### Observability

**The metrics endpoint:** `http://<host>:<metrics.port>/` (any path) — the Prometheus handler is mounted at the root, so every HTTP path on port 4001 returns the full Prometheus text exposition. Plain HTTP only, no auth, no path routing. Reference scrape config at [`monitoring/prometheus/prometheus.yml`](https://github.com/erpc/erpc/blob/main/monitoring/prometheus/prometheus.yml) (scrape interval 10 s, default target `host.docker.internal:4001`).

**Stock collectors** (present unless `ERPC_NOMETRICS=1`): `go_*` — Go runtime stats; `process_*` — OS process stats; `promhttp_metric_handler_*` — handler self-scrape stats.

**Bundled alerting rules** (`monitoring/prometheus/alert.rules`):

| Alert | Expression | Threshold |
|---|---|---|
| HighErrorRate | `rate(erpc_upstream_request_errors_total[5m]) / rate(erpc_upstream_request_total[5m]) > 0.05` by upstream | 5% error rate for 5 min |
| SlowRequests | `histogram_quantile(0.95, rate(erpc_upstream_request_duration_seconds_bucket[5m])) > 1` by upstream | p95 &gt; 1 s for 5 min |
| HighRateLimiting | references stale metric `erpc_upstream_request_self_rate_limited_total` — use `erpc_rate_limits_total` instead | — |
| NetworkRateLimiting | references stale metric `erpc_network_request_self_rate_limited_total` — use `erpc_rate_limits_total` instead | — |
| HighRequestRate | `rate(erpc_upstream_request_total[5m]) by (upstream) > 1000` | &gt;1000 rps per upstream |
| LowRequestRate | `rate(erpc_upstream_request_total[5m]) by (upstream) < 1` | &lt;1 rps per upstream for 15 min |

**Metrics server log lines** (all emitted by `erpc/init.go`):

| Level | Message |
|---|---|
| Info | `"starting metrics server on port: %d"` |
| Error | `"error starting metrics server: %s"` |
| Info | `"shutting down metrics server..."` |
| Error | `"metrics server forced to shutdown: %s"` |
| Info | `"metrics server stopped"` |
| Error | `"failed to apply metrics configuration; no metric families are registered"` — a malformed `metrics.customizations` entry. `telemetry.Configure` returns before registering anything, so `/metrics` serves only the stock `go_`/`process_` collectors. Alert on this. |
| Warn | `"failed to apply metrics configuration; falling back to default histogram buckets"` — an unparseable `metrics.histogramBuckets` float. Every family is still registered; only the bucket boundaries are wrong. |
| Info | `"metric customizations applied"` (with `customizations`/`exposed`/`total` counts; only when `metrics.customizations` is set) |
| Warn | `"metrics.customizations subjects match no known metric family; check for typos"` |
| Warn | `"some metrics.customizations rules do not apply to the family they name"` |

Both `"failed to apply metrics configuration"` lines are only reachable when `erpc.Init`
is called as a library: the CLI rejects a malformed customization and a bad bucket list in
`MetricsConfig.Validate` before `Init` runs. The wrapped error names the offending field
either way.

---

### Source code entry points

- [`telemetry/manager.go`](https://github.com/erpc/erpc/blob/main/telemetry/manager.go) — the definition index; the `Define*` factories; `Configure`; `SetHistogramBuckets`; `KnownFamilies`; `Gatherer`
- [`telemetry/policy.go`](https://github.com/erpc/erpc/blob/main/telemetry/policy.go) — `NormalizeMetricName`; `Customization`; `MetricPolicy` (subject/label precedence, per-family buckets, legacy desugaring)
- [`telemetry/gather_filter.go`](https://github.com/erpc/erpc/blob/main/telemetry/gather_filter.go) — `FilteredGatherer`, which applies the policy's drop rules to collectors the manager does not own
- [`telemetry/metrics.go`](https://github.com/erpc/erpc/blob/main/telemetry/metrics.go) — all 141 metric definitions; bucket constants; `DefaultHistogramBuckets`; `ParseHistogramBuckets`
- [`telemetry/labeled_histogram.go:L16-L134`](https://github.com/erpc/erpc/blob/main/telemetry/labeled_histogram.go#L16-L134) — `LabeledHistogram` with `WithLabelValues`, `DeleteLabelValues`, `ActiveLabelValues`, `rebuildInPlace`
- [`telemetry/handles.go:L59-L105`](https://github.com/erpc/erpc/blob/main/telemetry/handles.go#L59-L105) — `CounterHandle`/`GaugeHandle`/`ObserverHandle` caches; `ResetHandleCache`
- [`telemetry/labeled_counter.go:L14-L122`](https://github.com/erpc/erpc/blob/main/telemetry/labeled_counter.go#L14-L122) — `LabeledCounter` with `WithLabelValues`, `DeleteLabelValues`, `ActiveLabelValues`, `rebuildInPlace`
- [`erpc/init.go`](https://github.com/erpc/erpc/blob/main/erpc/init.go) — initialization sequence: `MetricsConfig.TelemetryOptions` → `telemetry.Configure` → unmatched/ignored-customization warnings → network alias resolver → metrics HTTP server lifecycle
- [`common/config.go`](https://github.com/erpc/erpc/blob/main/common/config.go) — `MetricsConfig`, `MetricsCustomizationConfig` and `MetricLabelCustomizationConfig` structs; `TelemetryOptions`
- [`common/defaults.go`](https://github.com/erpc/erpc/blob/main/common/defaults.go) — `MetricsConfig.SetDefaults`
- [`common/errors.go:L17-L114`](https://github.com/erpc/erpc/blob/main/common/errors.go#L17-L114) — `errorLabelMode` global; `SetErrorLabelMode`; `ErrorSummary`
- [`health/tracker.go:L481-L667`](https://github.com/erpc/erpc/blob/main/health/tracker.go#L481-L667) — `DefaultIdleEvictionAfter`; `rotateMetricsLoop`; `sweepIdle`; `sweepIdleObservers`
- [`cmd/erpc/initflags.go:L22-L28`](https://github.com/erpc/erpc/blob/main/cmd/erpc/initflags.go#L22-L28) — `ERPC_NOMETRICS=1` registry replacement
- [`monitoring/catch-up-metrics.md`](https://github.com/erpc/erpc/blob/main/monitoring/catch-up-metrics.md) — operator guide for reading `erpc_network_retry_attempt_total` + `erpc_network_data_unavailable_wait_seconds` together; covers `CatchUpWaitHistogramBuckets` design rationale, finality-label split, Little's Law pressure interpretation, and tuning knobs

### Related pages

- [Rate limiters](/config/rate-limiters.llms.txt) — the `erpc_rate_limits_total` counter tracks every budget decision.
- [Failsafe: Hedge](/config/failsafe/hedge.llms.txt) — `erpc_network_hedged_request_total` and `erpc_network_hedge_winner_total` are the hedge observability surface.
- [Failsafe: Retry](/config/failsafe/retry.llms.txt) — `erpc_network_retry_attempt_total` with its `reason` label tracks retry behavior.
- [Selection policies](/config/projects/selection-policies.llms.txt) — the `erpc_selection_*` family tracks tick-by-tick upstream ranking.
- [Cache](/config/database.llms.txt) — `erpc_cache_get_*` and `erpc_cache_set_*` families cover all cache connector activity.
- [Auth](/config/auth.llms.txt) — `erpc_auth_failed_total` for all auth strategy failures.

---


## Navigation (machine-readable surface)

- Up: [All pages index](https://docs.erpc.cloud/llms.txt)
- Root index of every page: [llms.txt](https://docs.erpc.cloud/llms.txt) · everything in one file: [llms-full.txt](https://docs.erpc.cloud/llms-full.txt)

### Sibling pages

- [Error taxonomy](https://docs.erpc.cloud/reference/errors.llms.txt) — Every error eRPC can emit — typed, categorized, with retryability flags, wire HTTP status, and JSON-RPC codes — so you can interpret metrics, write alerts, and debug routing decisions confidently.
- [gRPC & BDS streaming](https://docs.erpc.cloud/reference/grpc-bds.llms.txt) — Use typed protobuf APIs for block, transaction, and log lookups — eRPC routes, caches, and protects every gRPC call exactly like HTTP, with built-in deadlock defenses that keep stuck H2 streams from stalling your traffic.
- [HTTP Client & Proxy Pools](https://docs.erpc.cloud/reference/http-client.llms.txt) — eRPC keeps a single pre-warmed, high-throughput connection to every upstream — and can rotate traffic across a fleet of SOCKS5 or HTTP proxies with zero extra latency.
- [Lanes & concurrency](https://docs.erpc.cloud/reference/lanes.llms.txt) — Route a class of requests to a specific provider group and eRPC computes that group's own block tip — eliminating "block not found" churn caused by cross-provider tip pollution.
- [Simulator](https://docs.erpc.cloud/reference/simulator.llms.txt) — A local browser playground that runs a real eRPC instance against synthetic upstreams — explore routing, failsafe, and selection-policy behavior in seconds, no credentials needed.