Healthcheck

How it works

Trigger paths. Any HTTP request that is not POST or OPTIONS is treated as a healthcheck by the URL parser. A trailing /healthcheck path segment also forces the healthcheck branch regardless of method. Valid probe paths, from broadest to narrowest scope:

• GET / — global (all projects)
• GET /healthcheck — same as above
• GET /<projectId>/healthcheck — per-project
• GET /<projectId>/<architecture>/<chainId>/healthcheck — per-network (e.g. GET /main/evm/1/healthcheck)

For domain-aliased deployments where a domain preselects all three of project + arch + chain, GET / resolves to a per-network probe after alias expansion.

Draining guard. Before any evaluation the handler checks a draining flag. When the server context is cancelled (SIGTERM received), draining flips to true and the endpoint immediately returns HTTP 503 with plain-text body "shutting down". This is the Kubernetes readiness-probe hook: the pod is removed from the load balancer pool while in-flight requests complete. Because 503 means "draining" and 502 means "unhealthy", Kubernetes can distinguish the two states and avoid triggering a pod restart during normal rollout.

Auth. When healthCheck.auth is configured, an independent AuthRegistry is created at startup. On every probe call the registry authenticates using the real client IP (resolved from trusted-proxy headers). Auth is completely separate from per-project auth — monitoring systems can reach the healthcheck without sharing application credentials.

Scope resolution. With no projectId, the handler fetches all registered projects (HTTP 500 only if zero projects exist). With projectId set, a missing project returns the same 404 structure as the proxy path. When both architecture and chainId are in the URL, only that network is evaluated and the project result mirrors it; otherwise evaluateProjectHealth aggregates across all upstreams.

Lazy upstream initialization. When a per-network probe arrives and no upstreams are registered for that network yet, the handler calls PrepareUpstreamsForNetwork — the exact same path used by the first real RPC request. This prevents a false-negative "no upstreams initialized" on the very first probe after pod startup. The first probe for a new network may take hundreds of milliseconds; subsequent probes are fast.

Provider-only project shortcut. If a project has providers configured but zero statically-defined upstreams and zero initialized upstreams, the handler marks the project healthy immediately (message: "no upstreams initialized yet, and no networks configured, but there are providers configured, send first actual request to initialize the upstreams"). This prevents provider-based projects from appearing unhealthy before the first real RPC call.

Evaluation strategies. The ?eval= query parameter (or healthCheck.defaultEval, or the hard-coded default any:initializedUpstreams) selects the evaluation strategy. An unknown eval value returns HTTP 502 with "unknown evaluation strategy: <value>" — not HTTP 400. The ?eval= parameter always overrides healthCheck.defaultEval per-request.

Error-rate strategies. These compute errorsTotal / requestsTotal from the health tracker's *-method and DataFinalityStateAll bucket. Only upstreams with at least one request are included; upstreams with zero requests are completely excluded — they do not count toward threshold violations, but they also do not count as healthy for the all:… variants. If all upstreams have zero traffic, the result is unhealthy: "no error rate data available yet".

Chain-ID strategy. checkEvmChainId fans out concurrent eth_chainId RPC calls (semaphore capped at 10, 5-second per-upstream timeout) and compares each response to ups.Config().Evm.ChainId. For any:, one success passes the probe; for all:, every upstream must pass. With partial success under any:, the message is "N / M upstreams passed (K failed)".

Active-upstreams strategy. all:activeUpstreams checks: (1) at least one upstream or provider is configured; (2) all statically declared upstreams are initialized; (3) none are cordoned. For provider-only setups, check (2) is skipped. When evaluating a specific network, only upstreams whose chain ID matches are counted — cross-network upstreams are not penalized.

Response modes. Controlled by healthCheck.mode:

• simple — HTTP 200 with plain ASCII OK; HTTP 502 with a JSON-RPC ErrHealthCheckFailed error body on failure.
• networks — HTTP 200/502 with Content-Type: application/json; body {"projectId": [{id, alias, blockTimeMs, state}]}.
• verbose — Full JSON {status, message, details} with per-upstream metrics and EVM diagnostics. Metrics use *-wildcard aggregate (all methods, all finality states); method-level breakdown requires Prometheus.

HTTP status code semantics:

Config schema

All fields are under healthCheck. at the top level of the eRPC config. Struct at common/config.go:L186-208.

Eval strategy constants (for healthCheck.defaultEval or ?eval= query parameter). Source: common/config.go:L200-208.

Worked examples

All patterns below are distilled from real production fleets; comments explain the non-obvious choices.

1. The production standard: verbose mode + 30s drain window (high-volume fleet). Every Some production deployments use verbose mode so internal dashboards can inspect per-upstream metrics on a single URL, paired with a 30s drain window that safely outlasts the longest in-flight RPC batch:

server:  # drain window must exceed readinessProbe.periodSeconds × failureThreshold  # so the pod is fully deregistered before connections close  waitBeforeShutdown: 30s  waitAfterShutdown: 30s
healthCheck:  # verbose exposes per-upstream metrics + evmDiagnostics in a single response —  # no Prometheus scrape needed for incident triage  mode: verbose

2. Kubernetes startup + readiness split (recommended general shape). Traffic-facing clusters separate startup (just needs an upstream registered) from readiness (needs live traffic flowing). This avoids a 502 storm on cold start before any upstream has been hit:

server:  # 30s outlasts the readiness window (5s × 2 + 20s buffer)  waitBeforeShutdown: 30s
healthCheck:  # networks mode is cheaper for k8s probes — no per-upstream JSON overhead  mode: networks  # default: any:initializedUpstreams — safe at cold start, used by readiness  defaultEval: "any:initializedUpstreams"  auth:    strategies:      - type: network        network:          # kubelet probes originate from the node IP, not 127.0.0.1 —          # allowLocalhost: true alone is not enough, add node/pod CIDRs          allowLocalhost: true          allowedCIDRs:            - "10.0.0.0/8"

Kubernetes probe manifest to pair with the config above:

startupProbe:
  httpGet:
    path: /healthcheck?eval=any:initializedUpstreams
    port: 4000
  initialDelaySeconds: 10
  periodSeconds: 10
  timeoutSeconds: 5
  # allow up to 60s for the very first upstream to initialize
  failureThreshold: 6
 
readinessProbe:
  httpGet:
    # uses defaultEval from config (any:initializedUpstreams)
    path: /healthcheck
    port: 4000
  periodSeconds: 5
  timeoutSeconds: 5
  failureThreshold: 2
  successThreshold: 1
 
livenessProbe:
  # TCP socket — never use httpGet here: 503 during normal drain triggers restart
  tcpSocket:
    port: 4000
  initialDelaySeconds: 30
  periodSeconds: 10
  failureThreshold: 3

3. Strict chain-ID verification before routing traffic. Use when upstreams could silently be on the wrong chain — after a provider migration, when serving high-value transactions, or any time a misconfigured chain would cause irreversible damage. The fan-out runs at concurrency 10 with a 5s per-upstream timeout:

readinessProbe:
  httpGet:
    # all: variant: every upstream must return the expected chain ID
    path: /healthcheck?eval=all:evm:eth_chainId
    port: 4000
  periodSeconds: 15
  # budget ceil(N_upstreams / 10) × 5s + 2s margin
  # e.g. 6 upstreams → ceil(6/10)×5 + 2 = 7s; use 15 for safety
  timeoutSeconds: 15
  failureThreshold: 2

4. Multi-region edge with per-network probe. Multi-region edge clusters use networks mode so load-balancer health checks see a compact JSON list. The per-network URL lets the LB probe only the chain it cares about, avoiding false positives from an unrelated chain's degradation:

healthCheck:  # compact JSON list: {"projectId": [{id, alias, blockTimeMs, state}]}  # fast for LB health checks; verbose is too heavy at LB poll frequency  mode: networks

Probe URL (narrows to one chain — first call for an uninitialized network triggers lazy init, budget extra timeoutSeconds):

GET /myproject/evm/1/healthcheck?eval=any:evm:eth_chainId

5. Verbose diagnostics endpoint for monitoring with RFC-1918 auth. Internal SRE dashboards and alert rules need per-upstream detail without sharing application API keys. Lock to private CIDRs — both allowLocalhost and RFC-1918 ranges are needed because monitoring agents typically run on a pod IP, not 127.0.0.1:

healthCheck:  mode: verbose  # all:activeUpstreams fails if any declared upstream is uncordoned but uninitialized  # — stronger than error-rate strategies which require prior traffic  defaultEval: "all:activeUpstreams"  auth:    strategies:      - type: network        network:          allowLocalhost: true          allowedCIDRs:            - "10.0.0.0/8"            - "172.16.0.0/12"            - "192.168.0.0/16"

Request/response behavior

Response shape — networks mode (HTTP 200):

{
  "projectId": [
    {"id": "evm:1", "alias": "mainnet", "blockTimeMs": 12000.0, "state": "OK"},
    {"id": "evm:42161", "state": "ERROR"}
  ]
}

Response shape — verbose mode (HTTP 200):

{
  "status": "OK",
  "message": "all systems operational",
  "details": {
    "projectId": {
      "status": "OK",
      "message": "...",
      "config": {"networks": 2, "upstreams": 3, "providers": 0},
      "initializer": {"...": "InitializerStatus"},
      "upstreams": {
        "upsId": {
          "network": "evm:1",
          "metrics": {"...": "aggregate across all methods"},
          "evmDiagnostics": {"...": "EVM state poller data"}
        }
      },
      "networks": {
        "evm:1": {"status": "OK", "alias": "mainnet", "blockTimeMs": 12000.0}
      }
    }
  }
}

Response shape — simple mode, unhealthy (HTTP 502):

The body is a structured ErrHealthCheckFailed JSON-RPC error — not a plain string. Clients that parse JSON-RPC errors get structured data even in simple mode.

Drain response (HTTP 503, any mode):

http.Error(w, "shutting down", 503) sets Content-Type: text/plain. Clients that always expect JSON will fail to parse this. This is intentional — it must be distinguishable from a normal 502 unhealthy response.

Best practices

• Use separate strategies for startup vs. readiness probes: any:initializedUpstreams for startup (no traffic required), any:errorRateBelow90 for readiness (requires live traffic). Mixing them leads to either false-positive failures at startup or missed degradation in production.
• Never put the healthcheck path on a liveness probe. The endpoint returns 503 during normal graceful shutdown, which triggers pod restart on every rollout. Use a TCP socket liveness probe instead.
• Add node/pod CIDRs to allowedCIDRs when using healthCheck.auth — kubelet probes originate from the node IP, not 127.0.0.1, so allowLocalhost: true alone will silently reject them.
• Budget extra timeoutSeconds when using any:evm:eth_chainId or all:evm:eth_chainId with many upstreams: the fan-out runs at concurrency 10 with a 5s per-upstream timeout, so worst-case latency is ceil(N/10) × 5s.
• Set waitBeforeShutdown ≥ periodSeconds × failureThreshold + 1s on your readiness probe. Without this, traffic may still be routed to the pod after 503 has been returned, causing request failures during rollout.
• Use mode: verbose only for monitoring dashboards or debugging. In high-cardinality deployments the verbose response can be large; networks mode is sufficient for Kubernetes probes.
• For provider-based projects (upstreams created dynamically on first RPC call), any:initializedUpstreams will return unhealthy until the first real request triggers initialization. Accept this false-negative window or use the startup probe to absorb it.

Edge cases & gotchas

1. Draining returns 503 plain text, not JSON. http.Error(w, "shutting down", 503) sets Content-Type: text/plain. Clients that always expect JSON will fail to parse this response. This is the readiness-probe hook — liveness probes on /healthcheck will trigger pod restart during normal graceful shutdown.
2. any:initializedUpstreams does not confirm the upstream can serve requests. An upstream registers even if its chain-ID probe is still pending or has failed. Use all:activeUpstreams or a chain-ID strategy for stronger guarantees. Source: erpc/healthcheck.go:L59-77
3. Error-rate strategies require prior traffic. Fresh instances return "no error rate data available yet" (HTTP 502) until at least one upstream has processed a request. Use any:initializedUpstreams for startup probes. Source: erpc/healthcheck.go:L465-471
4. First per-network probe may be slow. PrepareUpstreamsForNetwork triggers internal chain-ID RPC probes on first call; allow extra timeoutSeconds on the startup probe for this network initialization. Source: erpc/healthcheck.go:L175-181
5. Global healthcheck fails if no projects are registered. If all projects fail to register at startup, GET /healthcheck returns HTTP 500 even though the HTTP server itself is up. Source: erpc/healthcheck.go:L122-127
6. Unknown eval value returns HTTP 502, not HTTP 400. evaluateNetworkHealth falls through to its default: branch — watch for the exact prefix "unknown evaluation strategy: " when debugging misconfigured eval strings. Source: erpc/healthcheck.go:L550-551
7. Healthcheck auth always exposes full error details. Unlike most proxy errors that respect server.includeErrorDetails, project/network resolution errors from the healthcheck handler always include the project ID in the error body. Source: erpc/healthcheck.go:L130-133
8. Duplicate network alias is silently dropped. If two NetworkConfig entries share the same alias, only the first registration wins; the second is dropped with a WARN log. Source: erpc/networks_registry.go:L340-344
9. all:activeUpstreams with provider-only projects bypasses the initialization check. Even if the provider has 0 initialized upstreams, hasUninitializedUpstreams is forced false. The probe can still fail if 0 upstreams AND 0 providers are configured. Source: erpc/healthcheck.go:L663-670
10. verbose mode metrics use * wildcard aggregate. metricsTracker.GetUpstreamMethodMetrics(ups, "*", DataFinalityStateAll) is rolled up across all methods and finality states. Method-level breakdown requires Prometheus. Source: erpc/healthcheck.go:L259-262
11. any:evm:eth_chainId with partial success is healthy. If some upstreams fail but at least one passes, the result is HTTP 200 with message "N / M upstreams passed (K failed)". Under all:, any single failure makes the probe unhealthy. Source: erpc/healthcheck.go:L896-905
12. LastEvalAt is non-zero after Bootstrap. The health tracker records the timestamp of the most recent evaluation in LastEvalAt. Health-check exporters and external probers can use this field to detect stale selection state (no policy evaluation tick has occurred since startup). Source: erpc/healthcheck_test.go:L26-L45 (opens in a new tab)

Observability

The healthcheck handler does not emit any dedicated Prometheus metrics. The general erpc_unexpected_panic_total counter fires if the handler panics.

OTel tracing: handleHealthCheck runs within the Http.ReceivedRequest span started by the top-level handler. common.EnrichHTTPServerSpan sets the http.status_code attribute before the response is written.

Log messages:

• "entering draining mode → healthcheck will fail" — INFO when app context is cancelled
• "failed to encode health check response" — ERROR on JSON encoding failure
• "registered network alias" — DEBUG on each successful alias registration. Source: erpc/networks_registry.go:L347 (opens in a new tab)
• "skipping duplicate alias registration with different target" — WARN on duplicate network alias

Source code entry points

• erpc/healthcheck.go:L1-L913 (opens in a new tab) — handleHealthCheck, evaluateNetworkHealth, evaluateProjectHealth, formatHealthDataForMode, checkEvmChainId; all response struct definitions
• erpc/http_server.go:L78-L83 (opens in a new tab) — draining flag set when app context is done; INFO log "entering draining mode → healthcheck will fail"
• erpc/http_server.go:L201-L207 (opens in a new tab) — healthcheck auth registry construction at startup
• erpc/http_server.go:L836-L849 (opens in a new tab) — URL path parsing: healthcheck detection logic (non-POST/OPTIONS + trailing /healthcheck)
• common/config.go:L186-L208 (opens in a new tab) — HealthCheckConfig struct, HealthCheckMode constants, eval-strategy constants
• common/defaults.go:L738-L741 (opens in a new tab) — HealthCheckConfig.SetDefaults; canonical default mode "networks"
• erpc/healthcheck_test.go (opens in a new tab) — integration tests for all eval strategies, drain mode, auth, provider-only, specific network scoping
• erpc/networks_registry.go:L247-L255 (opens in a new tab) — ResolveAlias used to expand per-project network aliases during healthcheck path parsing

Related pages

• Auth — same strategy schema used for healthCheck.auth
• Server configuration — server.waitBeforeShutdown must be tuned alongside readiness probe thresholds
• Rate limiters — independent from healthcheck but affects upstream error rates that feed into errorRate* eval strategies
• Deployment — Kubernetes deployment patterns including probe configuration
• Observability — Prometheus metrics for upstream health beyond what healthcheck exposes