Production guidelines

Memory / GC tuning

eRPC is a Go process. The runtime's default GC target (GOGC=100) is appropriate for development but often too loose for containers with hard memory limits.

Recommended production pair:

GOGC=30          # run GC after heap grows 30 % — smaller heap, more frequent collections
GOMEMLIMIT=2GiB  # soft ceiling — GC fires when RSS nears this value

Set GOMEMLIMIT to ~80 % of your container memory limit. For example: 2 GiB limit → GOMEMLIMIT=1600MiB. Setting it equal to the limit leaves no headroom and risks GC thrash or OOM from transient allocation bursts.

Caution: GOGC < 10 causes GC thrashing — the runtime spends most CPU collecting, not serving requests. Values of 20–50 are the practical floor.

If you have abundant RAM and want to reduce CPU overhead, raise GOGC (e.g. 200). The heap will grow larger but GC runs less often.

Healthcheck rollout pattern

eRPC's shutdown sequence:

1. Receive SIGTERM.
2. Stop accepting new connections (waitBeforeShutdown delay — readiness probe starts failing).
3. Drain in-flight requests.
4. Wait waitAfterShutdown (keeps the process alive so the LB/proxy can close open connections).
5. Exit 0.

For Kubernetes with Cilium/Envoy, both values should be at least 30 s:

server:
  waitBeforeShutdown: 30s
  waitAfterShutdown:  30s

The readiness probe should return unhealthy within 10 s of SIGTERM (before waitBeforeShutdown expires) so the orchestrator removes the endpoint before connections are refused.

Kubernetes terminationGracePeriodSeconds must be greater than waitBeforeShutdown + waitAfterShutdown + time to drain. Set it to at least 90 s for the 30 s + 30 s pattern above.

responseHeaders for instance identification

server.responseHeaders is a map of header name → value. Values support \${VAR} env-var expansion. Headers with an empty value after expansion are silently omitted (safe to use with optional env vars).

Useful headers:

Combine with tracing resource attributes (tracing.resourceAttributes) so every trace span carries the same instance label as the HTTP response header.

includeErrorDetails

Controls whether upstream error messages and internal endpoint information appear in JSON-RPC error responses returned to callers.

• Default: true (errors are verbose — helpful for development).
• Production: set to false to prevent leaking upstream URLs, API key fragments, and internal error strings.

Errors are still logged internally at full verbosity regardless of this setting.

trustedIPForwarders + trustedIPHeaders

When eRPC sits behind a reverse proxy, load-balancer, or CDN, the TCP source IP is always the proxy's address. To recover the real client IP:

1. trustedIPForwarders — list of CIDR blocks (or individual IPs) whose X-Forwarded-For (or the named headers) are trusted. Requests from outside these ranges have their forwarded headers ignored.
2. trustedIPHeaders — ordered list of headers to read. eRPC picks the first header that is present on a request from a trusted forwarder.

This real IP is then used for:

• IP-based rate limiting (network auth strategy allowedIPs)
• Per-IP metric labels
• Any upstream selection that keys on client IP

Without this config, all requests appear to originate from your LB IP and IP-based policies are effectively global.

Metrics tuning

See Monitoring for metrics.histogramDropLabels — dropping high-cardinality label combinations (e.g. per-upstream request-size histograms) avoids cardinality explosion in Prometheus.

Tracing in production

See Tracing for OTLP exporter setup, sampling rate config, and adding custom resource attributes. Use tracing.resourceAttributes to attach region/instance labels that correlate with the responseHeaders you set above.

Rate-limit budgets per project / upstream

See Rate Limiters for rateLimiters.budgets — define per-project or per-upstream budgets and reference them from auth strategies (per-API-key limits) or directly from upstream config (cap upstream call rate to protect a paid plan).

Common pitfalls

• GOMEMLIMIT without GOGC — the runtime relies solely on the soft limit, leading to large heap swings just under the ceiling. Always pair them.
• GOGC=100 with a tight container limit — the heap can double in size before GC fires. A container with a 512 MiB limit can OOM before GC triggers.
• waitBeforeShutdown too short — load-balancers / service meshes can take several seconds to drain an endpoint after a readiness probe fails. Values below 10 s risk connection resets on rolling restarts with Envoy.
• waitAfterShutdown too short — if the process exits before the proxy finishes draining, in-flight requests to that pod are reset. 30 s is a safe default.
• terminationGracePeriodSeconds too short — Kubernetes SIGKILL fires when this expires. It must exceed waitBeforeShutdown + waitAfterShutdown + expected drain time.
• includeErrorDetails: true in production — upstream error messages often contain full endpoint URLs with API keys embedded. Set to false before exposing eRPC to external callers.
• Missing trustedIPForwarders — IP-based rate limits and auth policies all see the LB IP, effectively becoming global instead of per-client.
• Chain ID auto-detection in large deployments — every eRPC replica calls eth_chainId on every upstream at startup. With many replicas and many upstreams this creates a startup burst. Configuring evm.chainId explicitly eliminates it.