Production checklist

How it works

GOMEMLIMIT and GC tuning. eRPC's Kubernetes and Docker manifests ship without GOMEMLIMIT. With Go's default GC (GOGC=100) and a 3 Gi container limit, the heap can approach the hard limit before a GC cycle fires — the OS OOM killer then terminates the container before Go can reclaim memory. Set GOMEMLIMIT to 90 % of the container memory limit (e.g. 2700MiB for a 3Gi pod) so Go triggers a soft GC pass instead. Pair it with GOGC=30 to reduce heap swing below the ceiling; values below 10 cause GC thrash. Source: kube/erpc.yml:L33-36

Server timeouts. Three fields govern the full request lifecycle: server.maxTimeout (default 150s) is a custom TimeoutHandler wrapping the entire HTTP handler chain — it is mandatory and validation rejects 0. server.readTimeout (default 30s) covers header and body read; server.writeTimeout (default 120s) covers response write. These are Go http.Server fields. A maxTimeout hit returns JSON-RPC -32603 with HTTP 200 for POST requests. Integer YAML values are interpreted as milliseconds — maxTimeout: 150 means 150 ms, not 150 s; always use the string form.

Graceful drain (healthcheck probes). On SIGTERM the shutdown sequence is: (1) draining flag flips immediately — healthcheck returns HTTP 503; (2) process sleeps server.waitBeforeShutdown (default 10s) to let load-balancer drain the endpoint; (3) http.Server.Shutdown starts with a hardcoded 30 s budget; (4) after appCtx is done, Init sleeps server.waitAfterShutdown (default 10s) before process exit. For Kubernetes with Cilium/Envoy, set both waits to 30s. terminationGracePeriodSeconds must exceed waitBeforeShutdown + waitAfterShutdown + expected drain time or the process receives SIGKILL mid-drain. Sources: erpc/http_server.go:L78-83, erpc/http_server.go:L209-221, erpc/init.go:L172-177

Trusted-IP forwarding. When eRPC runs behind a load-balancer, server.trustedIPForwarders defaults to loopback only (["127.0.0.1/8", "::1/128"]). IP-based rate limits and network auth strategies apply to the proxy IP unless you add the LB CIDR to the forwarders list and name the forwarding header in server.trustedIPHeaders. The resolver walks headers in preference order, strips trailing trusted proxies from the XFF list right-to-left, and picks the nearest untrusted hop.

Error details in responses. server.includeErrorDetails defaults to true. Upstream error messages frequently contain endpoint URLs with embedded API keys. Set to false before exposing eRPC to external callers. Errors are still logged internally at full verbosity regardless of this setting.

Prometheus metrics and alerting. The metrics server starts on port 4001 (all interfaces). metrics.errorLabelMode: compact (the default) produces short stable error codes — verbose mode embeds full messages including block numbers or IP addresses, creating unbounded Prometheus cardinality. Six alert rules ship in monitoring/prometheus/alert.rules; two reference stale metric names (erpc_upstream_request_self_rate_limited_total, erpc_network_request_self_rate_limited_total) — use erpc_rate_limits_total instead. The monitoring container (Prometheus + Grafana combined) runs both processes under tail -f /dev/null; not suitable for production — run them as separate services.

Chain ID explicit configuration. Without explicit evm.chainId in network and upstream config, eRPC calls eth_chainId on every upstream at startup. With many replicas and upstreams this produces a startup request burst. Set evm.chainId explicitly to eliminate it.

Config file discovery. At startup eRPC searches for a config file in this order: --config <path> CLI flag → first positional argument → first match among ./erpc.yaml, ./erpc.yml, ./erpc.ts, ./erpc.js, /erpc.yaml, /erpc.yml, /erpc.ts, /erpc.js, /root/erpc.yaml, /root/erpc.yml, /root/erpc.ts, /root/erpc.js. If --require-config is set (or an explicit path was given) and no file is found, startup fails. Source: cmd/erpc/main.go:L279-L294 (opens in a new tab)

CLI subcommands. Three subcommands are available: erpc [config] / erpc start starts the server; erpc validate [--format json|md] parses config, runs erpc.GenerateValidationReport, and exits non-zero on errors — useful for CI gates before a production deploy; erpc dump [--format yaml|json] parses config, resolves effective selection policies, and writes the merged config to stdout. Source: cmd/erpc/main.go (opens in a new tab)

Runtime log environment variables. Two env vars control logging at runtime without touching the config file: LOG_LEVEL overrides cfg.LogLevel and is parsed as a zerolog level (trace, debug, info, warn, error); LOG_WRITER=console switches to a human-readable zerolog console writer with 04:05.000ms time format (useful in local dev or when piping to a log aggregator that cannot parse JSON). Source: cmd/erpc/main.go:L354-L363 (opens in a new tab)

Config schema

Fields relevant to production hardening. All under server. or metrics. unless noted.

Hardcoded constants (not configurable):

• http.Server.IdleTimeout = 300 s — erpc/http_server.go:L185
• MaxHeaderBytes = 1 MiB — erpc/http_server.go:L186
• Graceful shutdown budget = 30 s — erpc/http_server.go:L1642
• Metrics shutdown budget = 5 s — erpc/init.go:L162-168
• Metrics server binds 0.0.0.0:<port> — hostV4/hostV6 config fields are defined but unused — erpc/init.go:L149

Worked examples

All patterns below are distilled from real production fleets; comments explain the non-obvious choices.

1. Kubernetes deployment — full hardened server block. A customer-facing multi-region edge deployment behind a load balancer. Both drain waits at 30s match a typical LB drain window; terminationGracePeriodSeconds: 90 in the pod spec leaves buffer. The responseHeaders block injects per-machine routing metadata without any per-request overhead:

# Pod environment — set alongside your eRPC container spec.
# GOMEMLIMIT omitted from all bundled kube manifests; add it yourself.
env:
  - name: GOMEMLIMIT
    # 90% of the container memory limit (3Gi pod → 2700MiB).
    # Without this Go GC allows the heap to reach the hard limit before
    # triggering a collection, causing OOM kills instead of soft GC passes.
    value: "2700MiB"
  - name: GOGC
    # Reduce heap swing below GOMEMLIMIT ceiling; values < 10 cause GC thrash.
    value: "30"
resources:
  limits:
    memory: "3Gi"
  requests:
    memory: "1Gi"

server:  maxTimeout: 150s  # LB drain window — most load balancers and k8s ingresses need 20-30s to drain an endpoint.  waitBeforeShutdown: 30s  # Keep the process alive after shutdown so proxies can close open connections.  waitAfterShutdown: 30s  # CRITICAL for external-facing deployments — upstream errors routinely embed API keys.  includeErrorDetails: false  # RFC-1918 covers most private meshes and cloud VPCs; tighten if your LB  # uses a narrower range.  trustedIPForwarders:    - "10.0.0.0/8"    - "172.16.0.0/12"    - "192.168.0.0/16"  # some platforms inject the real client IP in their own header (e.g. Fly-Client-IP, CF-Connecting-IP); prefer it over X-Forwarded-For  # which can be spoofed by the caller.  trustedIPHeaders:    - "CF-Connecting-IP"  # Env vars are expanded once at startup; if REGION is unset the header is silently  # dropped — no warning. Confirm both vars are present in your machine environment.  responseHeaders:    X-ERPC-Region: ${REGION}    X-ERPC-Machine: ${MACHINE_ID}metrics:  enabled: true  port: 4001  # compact is the default; confirm it explicitly so a future config merge can't flip it.  errorLabelMode: compact  # Drop user + agent_name from all histograms to keep Prometheus series bounded.  # erpc_upstream_request_duration_seconds × user × agent_name = O(users × upstreams)  # which blows up TSDB at multi-tenant scale. Re-add selectively below.  histogramDropLabels:    - user    - composite  # Re-add user only on the network histogram where per-user latency analysis matters.  histogramLabelOverrides:    network_request_duration_seconds:      - user

2. Internal indexing deployment (no public exposure). An internal indexing fleet on Kubernetes with gRPC enabled for cache connectors. includeErrorDetails is omitted (left at default true) because callers are internal services that need the full upstream error for debugging. logLevel: error cuts log volume drastically for high-QPS indexers where warn produces too much noise:

logLevel: errorserver:  # Same 30s drain waits as edge — service-mesh networking also needs the full window.  waitBeforeShutdown: 30s  waitAfterShutdown: 30s  # gRPC enabled for cache connectors; shares port 4000 with HTTP by default.  # If you separate the ports add an explicit Docker -p flag — only 4000/4001/6060 are EXPOSED.  grpcEnabled: truemetrics:  enabled: true  port: 4001  errorLabelMode: compact  # Fine-grained histogram buckets tuned for fast internal RPC (10ms–30s range).  # Default buckets miss the 10–100ms region where most indexer calls land.  histogramBuckets: "0.010,0.030,0.050,0.100,0.250,0.500,1,3,5,10,30"

3. Docker Compose / local staging — minimal hardening. Smaller memory budget; drain waits shortened to 5s because no Kubernetes LB lifecycle is involved. includeErrorDetails: false should still be set even in staging so configs don't accidentally mirror prod with it enabled:

server:  maxTimeout: 60s  waitBeforeShutdown: 5s  waitAfterShutdown: 5s  includeErrorDetails: falsemetrics:  enabled: true  port: 4001  errorLabelMode: compact

4. Multi-tenant high-cardinality deployment with selective label overrides. When serving many users across many upstreams, erpc_upstream_request_duration_seconds with user and agent_name labels creates O(users × upstreams) Prometheus series. The production pattern drops both globally and re-adds user only on the network-level histogram where per-user latency SLO reporting is needed:

metrics:  errorLabelMode: compact  # Drop high-cardinality labels from ALL histograms first.  # erpc_upstream_request_duration_seconds × 50 users × 20 upstreams = 1000 series minimum.  histogramDropLabels:    - user    - agent_name  # Then selectively restore user on the one histogram that feeds per-user latency dashboards.  histogramLabelOverrides:    network_request_duration_seconds:      - user

Request/response behavior

• On SIGTERM, healthcheck returns HTTP 503 plain-text "shutting down" immediately, before any drain wait. [erpc/http_server.go:L78-83]
• waitBeforeShutdown elapses before http.Server.Shutdown starts; waitAfterShutdown elapses after appCtx is done, before os.Exit. [erpc/http_server.go:L209-221, erpc/init.go:L172-177]
• A maxTimeout hit returns JSON-RPC -32603 "http request handling timeout" with HTTP 200 for POST, 504 for non-POST — not a Go deadline cancellation. [erpc/http_timeout.go:L107-122]
• server.responseHeaders env expansion runs once at NewHttpServer construction time, not per-request. Empty-after-expansion values are silently dropped. [erpc/http_server.go:L135-148]
• The metrics server binds 0.0.0.0:<port> — metrics.hostV4 / hostV6 are defined but unused. Restrict scrape access via firewall or network policy. [erpc/init.go:L149]
• metrics.errorLabelMode: compact is applied at erpc.Init; any series created before Init carry verbose labels until they expire. [erpc/init.go:L138-140]
• Idle-series sweep evicts stale upstream_request_duration_seconds and rate_limits_total Prometheus series every ~5 minutes with a 30-minute idle window. [health/tracker.go:L557-563, health/tracker.go:L598-631]

Best practices

• Set GOMEMLIMIT to 90 % of your container memory limit. The bundled kube/erpc.yml omits it entirely — Go's default GC will let the heap grow to the container ceiling, causing OOM kills rather than soft GC passes.
• Both waitBeforeShutdown and waitAfterShutdown default to 10s — raise both to 30s for Kubernetes + Envoy/Cilium environments where LB drain windows are typically 20–30 s.
• terminationGracePeriodSeconds must exceed waitBeforeShutdown + waitAfterShutdown + expected drain time. With both waits at 30 s, use terminationGracePeriodSeconds: 90 minimum.
• Always set includeErrorDetails: false before exposing eRPC externally — the default is true and upstream error messages routinely contain embedded API keys.
• Set evm.chainId explicitly on every network and upstream to avoid an eth_chainId call burst on every pod restart or scale-out event.
• For high-traffic multi-tenant deployments, add user and agent_name to metrics.histogramDropLabels — erpc_upstream_request_duration_seconds with those labels can produce thousands of Prometheus series; use histogramLabelOverrides to selectively re-add them on specific histograms where per-user analysis is needed.
• Alert on erpc_unexpected_panic_total at any non-zero rate — recovered panics mean a request goroutine crashed; they are surfaced as -32603 errors to the caller rather than crashing the process, so they can go undetected without an alert.

Edge cases & gotchas

1. maxTimeout: 0 (or bare 0) fails validation — "server.maxTimeout is required". Integer YAML values are milliseconds: maxTimeout: 150 = 150 ms. Always use string form "150s". Source: common/validation.go:L90-92, common/duration.go:L8-32
2. GOMEMLIMIT is absent from all bundled manifests. With a 3 Gi kube limit and default GC, heap can approach the limit before GC fires. Set GOMEMLIMIT=2700MiB as a container env var. Source: kube/erpc.yml:L33-36
3. terminationGracePeriodSeconds must exceed waitBeforeShutdown + waitAfterShutdown + drain time. With both set to 30 s, use terminationGracePeriodSeconds: 90 minimum. Kubernetes SIGKILL fires when this expires, cutting in-flight requests.
4. metrics.hostV4: 127.0.0.1 does not restrict the metrics port. The bind address is always 0.0.0.0:<port>. Restrict via firewall, network policy, or run the metrics server inside a VPC. Source: erpc/init.go:L149
5. server.responseHeaders with an env var that is unset at startup silently omits the header. No warning is emitted. If X-ERPC-Region: ${REGION} is configured and that env var is absent in a new region, the header disappears with no observable error. Source: erpc/http_server.go:L135-148
6. metrics.errorLabelMode: verbose can create unbounded Prometheus cardinality. Full error messages include block numbers and IP addresses. Existing compact series and new verbose series co-exist until the verbose ones expire. Source: common/errors.go:L17-66
7. The monitoring container (Prometheus + Grafana combined) is not production-suitable. It runs both processes under tail -f /dev/null; if either crashes neither restarts. Use separate Prometheus and Grafana deployments in production. Source: monitoring/Dockerfile
8. Alert rules reference stale metric names. HighRateLimiting and NetworkRateLimiting use erpc_upstream_request_self_rate_limited_total / erpc_network_request_self_rate_limited_total. The current metric is erpc_rate_limits_total. Source: monitoring/prometheus/alert.rules
9. kube/postgres.yml pins postgres:latest. No digest. Pin a specific PostgreSQL version before deploying the bundled manifest. Source: kube/postgres.yml
10. docker compose down (without --volumes) preserves monitoring data; make down passes --volumes and destroys prometheus_data and grafana_data. Source: Makefile:L114
11. trustedIPHeaders: [] (the default) means all requests appear to originate from the load-balancer IP. IP-based rate limits and network auth strategies become effectively global rather than per-client. Source: common/defaults.go:L730-733
12. includeErrorDetails: true in production exposes upstream endpoint URLs and potentially embedded API keys in JSON-RPC error.data. Source: erpc/http_server.go:L1424-1433
13. gRPC traffic on the shared port bypasses maxTimeout and the gzip handler. By default server.grpcPortV4 = server.httpPortV4 = 4000. When grpcEnabled: true, gRPC and HTTP share port 4000 via in-handler mux. If you separate the ports (e.g. gRPC on 4002), add the explicit Docker -p 4002:4002 flag — only ports 4000, 4001, and 6060 are EXPOSEd in the Dockerfile. Source: erpc/http_server.go:L158-177, Dockerfile:L84
14. The distroless/nonroot image has no shell, no package manager, and no debug tools. The final container runs as a nonroot user with the minimal distroless base. Exec-ing into the container, mounting files dynamically, or attaching profilers requires a privileged sidecar. The default CMD is /erpc-server, not a shell wrapper. Source: Dockerfile
15. TypeScript config files (erpc.ts) require Node.js, which is NOT in the distroless image. The compiled TypeScript SDK and node_modules are present, and eRPC invokes a bundled Node execution path for .ts config loading. If node_modules is stripped from a custom image layer, TypeScript config loading fails silently and the process may not find a valid config file.
16. erpc-server-pprof is present in every image but is never the default CMD. The pprof binary is always built into the image at /erpc-server-pprof but the default CMD is /erpc-server. Operators must override the container command to use pprof — it is not activated by any env var or config flag. Source: Dockerfile:L84, Makefile (run-pprof target)
17. The prometheus.yaml at the repo root is NOT the monitoring container's config. It is the production Prometheus Operator config (with remote_write to Mimir, in-cluster job discovery). The monitoring container at monitoring/prometheus/prometheus.yml is a separate file with different scrape targets. Loading the wrong one into the monitoring container will produce empty or incorrect dashboards. Source: prometheus.yaml, monitoring/prometheus/prometheus.yml

Observability

Bundled alert rules (monitoring/prometheus/alert.rules):

Source code entry points

• erpc/http_server.go:L78-L83 (opens in a new tab) — drain flag set on SIGTERM; healthcheck returns 503
• erpc/http_server.go:L209-L221 (opens in a new tab) — waitBeforeShutdown sleep + Shutdown call with 30 s budget
• erpc/init.go:L172-L177 (opens in a new tab) — waitAfterShutdown sleep before process exit
• erpc/http_timeout.go:L36-L143 (opens in a new tab) — custom TimeoutHandler (timeout/cancel bodies, buffered response, panic propagation)
• erpc/init.go:L137-L170 (opens in a new tab) — metrics HTTP server construction; promhttp.Handler() on :%d; 5 s shutdown
• common/defaults.go:L694-L733 (opens in a new tab) — ServerConfig.SetDefaults: maxTimeout, readTimeout, writeTimeout, waitBeforeShutdown, waitAfterShutdown, includeErrorDetails, executionHeaders, trustedIPForwarders, trustedIPHeaders
• common/defaults.go:L749-L767 (opens in a new tab) — MetricsConfig.SetDefaults: enabled, port, errorLabelMode
• kube/erpc.yml:L33-L36 (opens in a new tab) — reference kube resource spec (3 Gi / 2 CPU; no GOMEMLIMIT)
• monitoring/prometheus/alert.rules (opens in a new tab) — 6 bundled alerting rules
• cmd/erpc/main.go:L279-L294 (opens in a new tab) — config file discovery: 12-path ordered search
• cmd/erpc/main.go:L354-L363 (opens in a new tab) — LOG_LEVEL env var override
• cmd/erpc/main.go:L53-L57 (opens in a new tab) — LOG_WRITER=console zerolog console writer
• cmd/erpc/main.go (opens in a new tab) — validate and dump CLI subcommands

Related pages

• Deployment — Docker, Kubernetes, and Compose setup instructions.
• Rate limiters — per-IP and per-user budget configuration that depends on correct trustedIPHeaders.
• Auth — the network auth strategy uses the resolved client IP.
• Observability — full metrics and alerting reference.
• Fail-safe: timeout — per-request upstream deadline distinct from server.maxTimeout.