# Tracing & logging > Source: https://docs.erpc.cloud/operation/tracing > Every request, cache lookup, and upstream call becomes a searchable span — shipped to any OTel backend. Secrets never leave the process. > Format: machine-readable markdown export of the docs page above. > All collapsible AI sections are inlined and fully expanded. # Tracing & logging eRPC instruments every layer of the request pipeline — inbound HTTP, routing, cache lookups, upstream calls — and exports those spans over OTLP to any OpenTelemetry-compatible backend. Structured JSON logs run at five verbosity levels, secrets are redacted before any line is written, and a force-trace override lets you capture full detail for a single network or method without touching global sampling. **What you get** - End-to-end distributed traces from caller through every upstream hop - Two span tiers: always-on for data-plane ops, detailed for high-cardinality debug info - Force-trace by request header, query param, or config pattern — even at `sampleRate: 0` - Automatic secret redaction for all API keys and passwords in logs and config dumps - W3C TraceContext propagation: eRPC becomes a child of your caller's trace ## Quick taste Illustrative, not a tuned production config — 10 % sampling, gRPC export: **Config path:** `tracing` **YAML — `erpc.yaml`:** ```yaml tracing: enabled: true # OTLP gRPC collector — eRPC always overrides OTEL_EXPORTER_OTLP_ENDPOINT endpoint: localhost:4317 protocol: grpc # 10 % ambient sampling — sampleRate: 0 is rewritten to 1.0 by SetDefaults, use enabled: false to disable sampleRate: 0.1 serviceName: erpc-prod ``` **TypeScript — `erpc.ts`:** ```typescript tracing: { enabled: true, // OTLP gRPC collector — eRPC always overrides OTEL_EXPORTER_OTLP_ENDPOINT endpoint: "localhost:4317", protocol: "grpc", // 10 % ambient sampling — sampleRate: 0 is rewritten to 1.0 by SetDefaults, use enabled: false to disable sampleRate: 0.1, serviceName: "erpc-prod", } ``` ## Agent reference Copy one of these prompts into your AI agent session (Claude Code, Cursor, …) — each one points the agent at this page's machine-readable reference so it can do the work correctly: **Prompt Example #1: wire eRPC traces to my OTel collector** ```text Set up distributed tracing in my eRPC instance so every request, cache lookup, and upstream call produces spans exported to my OpenTelemetry collector. Work with my existing eRPC config. and my collector is at otel-collector:4317. Choose a sensible production sample rate and explain the sampleRate: 0 footgun. Read the full reference first: https://docs.erpc.cloud/operation/tracing.llms.txt ``` **Prompt Example #2: force full traces on debug methods only** ```text I want eRPC to sample at 5% normally but always capture 100% traces for debug_* and trace_* methods so I can triage slow calls without raising global sample rate. Update my eRPC config with the right forceTraceMatchers config. Reference: https://docs.erpc.cloud/operation/tracing.llms.txt ``` **Prompt Example #3: send traces to Grafana Cloud with auth** ```text Configure eRPC's OTLP exporter in my eRPC config to send traces to my Grafana Cloud OTLP HTTP endpoint with bearer-token auth headers and TLS enabled. Include the sample rate and serviceName settings appropriate for production. Reference: https://docs.erpc.cloud/operation/tracing.llms.txt ``` **Prompt Example #4: debug missing or noisy spans** ```text My tracing backend shows either no spans from eRPC or too many after a config change. Diagnose the issue in my eRPC config — check sampleRate zero-rewrite, OTEL env var overrides, and detailed mode cost — and fix the config so I get the spans I expect. Reference: https://docs.erpc.cloud/operation/tracing.llms.txt ``` --- ### Tracing & logging — full agent reference ### How it works **Tracing initialization.** On startup, `NewERPC` calls `common.InitializeTracing` exactly once (guarded by `sync.Once`). When enabled, it builds an OTLP exporter (gRPC via `otlptracegrpc` or HTTP via `otlptracehttp`), constructs a `TracerProvider` with a batcher and a custom sampler, wires W3C TraceContext + Baggage propagation, and stores any `forceTraceMatchers` for per-request evaluation. Two package-level booleans — `IsTracingEnabled` and `IsTracingDetailed` — are set once and consulted on every hot-path span call with zero allocation when tracing is off. [[`common/tracing_core.go:L53-148`](https://github.com/erpc/erpc/blob/main/common/tracing_core.go#L53-L148)] **Two span tiers.** All instrumented call sites use one of two helpers: - `StartSpan` — fires whenever `IsTracingEnabled`; covers major external interactions: cache, upstreams, connectors, rate limiters, consensus. - `StartDetailSpan` — fires only when both `IsTracingEnabled` and `IsTracingDetailed`; covers eRPC-internal operations and spans carrying high-cardinality attributes such as request params, block numbers, lock contention, and hook execution. Enabling `detailed: true` roughly doubles span count. When tracing is disabled, both helpers return the caller's unchanged context and a singleton `noopSpan` — no allocation on the hot path. [[`common/tracing_util.go:L1-234`](https://github.com/erpc/erpc/blob/main/common/tracing_util.go#L1-L234)] **Span hierarchy for a single HTTP request (normal mode):** ``` Http.ReceivedRequest (SpanKindServer) └─ Request.Handle (SpanKindInternal) └─ Network.Forward └─ Network.forwardAttempt ├─ Cache.Get └─ Upstream.Forward └─ HttpJsonRpcClient.sendSingleRequest ``` With `detailed: true`, additional spans appear inside the HTTP layer (`Http.ReadBody`, `Http.ParseRequests`, `HttpServer.WriteResponse`), between Forward and forwardAttempt (`Project.Forward`, `Network.TryForward`, `Network.UpstreamLoop`), and inside upstream execution (`RateLimiter.TryAcquirePermit`, `Upstream.tryForward.PreRequest`, `Upstream.tryForward.SendRequest`), cache lookup, lock/parse ops, and method-specific hooks. **Sampler logic.** `createTracingSampler` builds the base sampler from `sampleRate`: - `<= 0` → `NeverSample()` - `>= 1.0` → `AlwaysSample()` - otherwise → `ParentBased(TraceIDRatioBased(rate))` with all four parent-sampling options configured for consistency (no orphan spans) The `forceTraceSampler` always wraps the base sampler regardless of rate. [[`common/tracing_core.go:L229-268`](https://github.com/erpc/erpc/blob/main/common/tracing_core.go#L229-L268)] **Force-trace mechanism.** Three paths can force a span to be recorded regardless of `sampleRate`: 1. HTTP header `X-ERPC-Force-Trace: true` (or `1`, `yes`) — checked at request entry. 2. Query parameter `?force-trace=true` — same check. 3. Config-driven `forceTraceMatchers` — evaluated after the URL is parsed and the network is known. Each matcher can specify `network` (in `architecture:chainId` form, e.g. `evm:1`) and/or `method` (JSON-RPC method pattern). Pipe (`|`) = OR; wildcards supported. Both fields present on one matcher = AND semantics. In all cases, the force-trace decision is communicated to the OTel sampler by setting the span attribute `erpc.force_trace = true` at creation time. The custom `forceTraceSampler` wraps the base sampler and returns `RecordAndSample` unconditionally when it sees this attribute — even when `sampleRate: 0`. [[`common/tracing_core.go:L257-268`](https://github.com/erpc/erpc/blob/main/common/tracing_core.go#L257-L268)] **Context propagation.** Incoming HTTP requests have `traceparent`/`tracestate` extracted via W3C `TraceContext{}` carrier, making eRPC spans children of the caller's trace. After the response is written, `InjectHTTPResponseTraceContext` injects the active span context back into response headers so downstream callers can correlate their traces. gRPC requests have no OTel trace extraction — they start with a fresh root span. [[`common/tracing_util.go:L58-74`](https://github.com/erpc/erpc/blob/main/common/tracing_util.go#L58-L74)] **Logging system.** zerolog is the structured JSON logger used throughout eRPC. Default output is newline-delimited JSON with Unix-millisecond timestamps: ```json {"level":"info","time":1718000000123,"message":"...","key":"value"} ``` Set `LOG_WRITER=console` to switch to a human-readable format for local development: ``` 04:05.000ms INF message key=value ``` Five levels are available: `trace`, `debug`, `info`, `warn`, `error` (plus `fatal`, `panic`, `disabled`). `LOG_LEVEL` environment variable overrides `logLevel` config at startup and again when the config is loaded, so it always wins. `ERPC_NOLOGS=1` silences all output at the binary level. When the resolved level is `info` or lower, eRPC serializes the entire loaded config as a JSON field in a startup log line. All secrets are redacted before this serialization. **Secret redaction.** Two layers: 1. `MarshalJSON`/`MarshalYAML` overrides on config types — called whenever config is serialized — replace secret fields with `"REDACTED"` or apply `util.RedactEndpoint`: | Config type | Field | Replacement | |---|---|---| | `RedisConnectorConfig` | `password` | `"REDACTED"` | | `RedisConnectorConfig` | `uri` | `util.RedactEndpoint(uri)` | | `PostgreSQLConnectorConfig` | `connectionUri` | `util.RedactEndpoint(connectionUri)` | | `AwsAuthConfig` | `secretAccessKey` | `"REDACTED"` | | `ProviderConfig` | `settings` | `"REDACTED"` | | `UpstreamConfig` | `endpoint` | `util.RedactEndpoint(endpoint)` | | `SecretStrategyConfig` | `value` | `"REDACTED"` | [[`common/config.go:L397-503`](https://github.com/erpc/erpc/blob/main/common/config.go#L397-L503)] 2. `util.RedactEndpoint` — computes a SHA-256 of the full original URL, takes the first 5 hex chars as a stable identifier, then strips path and credentials according to scheme: - Native-protocol endpoints (`alchemyv2://`, `erigon://`, etc.): `scheme://host#redacted=` - Envio-suffix schemes: `scheme://host` (no hash needed — no secrets in path) - Repository-suffix schemes: `scheme://host#redacted=` - Standard `http`/`https` RPC endpoints: `scheme#redacted=` — **hostname is also dropped** - Unparseable URL: `"redacted="` Two endpoints with the same scheme but different paths (API keys) produce different hashes, enabling correlation without exposing secrets. [[`util/redact.go:L10-36`](https://github.com/erpc/erpc/blob/main/util/redact.go#L10-L36)] **OTel resource attributes.** The `TracerProvider` is built with a resource that includes `service.name` (from `tracing.serviceName`), `service.version` (the compiled-in `ErpcVersion` constant), `commit.sha` (the compiled-in `ErpcCommitSha` constant), and any key/value pairs from `tracing.resourceAttributes`. Environment variables in `resourceAttributes` values are expanded via `os.ExpandEnv` at config load; empty values after expansion are silently skipped. [[`common/tracing_core.go:L93-102`](https://github.com/erpc/erpc/blob/main/common/tracing_core.go#L93-L102)] The OTel instrumentation name registered by eRPC is `"github.com/erpc/erpc"`. [[`common/tracing_core.go:L26`](https://github.com/erpc/erpc/blob/main/common/tracing_core.go#L26)] **Error recording.** `SetTraceSpanError` checks `span.IsRecording()` before acting. For `StandardError` types it sets an `error.code` span attribute (the full error-code chain) and records the error as a span event; for plain `error` values it calls `span.RecordError` and sets the span status to `codes.Error`. [[`common/tracing_core.go:L163-183`](https://github.com/erpc/erpc/blob/main/common/tracing_core.go#L163-L183)] **Force-flush.** `common.ForceFlushTraces(ctx)` calls `tracerProvider.ForceFlush(ctx)`, flushing all batched but not-yet-exported spans to the collector immediately. It is called automatically on graceful shutdown and exposed for callers that need guaranteed delivery of critical traces. [[`common/tracing_core.go:L237-245`](https://github.com/erpc/erpc/blob/main/common/tracing_core.go#L237-L245)] **Graceful shutdown.** A goroutine watches `appCtx.Done` and flushes buffered spans with a 5-second grace period before the process exits. [[`erpc/erpc.go:L92-99`](https://github.com/erpc/erpc/blob/main/erpc/erpc.go#L92-L99)] ### Config schema #### Top-level `logLevel` | Field | Type | Default | Behavior / footguns | |---|---|---|---| | `logLevel` | string | `"INFO"` ([`common/defaults.go:L50-51`](https://github.com/erpc/erpc/blob/main/common/defaults.go#L50-L51)) | Controls log verbosity for all eRPC subsystems. Valid: `trace`, `debug`, `info`, `warn`, `error`, `fatal`, `panic`, `disabled` (case-insensitive). Invalid value → warning + fallback to `debug` ([`erpc/init.go:L27-32`](https://github.com/erpc/erpc/blob/main/erpc/init.go#L27-L32)). Overridden by `LOG_LEVEL` env var at config-load time ([`cmd/erpc/main.go:L354-363`](https://github.com/erpc/erpc/blob/main/cmd/erpc/main.go#L354-L363)). | #### `tracing.*` Full struct: [`common/config.go:L218-235`](https://github.com/erpc/erpc/blob/main/common/config.go#L218-L235). | Field | Type | Default | Behavior / footguns | |---|---|---|---| | `tracing.enabled` | bool | `false` (Go zero value) | Master switch. When `false`, sets `IsTracingEnabled = false` globally; all span calls are no-ops ([`common/tracing_core.go:L53-58`](https://github.com/erpc/erpc/blob/main/common/tracing_core.go#L53-L58)). | | `tracing.endpoint` | string | `"localhost:4317"` (gRPC) or `"http://localhost:4318"` (HTTP) — derived from `protocol` in `SetDefaults` ([`common/defaults.go:L622-628`](https://github.com/erpc/erpc/blob/main/common/defaults.go#L622-L628)) | OTLP collector endpoint. gRPC: `host:port` with no scheme. HTTP: full URL including scheme. **Footgun:** `OTEL_EXPORTER_OTLP_ENDPOINT` is always overridden by this field even when it holds the default value. | | `tracing.protocol` | TracingProtocol | `"grpc"` ([`common/defaults.go:L619-621`](https://github.com/erpc/erpc/blob/main/common/defaults.go#L619-L621)) | `"grpc"` uses `otlptracegrpc`; `"http"` uses `otlptracehttp`. Any other value → error at init ([`common/tracing_core.go:L71-78`](https://github.com/erpc/erpc/blob/main/common/tracing_core.go#L71-L78)). | | `tracing.sampleRate` | float64 | `1.0` ([`common/defaults.go:L629-631`](https://github.com/erpc/erpc/blob/main/common/defaults.go#L629-L631)) | `<= 0` → `NeverSample()`, `>= 1.0` → `AlwaysSample()`, otherwise → `ParentBased(TraceIDRatioBased(rate))`. **Footgun:** `SetDefaults` replaces `0` with `1.0`, so `sampleRate: 0` in YAML means "always sample", not "never sample". To keep tracing on but silent, use a very small positive value like `0.0000001`. | | `tracing.detailed` | bool | `false` (Go zero value) | When `true`, sets `IsTracingDetailed = true`, enabling `StartDetailSpan` calls. Roughly 2× span count. Includes request params, block numbers, lock spans, hook execution spans. | | `tracing.serviceName` | string | `"erpc"` ([`common/defaults.go:L632-634`](https://github.com/erpc/erpc/blob/main/common/defaults.go#L632-L634)) | OTel `service.name` resource attribute. | | `tracing.headers` | map[string]string | nil | Additional metadata headers sent with every OTLP export batch. Used for auth to managed collectors (Grafana Cloud, Honeycomb, Datadog). When set, takes precedence over `OTEL_EXPORTER_OTLP_HEADERS`. | | `tracing.tls.enabled` | bool | `false` (Go zero value) | Master switch for TLS on the OTLP exporter. When `false`, exporters are created with `WithInsecure()` ([`common/tracing_core.go:L187`](https://github.com/erpc/erpc/blob/main/common/tracing_core.go#L187)). | | `tracing.tls.certFile` | string | `""` | Path to PEM client certificate for mTLS. Both `certFile` and `keyFile` must be non-empty; supplying one without the other silently skips mutual TLS. | | `tracing.tls.keyFile` | string | `""` | Path to PEM private key corresponding to `certFile`. Required alongside `certFile` for mTLS. | | `tracing.tls.caFile` | string | `""` | Path to PEM CA certificate. When set, the collector's server cert is validated against this CA instead of the system pool. | | `tracing.tls.insecureSkipVerify` | bool | `false` | When `true`, disables verification of the OTLP collector's server cert. Does not affect client-cert loading. Only for development or trusted private networks. | | `tracing.resourceAttributes` | map[string]string | nil | Custom OTel resource attributes. Values are `os.ExpandEnv`-expanded at config load. Empty values after expansion are silently skipped ([`common/tracing_core.go:L93-97`](https://github.com/erpc/erpc/blob/main/common/tracing_core.go#L93-L97)). | | `tracing.forceTraceMatchers` | []*ForceTraceMatcher | nil | Matchers that bypass `sampleRate` and force `RecordAndSample`. | | `tracing.forceTraceMatchers[].network` | string | — | Network pattern in `architecture:chainId` form, e.g. `"evm:1"`, `"evm:1\|evm:42161"`, `"evm:*"`. Pipe = OR; wildcards via `WildcardMatch`. | | `tracing.forceTraceMatchers[].method` | string | — | JSON-RPC method pattern, e.g. `"eth_call"`, `"debug_*\|trace_*"`. Pipe = OR; wildcards supported. | **ForceTraceMatcher AND/OR semantics:** when both `network` and `method` are specified on one matcher, both must match (AND). If only one field is set, only that field is checked. An empty matcher (neither field) never matches ([`common/tracing_core.go:L306-330`](https://github.com/erpc/erpc/blob/main/common/tracing_core.go#L306-L330)). #### Standard `OTEL_*` environment variables | Env var | Interaction with eRPC config | |---|---| | `OTEL_EXPORTER_OTLP_ENDPOINT` | **Overridden.** eRPC always passes `WithEndpoint(cfg.Endpoint)`; this env var is silently ignored. | | `OTEL_EXPORTER_OTLP_HEADERS` | **Partially overridden.** Only takes effect when `tracing.headers` is nil/empty in config. | | `OTEL_SERVICE_NAME` | **Not explicitly overridden, but eRPC always sets `service.name`.** eRPC builds its resource with `semconv.ServiceNameKey.String(cfg.ServiceName)`. If the OTel SDK's default detectors also read `OTEL_SERVICE_NAME`, last-writer-wins resource merge semantics apply; eRPC's explicit value is always present. | | `OTEL_TRACES_SAMPLER` | **Ignored.** eRPC supplies its own sampler via explicit SDK option. | | `OTEL_PROPAGATORS` | **Ignored.** eRPC explicitly sets TraceContext + Baggage propagators. | | `OTEL_SDK_DISABLED` | **Honored.** Disables all OTel SDK operations globally before `InitializeTracing` runs. | ### Worked examples **1. Production setup: 10 % sampling with force-trace on debug methods.** Low ambient cost; debug-namespace calls always produce a full trace for triage: **Config path:** `tracing` **YAML — `erpc.yaml`:** ```yaml tracing: enabled: true endpoint: otel-collector:4317 protocol: grpc sampleRate: 0.1 serviceName: erpc-prod forceTraceMatchers: - method: "debug_*|trace_*" ``` **TypeScript — `erpc.ts`:** ```typescript tracing: { enabled: true, endpoint: "otel-collector:4317", protocol: "grpc", sampleRate: 0.1, serviceName: "erpc-prod", forceTraceMatchers: [{ method: "debug_*|trace_*" }], } ``` **2. Detailed mode for a staging environment.** Capture every span including request params, block numbers, and lock contention to understand slow calls — not recommended in production due to 2× span volume: **Config path:** `tracing` **YAML — `erpc.yaml`:** ```yaml tracing: enabled: true endpoint: tempo:4317 sampleRate: 1.0 detailed: true serviceName: erpc-staging ``` **TypeScript — `erpc.ts`:** ```typescript tracing: { enabled: true, endpoint: "tempo:4317", sampleRate: 1.0, detailed: true, serviceName: "erpc-staging", } ``` **3. Managed collector with auth headers and TLS.** For Grafana Cloud or Honeycomb where the OTLP endpoint requires a bearer token: **Config path:** `tracing` **YAML — `erpc.yaml`:** ```yaml tracing: enabled: true endpoint: "https://otlp-gateway.grafana.net/otlp" protocol: http sampleRate: 0.05 headers: Authorization: "Basic " ``` **TypeScript — `erpc.ts`:** ```typescript tracing: { enabled: true, endpoint: "https://otlp-gateway.grafana.net/otlp", protocol: "http", sampleRate: 0.05, headers: { Authorization: "Basic " }, } ``` **4. Silent tracing for force-trace-only capture.** Keep tracing on but produce no ambient spans — only requests that carry `X-ERPC-Force-Trace: true` or match a configured matcher are recorded. Use `sampleRate: 0.0000001` because `sampleRate: 0` is rewritten to `1.0` by `SetDefaults`: **Config path:** `tracing` **YAML — `erpc.yaml`:** ```yaml tracing: enabled: true endpoint: localhost:4317 sampleRate: 0.0000001 forceTraceMatchers: - network: "evm:1" method: "eth_call" ``` **TypeScript — `erpc.ts`:** ```typescript tracing: { enabled: true, endpoint: "localhost:4317", sampleRate: 0.0000001, forceTraceMatchers: [{ network: "evm:1", method: "eth_call" }], } ``` ### Request/response behavior - Incoming HTTP requests with `traceparent`/`tracestate` headers become children of the caller's trace. eRPC spans are nested under the caller's root span. - After the response is written, the active span's W3C context is injected back into HTTP response headers unconditionally — including error responses. Trace IDs are observable to callers. [[`erpc/http_server.go:L701`](https://github.com/erpc/erpc/blob/main/erpc/http_server.go#L701)] - For batch HTTP requests, each item gets its own `Request.Handle` child span under a single `Http.ReceivedRequest` parent. All child spans may be in-flight simultaneously. [[`erpc/http_server.go:L465`](https://github.com/erpc/erpc/blob/main/erpc/http_server.go#L465)] - gRPC requests start a fresh root span — no trace context extraction is implemented for the gRPC server path. [[`erpc/grpc_server.go`](https://github.com/erpc/erpc/blob/main/erpc/grpc_server.go)] - OTLP export failures are logged at `trace` level only and never propagate to callers. [[`common/tracing_core.go:L125-127`](https://github.com/erpc/erpc/blob/main/common/tracing_core.go#L125-L127)] - `Request.Handle` is started via `tracer.Start` directly (not `StartSpan`), so it fires whenever `IsTracingEnabled`, regardless of `IsTracingDetailed`. This is the correct attach point for distributed callers. [[`common/tracing_util.go:L165`](https://github.com/erpc/erpc/blob/main/common/tracing_util.go#L165)] **Span attributes on `Request.Handle`** (set by `StartRequestSpan`/`EndRequestSpan`): | Attribute | Mode | Value | |---|---|---| | `request.method` | always | JSON-RPC method name | | `erpc.force_trace` | always, when forced | `true` (bool, internal — used by sampler to force `RecordAndSample`; not a user-visible exported attribute) | | `erpc.forced_trace_reason` | when forced | `"header_or_query"` or `"network:,method:"` etc. | | `request.id` | detailed only | JSON-RPC request ID as string | | `request.jsonrpc.params` | detailed only | JSON-serialized params array | | `network.id` | detailed only, on response | Network ID string | | `user.id` | detailed only, on response | Auth user ID | | `request.finality` | detailed only, on response | Finality label (e.g. `"realtime"`, `"finalized"`) | | `response.finality` | detailed only, on response | Response finality | | `execution.attempts` | detailed only, on response | Total attempts | | `execution.retries` | detailed only, on response | Number of retries | | `execution.hedges` | detailed only, on response | Number of hedges | | `response.result_size` | detailed only, on response | Length of JSON-RPC result field | | `upstream.id` | always (if non-nil), on response | Upstream identifier | | `cache.hit` | always (if non-nil), on response | bool: whether response was served from cache | **Span attributes on `Http.ReceivedRequest`** (set by `StartHTTPServerSpan`/`EnrichHTTPServerSpan`): | Attribute | Phase | Value | |---|---|---| | `http.method` | on request | HTTP method (semconv) | | `http.url` | on request | Full request URL | | `http.scheme` | on request | URL scheme | | `http.user_agent` | on request | User-Agent header value | | `erpc.force_trace` | on request, when forced | `true` (if forced via header/query) | | `erpc.forced_trace_reason` | on request, when forced | `"header_or_query"` | | `http.status_code` | on response | HTTP response status code (semconv) | [[`common/tracing_util.go:L80-138`](https://github.com/erpc/erpc/blob/main/common/tracing_util.go#L80-L138)] ### Best practices - Set `sampleRate: 0.05`–`0.2` in production — `AlwaysSample()` at scale sends enormous span volume to your collector. - Use `forceTraceMatchers` with `method: "debug_*|trace_*"` to guarantee full traces on expensive diagnostic methods regardless of sample rate — this costs nothing extra for normal traffic. - Never set `sampleRate: 0` thinking it means "disabled" — `SetDefaults` rewrites it to `1.0`. Use `enabled: false` to disable tracing entirely. - Enable `detailed: true` only in staging or on-demand: 2× span count and high-cardinality attributes (params, block numbers) can saturate a collector or inflate storage costs. - Always set `tracing.endpoint` explicitly — do not rely on `OTEL_EXPORTER_OTLP_ENDPOINT` to route spans to a remote collector; eRPC overrides it unconditionally. - Enable `tracing.tls.enabled: true` for any span export that leaves a trusted network; the default is `WithInsecure()`. - Force-trace via the `X-ERPC-Force-Trace: true` request header during integration testing to capture full detail without raising global sample rate. ### Edge cases & gotchas 1. **`initOnce.Do` means tracing config cannot change after first `NewERPC`.** Tests or multi-tenant code creating multiple ERPC instances will silently use only the first instance's tracing config. Source: [`common/tracing_core.go:L53`](https://github.com/erpc/erpc/blob/main/common/tracing_core.go#L53) 2. **`sampleRate: 0` in YAML means "always sample", not "never sample".** `SetDefaults` replaces `0` with `1.0`. To keep tracing enabled but silent, set `0.0000001` or a very small positive value. Source: [`common/defaults.go:L629-631`](https://github.com/erpc/erpc/blob/main/common/defaults.go#L629-L631) 3. **Force-trace network matching requires URL parsing to have succeeded.** `forceTraceMatchers` with `network` patterns never match admin or healthcheck requests. Source: [`erpc/http_server.go:L285-288`](https://github.com/erpc/erpc/blob/main/erpc/http_server.go#L285-L288) 4. **W3C traceparent injection into HTTP responses is unconditional** when tracing is enabled — including error responses. Trace IDs are observable to callers. Source: [`erpc/http_server.go:L701`](https://github.com/erpc/erpc/blob/main/erpc/http_server.go#L701) 5. **gRPC server has no OTel trace extraction.** gRPC-originated requests start with a fresh root span, not as children of the caller's trace. Source: `erpc/grpc_server.go` 6. **TLS on the OTLP exporter is off by default.** Both gRPC and HTTP exporters use `WithInsecure()` unless `tracing.tls.enabled: true`. Production deployments sending spans over the internet should enable TLS. Source: [`common/tracing_core.go:L187`](https://github.com/erpc/erpc/blob/main/common/tracing_core.go#L187) 7. **`OTEL_EXPORTER_OTLP_ENDPOINT` is silently overridden.** eRPC always calls `WithEndpoint(cfg.Endpoint)`, so this env var has no effect. Set `tracing.endpoint` in config explicitly. Source: [`common/tracing_core.go:L197`](https://github.com/erpc/erpc/blob/main/common/tracing_core.go#L197) 8. **`OTEL_TRACES_SAMPLER` and `OTEL_PROPAGATORS` have no effect.** eRPC supplies its own sampler and propagator via explicit SDK options. Source: [`common/tracing_core.go:L109`](https://github.com/erpc/erpc/blob/main/common/tracing_core.go#L109) 9. **`OTEL_EXPORTER_OTLP_HEADERS` only takes effect when `tracing.headers` is nil/empty in config.** Source: [`common/tracing_core.go:L200-202`](https://github.com/erpc/erpc/blob/main/common/tracing_core.go#L200-L202) 10. **`util.RedactEndpoint` drops the entire hostname for standard http/https URLs.** An endpoint like `https://eth-mainnet.alchemyapi.io/v2/SECRET` becomes `https#redacted=ab3c4`. Only native-protocol and repository-type endpoints keep `scheme://host`. Source: [`util/redact.go:L32-34`](https://github.com/erpc/erpc/blob/main/util/redact.go#L32-L34) 11. **`LOG_LEVEL` env var is applied twice** — once globally before config load, and again when `getConfig` overrides `cfg.LogLevel`. The second application persists into the running instance and always wins over the YAML `logLevel`. Source: [`cmd/erpc/main.go:L354-363`](https://github.com/erpc/erpc/blob/main/cmd/erpc/main.go#L354-L363) 12. **OTel debug logging (zerologr bridge) is wired only at tracing init time.** If zerolog level is not `<= Debug` when `InitializeTracing` runs, OTel SDK internals are never logged, even if you lower the level later via the admin API. Source: [`common/tracing_core.go:L119-122`](https://github.com/erpc/erpc/blob/main/common/tracing_core.go#L119-L122) 13. **`console.error` in TypeScript config scripts throws `TypeError`.** Only `debug`, `info`, `log`, `trace`, and `warn` are registered in the Sobek runtime. Source: [`common/console.go:L15-21`](https://github.com/erpc/erpc/blob/main/common/console.go#L15-L21) ### Observability There are no Prometheus metrics emitted by the tracing or logging subsystem. The tracing pipeline is a pure OTel concern; OTLP export errors are logged at `trace` level only and do not increment any counter. **Notable log messages:** | Level | Message | When | |---|---|---| | info | `"OpenTelemetry tracing is disabled"` | `tracing.enabled = false` | | info | `"initializing OpenTelemetry tracing"` + endpoint/protocol/sampleRate/detailed | Before exporter creation | | info | `"OpenTelemetry debug logging enabled"` | Only if zerolog level ≤ debug at init time | | info | `"OpenTelemetry tracing initialized successfully"` | After successful init | | info | `"force-trace matcher configured"` + index/network/method | Once per configured matcher | | trace | `"open telemetry export error"` | OTLP export failure | | error | `"failed to create span exporter"` | Exporter init failure | | error | `"failed to create resource"` | OTel resource build failure | | error | `"failed to initialize tracing"` | From `NewERPC` when `InitializeTracing` errors | | error | `"failed to shutdown tracer provider"` | On process shutdown | | warn | `"invalid log level '%s', defaulting to 'debug'"` | Unrecognized `logLevel` value | | info | (empty message, `config` JSON field) | Startup when level ≤ info: full redacted config dump | **OTel span name inventory.** "normal" = emitted when `tracing.enabled`; "detail" = emitted only when `tracing.enabled && tracing.detailed`; "direct" = calls `tracer.Start` directly (same threshold as normal). OTel instrumentation name: `"github.com/erpc/erpc"`. All 126 span names (expand) | Span name | Mode | Call site | |---|---|---| | `Cache.FindGetPolicies` | detail | `architecture/evm/json_rpc_cache.go:L173` | | `Cache.Get` | normal | `architecture/evm/json_rpc_cache.go:L153` | | `Cache.GetForPolicy` | detail | `architecture/evm/json_rpc_cache.go:L241` | | `Cache.Set` | normal | `architecture/evm/json_rpc_cache.go:L572` | | `ConnectorFailsafe.Delete` | detail | `data/failsafe.go:L286` | | `ConnectorFailsafe.Get` | detail | `data/failsafe.go:L224` | | `ConnectorFailsafe.Set` | detail | `data/failsafe.go:L253` | | `Consensus.CollectResponses` | detail | `consensus/executor.go:L189` | | `Consensus.Run` | normal | `consensus/executor.go:L1282` | | `CounterInt64.TryUpdate` | normal | `data/shared_state_variable.go:L365` | | `CounterInt64.TryUpdateIfStale` | normal | `data/shared_state_variable.go:L391` | | `CounterInt64.TryUpdateIfStale.AcquireMutex` | normal | `data/shared_state_variable.go:L405` | | `CounterInt64.TryUpdateIfStale.ExecuteRefresh` | normal | `data/shared_state_variable.go:L430` | | `DynamoDBConnector.Delete` | normal | `data/dynamodb.go:L833` | | `DynamoDBConnector.Get` | normal | `data/dynamodb.go:L414` | | `DynamoDBConnector.List` | normal | `data/dynamodb.go:L874` | | `DynamoDBConnector.Lock` | normal | `data/dynamodb.go:L579` | | `DynamoDBConnector.Set` | normal | `data/dynamodb.go:L355` | | `DynamoDBConnector.Unlock` | normal | `data/dynamodb.go:L691` | | `DynamoDBConnector.getSimpleValue` | detail | `data/dynamodb.go:L766` | | `Evm.ExtractBlockReferenceFromRequest` | detail | `architecture/evm/block_ref.go:L19` | | `Evm.ExtractBlockReferenceFromResponse` | detail | `architecture/evm/block_ref.go:L118` | | `Evm.ExtractBlockTimestampFromResponse` | detail | `architecture/evm/block_ref.go:L191` | | `Evm.PickHighestBlock` | detail | `architecture/evm/eth_getBlockByNumber.go:L336` | | `Evm.extractRefFromJsonRpcRequest` | detail | `architecture/evm/block_ref.go:L240` | | `Evm.extractRefFromJsonRpcResponse` | detail | `architecture/evm/block_ref.go:L314` | | `EvmStatePoller.PollFinalizedBlockNumber` | detail | `architecture/evm/evm_state_poller.go:L491` | | `EvmStatePoller.PollLatestBlockNumber` | detail | `architecture/evm/evm_state_poller.go:L394` | | `GrpcBdsClient.GetBlockByHash` | detail | `clients/grpc_bds_client.go:L365` | | `GrpcBdsClient.GetBlockByNumber` | detail | `clients/grpc_bds_client.go:L422` | | `GrpcBdsClient.GetLogs` | detail | `clients/grpc_bds_client.go:L632` | | `GrpcBdsClient.QueryBlocks` | detail | `clients/grpc_bds_client.go:L1097` | | `GrpcBdsClient.QueryLogs` | detail | `clients/grpc_bds_client.go:L1173` | | `GrpcBdsClient.QueryTraces` | detail | `clients/grpc_bds_client.go:L1209` | | `GrpcBdsClient.QueryTransactions` | detail | `clients/grpc_bds_client.go:L1138` | | `GrpcBdsClient.QueryTransfers` | detail | `clients/grpc_bds_client.go:L1245` | | `GrpcBdsClient.SendRequest` | normal | `clients/grpc_bds_client.go:L194` | | `Http.ParseRequests` | detail | `erpc/http_server.go:L397` | | `Http.ReadBody` | detail | `erpc/http_server.go:L373` | | `Http.ReceivedRequest` | normal | `common/tracing_util.go:L96` | | `HttpJsonRpcClient.sendSingleRequest` | normal | `clients/http_json_rpc_client.go:L675` | | `HttpServer.WriteResponse` | detail | `erpc/http_server.go:L680` | | `JsonRpcRequest.Lock` | detail | `common/json_rpc.go:L1229` | | `JsonRpcRequest.RLock` | detail | `common/json_rpc.go:L1235` | | `JsonRpcResponse.IsResultEmptyish` | detail | `common/json_rpc.go:L799` | | `JsonRpcResponse.ParseFromStream` | detail | `common/json_rpc.go:L288` | | `JsonRpcResponse.PeekBytesByPath` | detail | `common/json_rpc.go:L486` | | `JsonRpcResponse.PeekStringByPath` | detail | `common/json_rpc.go:L464` | | `Multiplexer.Close` | detail | `erpc/multiplexer.go:L37` | | `Network.EnrichStatePoller` | detail | `erpc/networks.go:L2146` | | `Network.EvmHighestFinalizedBlockNumber` | detail | `erpc/networks.go:L679` | | `Network.EvmHighestLatestBlockNumber` | detail | `erpc/networks.go:L518` | | `Network.EvmLowestFinalizedBlockNumber` | detail | `erpc/networks.go:L855` | | `Network.Forward` | normal | `erpc/networks.go:L943` | | `Network.GetFinality` | detail | `erpc/networks.go:L1646` | | `Network.NormalizeResponse` | detail | `erpc/networks.go:L2235` | | `Network.PostForward.eth_getBlockByNumber` | detail | `architecture/evm/eth_getBlockByNumber.go:L44` | | `Network.PostForward.eth_sendRawTransaction` | detail | `architecture/evm/eth_sendRawTransaction.go:L280` | | `Network.PostForwardHook` | detail | `architecture/evm/hooks.go:L64` | | `Network.PreForwardHook` | detail | `architecture/evm/hooks.go:L41` | | `Network.PreForwardHook.eth_chainId` | detail | `architecture/evm/eth_chainId.go:L79` | | `Network.TryForward` | detail | `erpc/networks.go:L1137` | | `Network.UpstreamLoop` | detail | `erpc/networks.go:L1227` | | `Network.WaitForMultiplexResult` | normal | `erpc/networks.go:L2058` | | `Network.forwardAttempt` | normal | `erpc/networks.go:L1188` | | `PolicyEngine.GetOrdered` | detail | `erpc/networks.go:L1023` | | `PostgreSQLConnector.Delete` | normal | `data/postgresql.go:L1130` | | `PostgreSQLConnector.Get` | normal | `data/postgresql.go:L466` | | `PostgreSQLConnector.List` | normal | `data/postgresql.go:L1165` | | `PostgreSQLConnector.Lock` | normal | `data/postgresql.go:L526` | | `PostgreSQLConnector.PublishCounterInt64` | normal | `data/postgresql.go:L683` | | `PostgreSQLConnector.Set` | normal | `data/postgresql.go:L409` | | `PostgreSQLConnector.Unlock` | normal | `data/postgresql.go:L589` | | `PostgreSQLConnector.getCurrentValue` | detail | `data/postgresql.go:L974` | | `PostgreSQLConnector.getWithWildcard` | detail | `data/postgresql.go:L1006` | | `Project.Forward` | detail | `erpc/projects.go:L103` | | `Project.PreForwardHook` | detail | `architecture/evm/hooks.go:L14` | | `Project.PreForwardHook.eth_blockNumber` | detail | `architecture/evm/eth_blockNumber.go:L16` | | `Project.PreForwardHook.eth_chainId` | detail | `architecture/evm/eth_chainId.go:L30` | | `Project.executeShadowRequest` | detail | `erpc/shadow.go:L82` | | `Query.Execute` | detail | `erpc/query_executor.go:L45` | | `Query.ForwardSubrequest` | detail | `erpc/query_shim.go:L427` | | `Query.ResolveQueryBounds` | detail | `erpc/query_executor.go:L277` | | `Query.ShimBlocks` | detail | `erpc/query_shim.go:L18` | | `Query.ShimLogs` | detail | `erpc/query_shim.go:L112` | | `Query.ShimTraces` | detail | `erpc/query_shim.go:L187` | | `Query.ShimTransactions` | detail | `erpc/query_shim.go:L55` | | `QueryStream.Handle` | normal | `erpc/request_processor.go:L76` | | `RateLimiter.DoLimit` | normal | `upstream/ratelimiter_budget.go:L279` | | `RateLimiter.TryAcquirePermit` | detail | `upstream/ratelimiter_budget.go:L161` | | `RedisConnector.Delete` | normal | `data/redis.go:L636` | | `RedisConnector.Get` | normal | `data/redis.go:L341` | | `RedisConnector.List` | normal | `data/redis.go:L682` | | `RedisConnector.Lock` | normal | `data/redis.go:L435` | | `RedisConnector.PublishCounterInt64` | normal | `data/redis.go:L558` | | `RedisConnector.Set` | normal | `data/redis.go:L281` | | `RedisConnector.Unlock` | normal | `data/redis.go:L612` | | `Request.GenerateCacheHash` | detail | `common/json_rpc.go:L1387` | | `Request.Handle` | direct | `common/tracing_util.go:L165` | | `Request.Lock` | detail | `common/request.go:L925` | | `Request.RLock` | detail | `common/request.go:L931` | | `Request.ResolveJsonRpc` | detail | `common/request.go:L939` | | `Response.IsObjectNull` | detail | `common/response.go:L420` | | `Response.Lock` | detail | `common/response.go:L84` | | `Response.RLock` | detail | `common/response.go:L90` | | `Response.ResolveJsonRpc` | detail | `common/response.go:L293` | | `Upstream.Forward` | normal | `upstream/upstream.go:L416` | | `Upstream.PostForwardHook` | detail | `architecture/evm/hooks.go:L110` | | `Upstream.PostForwardHook.eth_getBlockByNumber` | detail | `architecture/evm/eth_getBlockByNumber.go:L435` | | `Upstream.PostForwardHook.eth_getBlockReceipts` | detail | `architecture/evm/eth_getBlockReceipts.go:L31` | | `Upstream.PostForwardHook.eth_getLogs` | detail | `architecture/evm/eth_getLogs.go:L350` | | `Upstream.PostForwardHook.eth_sendRawTransaction` | detail | `architecture/evm/eth_sendRawTransaction.go:L59` | | `Upstream.PostForwardHook.trace_filter` | detail | `architecture/evm/trace_filter.go:L362` | | `Upstream.PreForwardHook` | detail | `architecture/evm/hooks.go:L87` | | `Upstream.PreForwardHook.eth_chainId` | detail | `architecture/evm/eth_chainId.go:L124` | | `Upstream.PreForwardHook.eth_getLogs` | detail | `architecture/evm/eth_getLogs.go:L289` | | `Upstream.PreForwardHook.trace_filter` | detail | `architecture/evm/trace_filter.go:L298` | | `Upstream.tryForward.PreRequest` | detail | `upstream/upstream.go:L572` | | `Upstream.tryForward.SendRequest` | detail | `upstream/upstream.go:L630` | | `UpstreamsRegistry.GetNetworkUpstreams` | detail | `upstream/registry.go:L349` | | `UpstreamsRegistry.GetSortedUpstreams` | detail | `upstream/registry.go:L387` | | `UpstreamsRegistry.buildProviderBootstrapTask` | detail | `upstream/registry.go:L485` | | `UpstreamsRegistry.buildUpstreamBootstrapTask` | detail | `upstream/registry.go:L431` | | `createSyntheticSuccessResponse` | detail | `architecture/evm/eth_sendRawTransaction.go:L179` | | `extractTxHashFromSendRawTransaction` | detail | `architecture/evm/eth_sendRawTransaction.go:L136` | | `verifyAndHandleNonceTooLow` | detail | `architecture/evm/eth_sendRawTransaction.go:L206` | ### Source code entry points - [`common/tracing_core.go:L53-L170`](https://github.com/erpc/erpc/blob/main/common/tracing_core.go#L53-L170) — `InitializeTracing`: exporter creation, sampler construction, propagator setup, `forceTraceSampler`, `IsTracingEnabled`/`IsTracingDetailed` globals - [`common/tracing_util.go:L1-L234`](https://github.com/erpc/erpc/blob/main/common/tracing_util.go#L1-L234) — `noopSpan` singleton, `StartSpan`/`StartDetailSpan` helpers, `StartHTTPServerSpan`, `StartRequestSpan`/`EndRequestSpan`, `InjectHTTPResponseTraceContext` - [`common/defaults.go:L618-L637`](https://github.com/erpc/erpc/blob/main/common/defaults.go#L618-L637) — `TracingConfig.SetDefaults`: protocol, endpoint, sampleRate, serviceName - [`common/config.go:L218-L235`](https://github.com/erpc/erpc/blob/main/common/config.go#L218-L235) — `TracingConfig` struct definition - [`common/config.go:L397-L503`](https://github.com/erpc/erpc/blob/main/common/config.go#L397-L503) — `MarshalJSON`/`MarshalYAML` redaction overrides for secret-bearing config types - [`util/redact.go:L10-L36`](https://github.com/erpc/erpc/blob/main/util/redact.go#L10-L36) — `RedactEndpoint`: SHA-256-based URL sanitization with scheme-aware rules - [`cmd/erpc/main.go:L48-L66`](https://github.com/erpc/erpc/blob/main/cmd/erpc/main.go#L48-L66) — zerolog global setup: `TimeFieldFormat`, `ErrorMarshalFunc`, `LOG_WRITER` console mode, `LOG_LEVEL` env override - [`erpc/erpc.go:L31-L99`](https://github.com/erpc/erpc/blob/main/erpc/erpc.go#L31-L99) — calls `InitializeTracing` on startup, registers shutdown goroutine with 5-second grace period - [`erpc/http_server.go:L285-L288`](https://github.com/erpc/erpc/blob/main/erpc/http_server.go#L285-L288) — network context for force-trace matcher evaluation - [`erpc/http_server.go:L701`](https://github.com/erpc/erpc/blob/main/erpc/http_server.go#L701) — W3C traceparent injection into HTTP response headers - [`erpc/init.go:L27-L42`](https://github.com/erpc/erpc/blob/main/erpc/init.go#L27-L42) — parses `cfg.LogLevel` into zerolog level; logs full redacted config at info level - [`common/console.go:L15-L29`](https://github.com/erpc/erpc/blob/main/common/console.go#L15-L29) — JS/TS `console.{debug,info,log,trace,warn}` → zerolog bridge (note: `console.error` is absent) - [`common/runtime.go:L39`](https://github.com/erpc/erpc/blob/main/common/runtime.go#L39) — installs `console` object into every Sobek runtime - [`cmd/erpc/initflags.go`](https://github.com/erpc/erpc/blob/main/cmd/erpc/initflags.go) — `ERPC_NOLOGS=1` → zerolog.Disabled + io.Discard writer (build-tagged, non-test only) ### Related pages - [Metrics & Prometheus](/operation/metrics.llms.txt) — the Prometheus side of observability; no overlap with OTel spans. - [Admin API](/operation/admin.llms.txt) — runtime log-level changes via the admin endpoint. - [Health checks](/operation/health-check.llms.txt) — healthcheck requests bypass force-trace network matchers. - [Deployment](/deployment/docker.llms.txt) — how to wire an OTel collector sidecar alongside eRPC. --- ## Navigation (machine-readable surface) - Up: [All pages index](https://docs.erpc.cloud/llms.txt) - Root index of every page: [llms.txt](https://docs.erpc.cloud/llms.txt) · everything in one file: [llms-full.txt](https://docs.erpc.cloud/llms-full.txt) ### Sibling pages - [Admin API](https://docs.erpc.cloud/operation/admin.llms.txt) — A built-in operator control plane — inspect topology, cordon sick upstreams without restarts, and manage API keys, all over a secure JSON-RPC 2.0 endpoint. - [Batching & multiplexing](https://docs.erpc.cloud/operation/batch.llms.txt) — Send one request, get back a merged response — eRPC parallelises inbound batch arrays, re-batches calls to supporting upstreams, and collapses identical in-flight requests so each unique call hits the network exactly once. - [CLI & env vars](https://docs.erpc.cloud/operation/cli.llms.txt) — Start, validate, or inspect your eRPC config from the command line — then deploy with confidence knowing exactly what the engine will run. - [Cordoning](https://docs.erpc.cloud/operation/cordoning.llms.txt) — Pull any upstream out of routing instantly with one admin call — no metric window to wait for, no config redeploy required. - [Directives](https://docs.erpc.cloud/operation/directives.llms.txt) — Send an HTTP header or query param and change routing, caching, validation, or consensus for exactly that one request — no restarts, no config changes. - [Healthcheck](https://docs.erpc.cloud/operation/healthcheck.llms.txt) — One endpoint that tells Kubernetes exactly when your pod is ready, draining, or broken — with eight probe strategies from "any upstream alive" to live chain-ID verification. - [Monitoring & metrics](https://docs.erpc.cloud/operation/monitoring.llms.txt) — Every subsystem in eRPC — upstreams, cache, rate limits, consensus, hedging — emits Prometheus metrics. One scrape target, full visibility, zero instrumentation work. - [Production checklist](https://docs.erpc.cloud/operation/production.llms.txt) — Go live confidently — a short list of settings that separate a hardened eRPC deployment from a dev-mode one. - [URL structure](https://docs.erpc.cloud/operation/url.llms.txt) — One URL pattern routes every chain — domain and network aliases let you publish clean, memorable endpoints without touching your app code.