Tracing & logging

How it works

Tracing initialization. On startup, NewERPC calls common.InitializeTracing exactly once (guarded by sync.Once). When enabled, it builds an OTLP exporter (gRPC via otlptracegrpc or HTTP via otlptracehttp), constructs a TracerProvider with a batcher and a custom sampler, wires W3C TraceContext + Baggage propagation, and stores any forceTraceMatchers for per-request evaluation. Two package-level booleans — IsTracingEnabled and IsTracingDetailed — are set once and consulted on every hot-path span call with zero allocation when tracing is off. [common/tracing_core.go:L53-148]

Two span tiers. All instrumented call sites use one of two helpers:

• StartSpan — fires whenever IsTracingEnabled; covers major external interactions: cache, upstreams, connectors, rate limiters, consensus.
• StartDetailSpan — fires only when both IsTracingEnabled and IsTracingDetailed; covers eRPC-internal operations and spans carrying high-cardinality attributes such as request params, block numbers, lock contention, and hook execution. Enabling detailed: true roughly doubles span count.

When tracing is disabled, both helpers return the caller's unchanged context and a singleton noopSpan — no allocation on the hot path. [common/tracing_util.go:L1-234]

Span hierarchy for a single HTTP request (normal mode):

Http.ReceivedRequest  (SpanKindServer)
  └─ Request.Handle   (SpanKindInternal)
       └─ Network.Forward
            └─ Network.forwardAttempt
                 ├─ Cache.Get
                 └─ Upstream.Forward
                      └─ HttpJsonRpcClient.sendSingleRequest

With detailed: true, additional spans appear inside the HTTP layer (Http.ReadBody, Http.ParseRequests, HttpServer.WriteResponse), between Forward and forwardAttempt (Project.Forward, Network.TryForward, Network.UpstreamLoop), and inside upstream execution (RateLimiter.TryAcquirePermit, Upstream.tryForward.PreRequest, Upstream.tryForward.SendRequest), cache lookup, lock/parse ops, and method-specific hooks.

Sampler logic. createTracingSampler builds the base sampler from sampleRate:

• <= 0 → NeverSample()
• >= 1.0 → AlwaysSample()
• otherwise → ParentBased(TraceIDRatioBased(rate)) with all four parent-sampling options configured for consistency (no orphan spans)

The forceTraceSampler always wraps the base sampler regardless of rate. [common/tracing_core.go:L229-268]

Force-trace mechanism. Three paths can force a span to be recorded regardless of sampleRate:

1. HTTP header X-ERPC-Force-Trace: true (or 1, yes) — checked at request entry.
2. Query parameter ?force-trace=true — same check.
3. Config-driven forceTraceMatchers — evaluated after the URL is parsed and the network is known. Each matcher can specify network (in architecture:chainId form, e.g. evm:1) and/or method (JSON-RPC method pattern). Pipe (|) = OR; wildcards supported. Both fields present on one matcher = AND semantics.

In all cases, the force-trace decision is communicated to the OTel sampler by setting the span attribute erpc.force_trace = true at creation time. The custom forceTraceSampler wraps the base sampler and returns RecordAndSample unconditionally when it sees this attribute — even when sampleRate: 0. [common/tracing_core.go:L257-268]

Context propagation. Incoming HTTP requests have traceparent/tracestate extracted via W3C TraceContext{} carrier, making eRPC spans children of the caller's trace. After the response is written, InjectHTTPResponseTraceContext injects the active span context back into response headers so downstream callers can correlate their traces. gRPC requests have no OTel trace extraction — they start with a fresh root span. [common/tracing_util.go:L58-74]

Logging system. zerolog is the structured JSON logger used throughout eRPC. Default output is newline-delimited JSON with Unix-millisecond timestamps:

{"level":"info","time":1718000000123,"message":"...","key":"value"}

Set LOG_WRITER=console to switch to a human-readable format for local development:

04:05.000ms INF message key=value

Five levels are available: trace, debug, info, warn, error (plus fatal, panic, disabled). LOG_LEVEL environment variable overrides logLevel config at startup and again when the config is loaded, so it always wins. ERPC_NOLOGS=1 silences all output at the binary level.

When the resolved level is info or lower, eRPC serializes the entire loaded config as a JSON field in a startup log line. All secrets are redacted before this serialization.

Secret redaction. Two layers:

1. MarshalJSON/MarshalYAML overrides on config types — called whenever config is serialized — replace secret fields with "REDACTED" or apply util.RedactEndpoint:

   Config type	Field	Replacement
   RedisConnectorConfig	password	"REDACTED"
   RedisConnectorConfig	uri	util.RedactEndpoint(uri)
   PostgreSQLConnectorConfig	connectionUri	util.RedactEndpoint(connectionUri)
   AwsAuthConfig	secretAccessKey	"REDACTED"
   ProviderConfig	settings	"REDACTED"
   UpstreamConfig	endpoint	util.RedactEndpoint(endpoint)
   SecretStrategyConfig	value	"REDACTED"

   [common/config.go:L397-503]

2. util.RedactEndpoint — computes a SHA-256 of the full original URL, takes the first 5 hex chars as a stable identifier, then strips path and credentials according to scheme:

   • Native-protocol endpoints (alchemyv2://, erigon://, etc.): scheme://host#redacted=<hash>
   • Envio-suffix schemes: scheme://host (no hash needed — no secrets in path)
   • Repository-suffix schemes: scheme://host#redacted=<hash>
   • Standard http/https RPC endpoints: scheme#redacted=<hash> — hostname is also dropped
   • Unparseable URL: "redacted=<hash>"

   Two endpoints with the same scheme but different paths (API keys) produce different hashes, enabling correlation without exposing secrets.

[util/redact.go:L10-36]

OTel resource attributes. The TracerProvider is built with a resource that includes service.name (from tracing.serviceName), service.version (the compiled-in ErpcVersion constant), commit.sha (the compiled-in ErpcCommitSha constant), and any key/value pairs from tracing.resourceAttributes. Environment variables in resourceAttributes values are expanded via os.ExpandEnv at config load; empty values after expansion are silently skipped. [common/tracing_core.go:L93-102]

The OTel instrumentation name registered by eRPC is "github.com/erpc/erpc". [common/tracing_core.go:L26]

Error recording. SetTraceSpanError checks span.IsRecording() before acting. For StandardError types it sets an error.code span attribute (the full error-code chain) and records the error as a span event; for plain error values it calls span.RecordError and sets the span status to codes.Error. [common/tracing_core.go:L163-183]

Force-flush. common.ForceFlushTraces(ctx) calls tracerProvider.ForceFlush(ctx), flushing all batched but not-yet-exported spans to the collector immediately. It is called automatically on graceful shutdown and exposed for callers that need guaranteed delivery of critical traces. [common/tracing_core.go:L237-245]

Graceful shutdown. A goroutine watches appCtx.Done and flushes buffered spans with a 5-second grace period before the process exits. [erpc/erpc.go:L92-99]

Config schema

Top-level logLevel

tracing.*

Full struct: common/config.go:L218-235.

ForceTraceMatcher AND/OR semantics: when both network and method are specified on one matcher, both must match (AND). If only one field is set, only that field is checked. An empty matcher (neither field) never matches (common/tracing_core.go:L306-330).

Standard OTEL_* environment variables

Worked examples

1. Production setup: 10 % sampling with force-trace on debug methods. Low ambient cost; debug-namespace calls always produce a full trace for triage:

tracing:  enabled: true  endpoint: otel-collector:4317  protocol: grpc  sampleRate: 0.1  serviceName: erpc-prod  forceTraceMatchers:    - method: "debug_*|trace_*"

2. Detailed mode for a staging environment. Capture every span including request params, block numbers, and lock contention to understand slow calls — not recommended in production due to 2× span volume:

tracing:  enabled: true  endpoint: tempo:4317  sampleRate: 1.0  detailed: true  serviceName: erpc-staging

3. Managed collector with auth headers and TLS. For Grafana Cloud or Honeycomb where the OTLP endpoint requires a bearer token:

tracing:  enabled: true  endpoint: "https://otlp-gateway.grafana.net/otlp"  protocol: http  sampleRate: 0.05  headers:    Authorization: "Basic <base64-token>"

4. Silent tracing for force-trace-only capture. Keep tracing on but produce no ambient spans — only requests that carry X-ERPC-Force-Trace: true or match a configured matcher are recorded. Use sampleRate: 0.0000001 because sampleRate: 0 is rewritten to 1.0 by SetDefaults:

tracing:  enabled: true  endpoint: localhost:4317  sampleRate: 0.0000001  forceTraceMatchers:    - network: "evm:1"      method: "eth_call"

Request/response behavior

• Incoming HTTP requests with traceparent/tracestate headers become children of the caller's trace. eRPC spans are nested under the caller's root span.
• After the response is written, the active span's W3C context is injected back into HTTP response headers unconditionally — including error responses. Trace IDs are observable to callers. [erpc/http_server.go:L701]
• For batch HTTP requests, each item gets its own Request.Handle child span under a single Http.ReceivedRequest parent. All child spans may be in-flight simultaneously. [erpc/http_server.go:L465]
• gRPC requests start a fresh root span — no trace context extraction is implemented for the gRPC server path. [erpc/grpc_server.go]
• OTLP export failures are logged at trace level only and never propagate to callers. [common/tracing_core.go:L125-127]
• Request.Handle is started via tracer.Start directly (not StartSpan), so it fires whenever IsTracingEnabled, regardless of IsTracingDetailed. This is the correct attach point for distributed callers. [common/tracing_util.go:L165]

Span attributes on Request.Handle (set by StartRequestSpan/EndRequestSpan):

Span attributes on Http.ReceivedRequest (set by StartHTTPServerSpan/EnrichHTTPServerSpan):

[common/tracing_util.go:L80-138]

Best practices

• Set sampleRate: 0.05–0.2 in production — AlwaysSample() at scale sends enormous span volume to your collector.
• Use forceTraceMatchers with method: "debug_*|trace_*" to guarantee full traces on expensive diagnostic methods regardless of sample rate — this costs nothing extra for normal traffic.
• Never set sampleRate: 0 thinking it means "disabled" — SetDefaults rewrites it to 1.0. Use enabled: false to disable tracing entirely.
• Enable detailed: true only in staging or on-demand: 2× span count and high-cardinality attributes (params, block numbers) can saturate a collector or inflate storage costs.
• Always set tracing.endpoint explicitly — do not rely on OTEL_EXPORTER_OTLP_ENDPOINT to route spans to a remote collector; eRPC overrides it unconditionally.
• Enable tracing.tls.enabled: true for any span export that leaves a trusted network; the default is WithInsecure().
• Force-trace via the X-ERPC-Force-Trace: true request header during integration testing to capture full detail without raising global sample rate.

Edge cases & gotchas

1. initOnce.Do means tracing config cannot change after first NewERPC. Tests or multi-tenant code creating multiple ERPC instances will silently use only the first instance's tracing config. Source: common/tracing_core.go:L53
2. sampleRate: 0 in YAML means "always sample", not "never sample". SetDefaults replaces 0 with 1.0. To keep tracing enabled but silent, set 0.0000001 or a very small positive value. Source: common/defaults.go:L629-631
3. Force-trace network matching requires URL parsing to have succeeded. forceTraceMatchers with network patterns never match admin or healthcheck requests. Source: erpc/http_server.go:L285-288
4. W3C traceparent injection into HTTP responses is unconditional when tracing is enabled — including error responses. Trace IDs are observable to callers. Source: erpc/http_server.go:L701
5. gRPC server has no OTel trace extraction. gRPC-originated requests start with a fresh root span, not as children of the caller's trace. Source: erpc/grpc_server.go
6. TLS on the OTLP exporter is off by default. Both gRPC and HTTP exporters use WithInsecure() unless tracing.tls.enabled: true. Production deployments sending spans over the internet should enable TLS. Source: common/tracing_core.go:L187
7. OTEL_EXPORTER_OTLP_ENDPOINT is silently overridden. eRPC always calls WithEndpoint(cfg.Endpoint), so this env var has no effect. Set tracing.endpoint in config explicitly. Source: common/tracing_core.go:L197
8. OTEL_TRACES_SAMPLER and OTEL_PROPAGATORS have no effect. eRPC supplies its own sampler and propagator via explicit SDK options. Source: common/tracing_core.go:L109
9. OTEL_EXPORTER_OTLP_HEADERS only takes effect when tracing.headers is nil/empty in config. Source: common/tracing_core.go:L200-202
10. util.RedactEndpoint drops the entire hostname for standard http/https URLs. An endpoint like https://eth-mainnet.alchemyapi.io/v2/SECRET becomes https#redacted=ab3c4. Only native-protocol and repository-type endpoints keep scheme://host. Source: util/redact.go:L32-34
11. LOG_LEVEL env var is applied twice — once globally before config load, and again when getConfig overrides cfg.LogLevel. The second application persists into the running instance and always wins over the YAML logLevel. Source: cmd/erpc/main.go:L354-363
12. OTel debug logging (zerologr bridge) is wired only at tracing init time. If zerolog level is not <= Debug when InitializeTracing runs, OTel SDK internals are never logged, even if you lower the level later via the admin API. Source: common/tracing_core.go:L119-122
13. console.error in TypeScript config scripts throws TypeError. Only debug, info, log, trace, and warn are registered in the Sobek runtime. Source: common/console.go:L15-21

Observability

There are no Prometheus metrics emitted by the tracing or logging subsystem. The tracing pipeline is a pure OTel concern; OTLP export errors are logged at trace level only and do not increment any counter.

Notable log messages:

OTel span name inventory. "normal" = emitted when tracing.enabled; "detail" = emitted only when tracing.enabled && tracing.detailed; "direct" = calls tracer.Start directly (same threshold as normal). OTel instrumentation name: "github.com/erpc/erpc".

Source code entry points

• common/tracing_core.go:L53-L170 (opens in a new tab) — InitializeTracing: exporter creation, sampler construction, propagator setup, forceTraceSampler, IsTracingEnabled/IsTracingDetailed globals
• common/tracing_util.go:L1-L234 (opens in a new tab) — noopSpan singleton, StartSpan/StartDetailSpan helpers, StartHTTPServerSpan, StartRequestSpan/EndRequestSpan, InjectHTTPResponseTraceContext
• common/defaults.go:L618-L637 (opens in a new tab) — TracingConfig.SetDefaults: protocol, endpoint, sampleRate, serviceName
• common/config.go:L218-L235 (opens in a new tab) — TracingConfig struct definition
• common/config.go:L397-L503 (opens in a new tab) — MarshalJSON/MarshalYAML redaction overrides for secret-bearing config types
• util/redact.go:L10-L36 (opens in a new tab) — RedactEndpoint: SHA-256-based URL sanitization with scheme-aware rules
• cmd/erpc/main.go:L48-L66 (opens in a new tab) — zerolog global setup: TimeFieldFormat, ErrorMarshalFunc, LOG_WRITER console mode, LOG_LEVEL env override
• erpc/erpc.go:L31-L99 (opens in a new tab) — calls InitializeTracing on startup, registers shutdown goroutine with 5-second grace period
• erpc/http_server.go:L285-L288 (opens in a new tab) — network context for force-trace matcher evaluation
• erpc/http_server.go:L701 (opens in a new tab) — W3C traceparent injection into HTTP response headers
• erpc/init.go:L27-L42 (opens in a new tab) — parses cfg.LogLevel into zerolog level; logs full redacted config at info level
• common/console.go:L15-L29 (opens in a new tab) — JS/TS console.{debug,info,log,trace,warn} → zerolog bridge (note: console.error is absent)
• common/runtime.go:L39 (opens in a new tab) — installs console object into every Sobek runtime
• cmd/erpc/initflags.go (opens in a new tab) — ERPC_NOLOGS=1 → zerolog.Disabled + io.Discard writer (build-tagged, non-test only)

Related pages

• Metrics & Prometheus — the Prometheus side of observability; no overlap with OTel spans.
• Admin API — runtime log-level changes via the admin endpoint.
• Health checks — healthcheck requests bypass force-trace network matchers.
• Deployment — how to wire an OTel collector sidecar alongside eRPC.