OpenTelemetry tracing

TracingConfig — every field

Key spans you'll see:

• HTTP server request handling
• Network-level (chain) forwarding — the Network.Forward span carries the full per-request execution trace (execution.attempts, upstreams.tried, upstreams.outcomes, upstreams.reasons, upstreams.durations_ms). See failsafe → Per-attempt observability.
• Upstream-level request forwarding
• Cache operations (get/set)
• Failsafe executor operations (hedges, retries, timeouts, breaker probes)
• HTTP client requests to upstreams
• Rate limiters

TLS sub-fields (when protocol: grpc with TLS)

The tls block uses the shared TLSConfig struct. See TLS configuration for the full field reference (enabled, certFile, keyFile, caFile, insecureSkipVerify).

Example for a gRPC OTLP collector with mutual-TLS:

tracing:
  endpoint: collector.example.com:4317
  protocol: grpc
  tls:
    enabled: true
    certFile: /path/to/client.crt        # optional client cert
    keyFile: /path/to/client.key         # optional client key
    caFile: /path/to/ca.crt              # CA chain for verifying the collector
    insecureSkipVerify: false            # set true for development; never in prod

ForceTraceMatcher

forceTraceMatchers:
  - network: <matcher-pattern>
    method: <matcher-pattern>

A request is force-traced if any matcher's network AND method both match. Force-traced requests bypass sampleRate entirely (treated as sampled-in regardless of the random draw).

Common uses:

• Always trace expensive methods: { method: "debug_*|trace_*" }
• Always trace one chain that's the focus of debugging: { network: "evm:42161" }
• Trace one method on one chain during a customer escalation: { network: "evm:1", method: "eth_call" }

Headers — concrete examples

For authenticated OTLP collectors:

# Honeycomb
tracing:
  endpoint: api.honeycomb.io:443
  protocol: grpc
  headers:
    x-honeycomb-team: ${HONEYCOMB_API_KEY}
 
# Grafana Cloud Tempo (basic auth via Authorization header)
tracing:
  endpoint: tempo-eu-west-0.grafana.net:443
  protocol: grpc
  headers:
    Authorization: Basic \${GRAFANA_CLOUD_BASIC_AUTH}
 
# Self-hosted with mTLS, no headers needed (use tls.* instead)

Authorization headers should use env-var interpolation; never commit credentials to YAML/TS.

Traced components

Every span eRPC emits is a child of an HTTP server root span. The instrumented layers:

detailed: true adds finer-grained internal spans (mutexes, internal selection-policy bookkeeping) and high-cardinality attributes.

Common pitfalls

• sampleRate: 0.1 and 100% of force-traced requests — forceTraceMatchers operates on top of sampling; force-traced requests are not subject to the rate. So your collector ingests sampleRate * total + forceTraced. Make sure that's affordable.
• detailed: true in production — request params and cache keys can be huge (think eth_getLogs filters), and they hit your ingest cost. Reserve for debugging.
• headers interpolation — ${VAR} is only resolved at config load time. Rotating tokens requires a config reload.
• tls.insecureSkipVerify: true — accepts any cert. Never set in production; use a proper CA chain or mTLS.
• protocol: http without https:// in endpoint — exports go to plaintext HTTP. Most managed providers require HTTPS; set the endpoint to the https://.../v1/traces form.
• OTLP/HTTP endpoint format — protocol: http expects a FULL URL including the /v1/traces path segment; protocol: grpc expects just host:port without scheme.