OpenTelemetry Tracing

eRPC includes support for distributed tracing using OpenTelemetry. This allows you to track requests as they flow through the system, identify performance bottlenecks, and debug issues in production environments.

Config

To enable tracing, add the following to your erpc.yaml configuration:

tracing:
  enabled: true
  endpoint: "localhost:4317"  # OTLP endpoint (Jaeger, Tempo, etc.)
  protocol: "grpc"            # "grpc" or "http"
  samplingRatio: 0.1          # Sample 10% of requests
  detailed: true              # Include detailed tracing information
  tls:
    enabled: false            # Enable TLS for secure connections
    # certFile: "/path/to/cert.pem"
    # keyFile: "/path/to/key.pem"
    # caFile: "/path/to/ca.pem"
server:
 # ...
projects:
 # ...

Detailed tracing

When tracing.detailed is set to true, eRPC will include detailed tracing information in the traces. This includes:

Internal operations and mutex locks (useful to debug long requests that are not waiting for any I/O)
High-cardinality attributes (e.g. request json-rpc IDs, request params, actual cache keys used, etc.)

Remember that detailed tracing can significantly increase the volume of traces, so use it judiciously.

Using with Jaeger

The included docker-compose.yml (opens in a new tab) file contains a Jaeger service for visualizing traces. To use it:

Start the Jaeger container:
```
docker-compose up jaeger
```

Configure eRPC to send traces to Jaeger:

tracing:
  enabled: true
  endpoint: "localhost:4317"
  protocol: "grpc"
  samplingRatio: 1.0 # Sample all requests during development
  detailed: true

Access the Jaeger UI at http://localhost:16686 (opens in a new tab)

Traced components

The following components are instrumented with tracing:

HTTP server request handling
Network-level (chain) forwarding
Upstream-level request forwarding
Cache operations (get/set)
Failsafe executor operations (hedges, retries)
HTTP client requests to upstreams
Rate limiters
And more...

If you noticed a missing component from tracing, free free to open an issue or PR (opens in a new tab)!

Monitoring Admin