Shadow upstreams

How it works

Registration. An upstream becomes a shadow by setting shadow.enabled: true. The registry (upstream/registry.go:L574-L584) routes it into networkShadowUpstreams[networkId] instead of networkUpstreams[networkId]. Shadow upstreams are never offered to the selection policy, never scored by the health tracker, and never returned by GetNetworkUpstreams. The normal routing path cannot reach them.

Trigger point. After p.doForward returns in PreparedProject.Forward (erpc/projects.go:L135-L163), if the network has shadow upstreams and the primary response is non-nil, eRPC first parses the primary response as a JSON-RPC object — if parsing fails it logs and skips shadow entirely. On success it clones both the response and request — full byte copies, not references — copying fields including upstream, fromCache, attempts, retries, hedges, evmBlockRef, and evmBlockNumber — then launches executeShadowRequests as a goroutine. The primary response is already committed to the caller before the goroutine starts.

Goroutine safety. The shadow goroutine receives cloned objects, not live references. Original request body is byte-copied with append([]byte(nil), body...); if the body is nil, the JSON-RPC object is marshalled to bytes and a new NormalizedRequest is constructed. Request directives and HTTP context are also copied. The JSON-RPC response is deep-copied into a new NormalizedResponse. No races between primary and shadow are possible.

Context lifecycle. The shadow goroutine derives its context from p.networksRegistry.appCtx (not the inbound request context). Client disconnects do not cancel shadow requests.

Per-upstream loop. For each shadow upstream, eRPC launches a dedicated goroutine with a fresh context derived from appCtx: (1) ShouldHandleMethod check — skip silently if the upstream's allow/ignore lists reject the method; (2) sample rate check — if rand.Float64() >= sampleRate, skip with debug log; (3) forward with byPassMethodExclusion=true (upstream/upstream.go:L410-L450) so the selection-engine's shouldSkip guard doesn't block it; (4) compare responses; (5) call shadowResp.Release() to free the cloned response.

Comparison algorithm. eRPC computes a content hash of both the primary and shadow responses. If ignoreFields is configured for the method, both hashes are computed with those JSON keys masked before comparison. Result is one of three outcomes: Identical (hashes match, or both emptyish) — logged at Trace; Mismatch (hashes differ) — logged at Error with full request, both responses, both hashes, and the ignored fields list; Error (shadow request failed or hash computation failed) — counter incremented with stable error fingerprint.

Panic isolation. A defer recover() in executeShadowRequests catches any panic, logs it, and increments erpc_unexpected_panic_total{type="shadow-upstreams"}. Shadow panics never propagate to the caller.

Config schema

All fields live under projects[*].upstreams[*].shadow. Struct at common/config.go:L982-986.

Worked examples

All patterns below are distilled from real production fleets; comments explain the non-obvious choices.

1. Archive node dark-launch with block-range scoping. A common production pattern: a new archive node only has data above a certain block, so blockAvailability.lower gates which requests can reach it — shadowing only fires on requests the node can actually answer, avoiding a flood of spurious error-total increments:

upstreams:  - id: new-archive-shadow    endpoint: https://rpc.example.com    vendorName: my-vendor    evm:      chainId: 1      blockAvailability:        lower:          # only shadow requests for blocks at or above this height;          # below it the node doesn't have data, so comparison is meaningless          exactBlock: 18820001    failsafe:      - timeout:          duration: 5s          # adaptive timeout: p95 of node's own latency distribution, clamped          # to [2s, 5s] so slow nodes don't hold the shadow goroutine open forever          quantile: 0.95          minDuration: 2s          maxDuration: 5s    shadow:      enabled: true      # nil sampleRate = always fire; start here and tune down if shadow load is high

2. Low-rate background validation — standard provider comparison. 10% sample rate is the typical starting point on busy fleets: shadow goroutines share outbound connection pools with normal upstreams so 100% shadow can double load on the candidate provider. Ignore timestamp on block methods to suppress clock-skew false positives:

upstreams:  - id: candidate-shadow    endpoint: https://rpc.example.com    shadow:      enabled: true      # 10% gives statistically meaningful coverage without doubling upstream load      sampleRate: 0.1      ignoreFields:        # providers differ on timestamp by ±1s due to clock skew — not a real mismatch        eth_getBlockByNumber: ["timestamp"]        eth_getBlockByHash: ["timestamp"]

3. L2 chain with many volatile L1 fee fields. OP-stack and Arbitrum chains embed L1 fee data in receipts that can differ between providers depending on which L1 oracle snapshot they use. The ignoreFields wildcard *.fieldName masks the field at every depth of an array response — critical for eth_getLogs and eth_getBlockReceipts where each entry in the array carries L1 fee metadata:

upstreams:  - id: l2-node-shadow    endpoint: https://rpc.example.com    shadow:      enabled: true      sampleRate: 0.1      ignoreFields:        # array-level wildcard: masks the field at every element of the logs array        eth_getLogs:          - "*.blockTimestamp"        # receipts carry multiple L1 fee sub-fields that differ by L1 oracle snapshot        eth_getBlockReceipts:          - "*.blockTimestamp"          - "*.l1BaseFeeScalar"          - "*.l1BlobBaseFee"          - "*.l1BlobBaseFeeScalar"          - "*.l1BlockNumber"          - "*.l1Fee"          - "*.l1GasPrice"          - "*.l1GasUsed"          - "*.logs.*.blockTimestamp"

4. Method-scoped shadow for log-indexing validation. When the only concern is whether a new node's event indexing matches production, scope the shadow to eth_getLogs only. The allowMethods list on the upstream ensures the shadow goroutine never attempts other methods (the ShouldHandleMethod pre-check silently skips them), keeping shadow traffic narrow and noise-free:

upstreams:  - id: logs-specialist-shadow    endpoint: https://rpc.example.com    # only shadow eth_getLogs — ShouldHandleMethod skips everything else silently    allowMethods:      - eth_getLogs    shadow:      enabled: true      sampleRate: 0.2

5. Pause shadow without removing config. Setting sampleRate: 0 suppresses all shadow traffic while preserving the full config. Flip to 0.1 to resume — no config restructuring, no deployment diff noise:

shadow:
  enabled: true
  sampleRate: 0   # all traffic suppressed; rand.Float64() >= 0.0 is always true
                  # change to 0.1 to resume 10% shadowing

Request/response behavior

• Shadow requests run asynchronously after the primary response is committed to the client. They add zero latency to the client-visible path.
• Shadow goroutines use appCtx — client disconnect does not cancel them. They always run to completion.
• byPassMethodExclusion=true is passed to ups.Forward so the shadow goroutine bypasses the shouldSkip selection-engine check. Method exclusion from the upstream config is still enforced by the pre-call ShouldHandleMethod check in the shadow layer itself. [erpc/shadow.go:L54-61]
• A shadow response that errors or returns nil increments erpc_shadow_response_error_total — no mismatch is recorded. The error label is a stable fingerprint, not the raw error string.
• Mismatch log level is Error — these will appear in error-rate dashboards and alert on any divergence. [erpc/shadow.go:L253-261]
• OTel trace span "Project.executeShadowRequest" is emitted per shadow goroutine, enabling independent shadow latency measurement in your tracing backend. [erpc/shadow.go:L82]

Best practices

• Start with sampleRate: 0.05–0.1 on high-QPS deployments. Shadow goroutines share outbound connection pools with normal upstreams on the same host; 100% shadow on a busy fleet can double upstream load.
• Use ignoreFields aggressively for volatile fields (timestamp, gasPrice, nonce on pending). False mismatches from clock skew will drown out real divergence signals.
• Watch erpc_shadow_response_mismatch_total segmented by finality label: mismatches on finality="unfinalized" are often expected (different block propagation timing); mismatches on finality="finalized" are signal for a real provider discrepancy.
• Use larger="true" on the mismatch metric to detect shadow upstreams that return extra undocumented fields — a common source of application-level bugs if you switch providers.
• Combine allowMethods on the upstream config with ignoreFields to narrow shadow scope per use-case (e.g., validate eth_getLogs only, ignore timestamp on block methods).
• Set sampleRate: 0 (not enabled: false) when you want to pause shadows temporarily — this preserves the config and makes re-enabling a one-field change.
• Shadow upstreams are not in the health-tracker scoring pool. Their error rate does not affect routing decisions; monitor them separately via erpc_shadow_response_error_total.

Edge cases & gotchas

1. Shadow fires on cached responses. The trigger checks resp != nil; it does not distinguish cache hits from upstream hits. Cache-served finalized responses usually match; unfinalized ones may not — monitor mismatch_total{finality="unfinalized"} separately.
2. sampleRate: 0 suppresses all shadow traffic. The check is rand.Float64() >= sampleRate; since rand.Float64() is always >= 0.0, zero suppresses every request.
3. sampleRate: nil and sampleRate: 1.0 are equivalent. Both fire every request.
4. Both hashes are recomputed when ignoreFields is set. Masking is applied to both sides before comparison — a field masked on one side is masked on both.
5. Hash error increments erpc_shadow_response_error_total{error="hash_error"}. No mismatch is recorded when hash computation fails.
6. ErrUpstreamShadowing is an internal guard, not a client-visible error. It fires inside shouldSkip only when a shadow upstream is reached through the normal forward path (misconfiguration). It carries a details field set to the shadow upstream's upstreamId. It has no ErrorStatusCode() method — there is no HTTP status override. The caller wraps it in ErrUpstreamRequestSkipped, which eRPC retries on a different upstream rather than surfacing to the client. The health tracker exempts ErrCodeUpstreamShadowing from ErrorsTotal so it does not degrade upstream scoring. [common/errors.go:L1416-1430] [health/tracker.go:L970-983]
7. The larger mismatch label identifies shadow returning more data than primary. shadowSize > originalSize sets larger="true". Useful for detecting shadow upstreams that return extra fields in receipts or traces.
8. Shadow upstreams with no healthy primary are still dispatched. If p.doForward returns a response object (even an error response), the shadow trigger fires. Only resp == nil suppresses it.
9. Panic isolation is complete. A panic inside any shadow goroutine is caught by recover(), logged, and counted; the primary goroutine stack is unaffected.
10. Method allow/ignore lists on the shadow upstream are fully enforced. ShouldHandleMethod is called before each shadow request; rejected methods are silently skipped. [erpc/shadow.go:L54-61]

Observability

Label notes:

• vendor — ups.VendorName(), e.g. "alchemy", "quicknode".
• category — JSON-RPC method name, e.g. "eth_getBlockByNumber".
• finality — shadowResp.Finality().String() on mismatch: "finalized", "unfinalized", or "n/a".
• emptyish — "true" when shadow returned null/""/[]/{}.
• larger — "true" when shadow byte size exceeded primary byte size.
• error — common.ErrorFingerprint(err): a stable short error-type string; prevents label cardinality explosion.

Log messages emitted:

• Error: "shadow response hash mismatch" — with full request, originalResponse, shadowResponse, expectedHash, shadowHash, ignoredFields.
• Debug: "shadow request returned error", "shadow request returned nil response", "shadow request skipped due to sampling", "method not allowed for shadow upstream".
• Trace: "shadow response identical to primary response".

Source code entry points

• erpc/shadow.go (opens in a new tab) — executeShadowRequests: request cloning, per-upstream loop, sample rate check, method check, forward, hash comparison, metric emit.
• erpc/projects.go:L135-L163 (opens in a new tab) — PreparedProject.Forward: trigger point after doForward, response cloning, goroutine launch.
• upstream/registry.go:L574-L603 (opens in a new tab) — shadow vs. normal upstream registration split; GetNetworkShadowUpstreams accessor at L295–L298.
• erpc/networks.go:L302-L303 (opens in a new tab) — Network.ShadowUpstreams() accessor delegating to the registry.
• upstream/upstream.go:L410-L450 (opens in a new tab) — Forward method; byPassMethodExclusion parameter that allows shadow goroutines to skip the shouldSkip block.
• common/config.go:L982-L986 (opens in a new tab) — ShadowUpstreamConfig struct definition.
• common/errors.go:L1416-L1430 (opens in a new tab) — ErrUpstreamShadowing / ErrCodeUpstreamShadowing definition.
• upstream/upstream.go:L1505-L1508 (opens in a new tab) — shouldSkip first check: returns NewErrUpstreamShadowing for shadow-enabled upstreams on the normal path.
• health/tracker.go:L970-L983 (opens in a new tab) — exemption: ErrCodeUpstreamShadowing does not increment ErrorsTotal.
• telemetry/metrics.go:L640-L656 (opens in a new tab) — MetricShadowResponseIdenticalTotal, MetricShadowResponseMismatchTotal, MetricShadowResponseErrorTotal definitions.
• health/tracker_test.go:L765 (opens in a new tab) — test asserting ErrUpstreamShadowing keeps ErrorsTotal=0 and ErrorRate=0.

Related pages

• Upstreams — upstream configuration, allowMethods/ignoreMethods, and health tracking.
• Selection policies — how upstreams are scored and chosen; shadow upstreams are excluded from this entirely.
• Rate limiters — cap outbound load to the shadow upstream if it's a metered provider.
• Survive provider outages — complementary resilience pattern: shadow before promoting, then failover after.