Consensus

How it works

Integration point. Consensus sits at networkExecutor.Run. When a ConsensusPolicyConfig is present and the SkipConsensus directive is false, the executor calls consensus.Run(ctx, req, slotInner) where slotInner is retry(hedge(tryOneUpstream)). Without consensus the path is retry(hedge(runUpstreamSweep)). Composition order: consensus(retry(hedge(tryOneUpstream))). (erpc/network_executor.go:L183-188)

Participant selection. Before spawning goroutines, if requiredParticipants is configured, reorderForParticipantQuota front-loads tag-matching upstreams so the first maxParticipants drawn satisfy every {tag, minParticipants} entry. Tag matching uses exact equality first, then glob wildcards (*, ?). This is best-effort: quota shortfalls are handled by lowParticipantsBehavior. The executor then spawns exactly maxParticipants goroutines, each calling slotInner independently. (consensus/quota.go:L27-79)

Collection and analysis loop. The runAnalyzer goroutine reads responses from a buffered channel. After each response it calls newConsensusAnalysis (classify, hash, deduplicate by upstream, group) and determineWinner (apply 24 priority-ordered rules). Responses are deduplicated by upstream ID before grouping — hedge/retry can reselect an upstream that already answered for another slot, and one node must not vote twice toward agreementThreshold; the best-ranked response per upstream keeps the vote (valid result > consensus-valid error > infrastructure error). If shouldShortCircuit returns true, the outcome is sent immediately and remaining requests are cancelled (or left running in fire-and-forget mode). After all participants respond — or a wait-cap deadline fires — the final outcome is sent to the caller's select.

Response classification. Each response is classified into one of four types: ResponseTypeNonEmpty (successful non-null result), ResponseTypeEmpty (successful but null/empty result), ResponseTypeConsensusError (JSON-RPC execution exception, client exception, unsupported, or missing-data), ResponseTypeInfrastructureError (all other errors including ErrUpstreamsExhausted). Infrastructure errors are excluded from validParticipants. ErrUpstreamsExhausted is always infra even when it wraps execution exceptions via the shared ErrorsByUpstream map. (consensus/analysis.go:L333-344) Specific error-code mappings: ErrCodeEndpointExecutionException (EVM revert, code 3) → ConsensusError; ErrCodeEndpointClientSideException, ErrCodeEndpointUnsupported, ErrCodeEndpointMissingData → ConsensusError; all other errors (timeout, network, server 500) → InfrastructureError; nil JSON-RPC response on a successful response object → InfrastructureError with hash "error:generic". A successful response where IsResultEmptyish returns true → ResponseTypeEmpty.

Hashing. Non-error responses are hashed via JsonRpcResponse.CanonicalHash() or CanonicalHashWithIgnoredFields(fields) when ignoreFields is configured for the method. Errors are hashed via errorToConsensusHash: JSON-RPC exceptions hash as "jsonrpc:<normalized_code>"; standard errors hash as their code; unknown errors hash as "error:generic". Within each hash group, LargestResult tracks the member with the highest CachedResponseSize; the winner returns group.LargestResult, not the first response. (consensus/analysis.go:L102-117)

Rule evaluation. determineWinner walks consensusRules in priority order (consensus/rules.go:L24). Full 24-rule ordering:

Short-circuit rules. shouldShortCircuit evaluates 3 rules after each response. (1) sendrawtx_first_success: fires on any single non-empty for eth_sendRawTransaction, never blocked. (2) consensus_error_threshold: best group is consensus-error ≥ threshold; blocked when preferHighestValueFor configured, or acceptMostCommon active AND (preferNonEmpty OR preferLargerResponses). (3) unassailable_lead: best non-empty ≥ threshold with unassailable lead; blocked when preferLargerResponses active, preferHighestValueFor configured, or preferNonEmpty active and leader is empty.

Misbehavior tracking and punishment. After all responses, trackAndPunishMisbehavingUpstreams compares each group against the consensus group. Data disagreements count as misbehavior; error disagreements are tracked separately via erpc_consensus_upstream_errors_total and are NOT misbehavior. For each misbehaving upstream: MetricConsensusMisbehaviorDetected fires and tracker.RecordUpstreamMisbehavior is called. Punishment requires strict majority (consensusGroup.Count > validParticipants / 2) plus token-bucket denial (disputeThreshold tokens per disputeWindow); when both hold, the upstream is cordoned and AfterFunc calls Uncordon after sitOutPenalty. (consensus/executor.go:L784-1229)

eth_sendRawTransaction broadcast mode. Rule 1 fires as soon as any single non-empty response exists — threshold is ignored entirely. Combined with fireAndForget: true, eRPC returns the first accepted tx hash immediately while remaining participant goroutines continue broadcasting using context.WithoutCancel(ctx) as their base, surviving HTTP client disconnection. They are bounded by upstream call completion, wait-cap timers, or process exit — not by graceful shutdown signals.

Wait caps. maxWaitOnResult (adaptive p50, default min=5ms, max=1s) arms when the first non-empty response arrives. maxWaitOnEmpty (adaptive p90, default min=50ms, max=2s) arms on the very first response of any kind. When a cap fires, the analyzer resolves with what it has collected, applying disputeBehavior or lowParticipantsBehavior as appropriate. erpc_consensus_wait_capped_total{trigger} tracks firings.

SkipConsensus directive. Any request can bypass consensus via X-ERPC-Skip-Consensus: true header, ?skip-consensus=true query param, or directiveDefaults.skipConsensus: true in network/project config. When set, the executor falls through to retry(hedge(runUpstreamSweep)). Only "true" activates bypass; any other value keeps consensus active.

Misbehavior export. When misbehaviorsDestination is configured, every round with at least one misbehaving upstream is serialized as a JSONL record containing timestamp, project/network/method/finality, full policy snapshot, request, winner snapshot, analysis snapshot (groups, counts, hashes), and per-participant snapshot. For type: file: records are written immediately to path/<resolved-filename>. For type: s3: records are buffered and flushed asynchronously when maxRecords, maxSize, or flushInterval fires; S3 uses PutObject — each flush replaces the object. JSONL record top-level fields: ts (int64, Unix ms), projectId (string), networkId (string), method (string), finality (string), policy (object: maxParticipants, agreementThreshold, disputeBehavior, lowParticipantsBehavior, preferNonEmpty, preferLargerResponses, ignoreFields), request ({jsonrpc, id, method, params}), winner ({responseType, hash, size, upstreamId}), analysis ({totalParticipants, validParticipants, bestByCount, groups[]} — each group has {hash, count, isTie, responseType, responseSize}), participants (array — per upstream: {upstreamId, vendor, responseType, responseHash, responseSize, response, error}).

Caller-abandoned handling. If the HTTP client disconnects before consensus completes, a drain goroutine waits for analyzerDone, drains outcomeCh, and releases the winner's response buffer to prevent memory leaks. Metrics record caller_abandoned outcome. (consensus/executor.go:L287-338)

Panic safety. Both executeParticipant and runAnalyzer have recover() defers. Panics send errPanicInConsensus on the relevant channel so the caller's select never deadlocks. MetricConsensusPanics is incremented on any recovery.

Config schema

All fields under networks[].failsafe[].consensus. (or project-level failsafe[].consensus.). Duration fields accept Go duration strings. *bool fields: unset means SetDefaults applies the documented default. Struct at common/config.go:L1591-1658.

Worked examples

All patterns below are distilled from real production fleets; comments explain the non-obvious choices.

1. Standard 3-of-2 agreement for unfinalized log and receipt reads. The most common production shape: fan out to 3 upstreams, require 2 to agree, accept the majority on disputes. Used on Base, Ethereum mainnet, Blast, and Sonic for eth_getLogs and eth_getBlockReceipts on recent (unfinalized) blocks where reorg-detection matters. preferLargerResponses: false is deliberate — keeping it false preserves the unassailable_lead short-circuit (returns as soon as 2 agree, without waiting for the 3rd), cutting worst-case latency significantly:

failsafe:  - matchMethod: "eth_getLogs|eth_getBlockReceipts"    matchFinality: "unfinalized"    consensus:      maxParticipants: 3      agreementThreshold: 2      disputeBehavior: acceptMostCommonValidResult      lowParticipantsBehavior: acceptMostCommonValidResult      # false keeps the unassailable_lead short-circuit active:      # once 2 of 3 agree, eRPC returns immediately without      # waiting for the third upstream to finish      preferLargerResponses: false      preferNonEmpty: true      ignoreFields:        eth_getLogs:          - "*.blockTimestamp"        eth_getTransactionReceipt:          - "blockTimestamp"          - "logs.*.blockTimestamp"          - "l1Fee"          - "l1GasUsed"        eth_getBlockReceipts:          - "*.blockTimestamp"          - "*.l1Fee"          - "*.l1GasUsed"

2. Highest-value consensus for fee and nonce reads. The standard production pattern for preventing stale-low fee estimates and stuck nonces: fan out to 4 upstreams and return whichever reports the highest value. The noBehaviors pattern (omitting disputeBehavior/lowParticipantsBehavior) is intentional — with preferHighestValueFor active, dispute/low-participant rules never fire because the numeric-value rule evaluates first. Used in production for every EVM chain that accepts writes:

failsafe:  - matchMethod: "eth_gasPrice|eth_maxPriorityFeePerGas"    consensus:      maxParticipants: 4      agreementThreshold: 1      # ignoreFields: null — hash-based grouping is irrelevant when      # preferHighestValueFor regrouping by numeric value is active      preferHighestValueFor:        eth_gasPrice:          - result        eth_maxPriorityFeePerGas:          - result  - matchMethod: "eth_getTransactionCount"    hedge:      quantile: 0.95      maxCount: 1      minDelay: 500ms      maxDelay: 10s    retry:      maxAttempts: 6      delay: 0    consensus:      maxParticipants: 4      agreementThreshold: 1      preferHighestValueFor:        eth_getTransactionCount:          - result

3. eth_sendRawTransaction broadcast with fire-and-forget. Return the first accepted tx hash immediately while the remaining upstream goroutines continue broadcasting in background using context.WithoutCancel. Rule 1 fires as soon as any single non-empty response exists — the threshold is bypassed entirely. In production, maxParticipants: 10 is paired with hedge.maxCount: 2 so the hedge races 3 upstreams simultaneously, with remaining slots filling in asynchronously. Account for these goroutines in graceful shutdown timeout budgets — process shutdown signals do not reach them:

failsafe:  - matchMethod: "eth_sendRawTransaction"    hedge:      quantile: 0.95      maxCount: 2      minDelay: 50ms      maxDelay: 1s    retry:      maxAttempts: 6      delay: 0    consensus:      maxParticipants: 10      # threshold 1: Rule 1 fires on first non-empty anyway,      # but setting 1 makes the intent explicit      agreementThreshold: 1      # goroutines continue after first hash is returned to client      fireAndForget: true

4. State-trie integrity checks for high-value reads. For Ethereum mainnet balance and call reads, production uses a strict 2-of-2 check: both participants must agree or the result is disputed. A tighter quorum (2/2 vs 2/3) makes it easier to detect a single rogue node, and disputeLogLevel: "error" escalates disagreements for alerting. ignoreFields: null is intentional — balance and call results have no metadata variance, so the default timestamp exclusions would only mask real bugs:

failsafe:  - matchMethod: "eth_getBalance|eth_call"    matchFinality: "unfinalized"    timeout:      duration: 60s    hedge:      quantile: 0.95      maxCount: 1      minDelay: 500ms      maxDelay: 10s    retry:      maxAttempts: 6      delay: 0    consensus:      maxParticipants: 2      agreementThreshold: 2      # error-level so a PagerDuty alert fires on any balance/call mismatch      disputeLogLevel: "error"      # null: no ignoreFields — balance and call results have no timestamp      # variance; omitting fields would only mask genuine disagreements

5. Misbehavior tracking with S3 export and automatic cordon. Used in production for Base eth_getLogs to detect and penalize upstreams that consistently return bad data. After 3 disputes within 5 minutes the offender is cordoned for 10 minutes. {dateByHour} in filePattern groups disputes by hour — each hourly flush overwrites the S3 object for that key, so use {timestampMs} if per-flush objects are preferred. The 100 MiB maxSize guard prevents unbounded memory use during high-dispute periods:

failsafe:  - matchMethod: "eth_getLogs|eth_getBlockReceipts"    matchFinality: "unfinalized"    consensus:      maxParticipants: 3      agreementThreshold: 2      disputeBehavior: acceptMostCommonValidResult      punishMisbehavior:        # 3 disputes within 5 minutes triggers a cordon        disputeThreshold: 3        disputeWindow: 5m        # cordon for 10 minutes then auto-uncordon        sitOutPenalty: 10m      misbehaviorsDestination:        type: s3        path: "s3://${S3_BUCKET}/misbehaviors/mainnet/"        filePattern: "{dateByHour}-{method}"        s3:          region: "${AWS_REGION}"          # flush at 100 records, 100 MiB, or every hour          maxRecords: 100          maxSize: 104857600          flushInterval: 1h

Request/response behavior

• Consensus wraps the composed retry(hedge(tryOneUpstream)) inner function. Retry and hedge still fire per participant slot; the consensus layer sees only the final result from each slot.
• A dispute produces ErrConsensusDispute: JSON-RPC wire code -32603, HTTP method-level status 409 Conflict (wire status 200 for POST JSON-RPC). The Cause is errors.Join(per-participant errors). Retryability: if ANY child error is retryable, the whole dispute error is retryable toward the network.
• A low-participants outcome produces ErrConsensusLowParticipants: JSON-RPC wire code -32603, HTTP method-level status 412 Precondition Failed (wire status 200). Same multi-error retryability rule.
• The SkipConsensus directive (X-ERPC-Skip-Consensus: true or ?skip-consensus=true) causes the executor to skip the consensus branch entirely. Retry, hedge, circuit breaker, and timeout still apply. Only the string "true" activates bypass.
• Within a consensus group, group.LargestResult is returned as the winner — not the first response received. This ensures the most complete representation is selected when multiple upstreams agree on the same canonical hash.
• When preferHighestValueFor is configured for a method, responses are regrouped by extracted numeric value (not hash), allowing differently-formatted representations of the same value to match.

Best practices

• Start with maxParticipants: 3 and agreementThreshold: 2. Adding more participants increases coverage but raises cost and worst-case latency.
• Disable preferLargerResponses in latency-sensitive scenarios — it disables all short-circuit and forces every request to wait for all participants.
• When extending ignoreFields, always re-include all three default entries (eth_getLogs, eth_getTransactionReceipt, eth_getBlockReceipts). Setting any entry replaces the entire map; omitting defaults causes spurious disputes on blockTimestamp fields.
• Set disputeLogLevel: "warn" (it is the default, but set it explicitly). Zerolog's zero value is TraceLevel; an unset field gets coerced to WarnLevel by the builder, which may mask trace-level intent.
• Use fireAndForget: true only for eth_sendRawTransaction broadcast. For read methods it wastes upstream capacity after the winner is known. Account for these goroutines in graceful shutdown timeout budgets — process shutdown signals do not reach them.
• For punishMisbehavior, set disputeWindow conservatively (e.g., 5m) and disputeThreshold to at least 3. A too-tight window with threshold 1 will cordon flapping nodes on the first transient disagreement.
• Monitor erpc_consensus_wait_capped_total{trigger} — frequent "result" firings mean your maxWaitOnResult is too short for the slowest participants; frequent "empty" firings indicate slow cold paths.

Edge cases & gotchas

1. ignoreFields is full set replacement. Adding one method removes all three built-in entries. Spurious disputes on blockTimestamp fields result. Always re-include all default entries when extending the map. (consensus/analysis.go:L442-453)
2. preferLargerResponses disables all short-circuit. Even a clear majority above threshold does not short-circuit because a larger response may still arrive. Disable in latency-sensitive scenarios.
3. eth_sendRawTransaction bypasses threshold entirely. Rule 1 fires on any single non-empty response regardless of how many others errored. Intentional for tx broadcasting.
4. preferNonEmpty + returnError escalates empty threshold winner to dispute. When empty meets threshold, non-empty minority exists, and preferNonEmpty: true under disputeBehavior: returnError, the result is a dispute — not the empty winner and not the minority non-empty. (Rule 11)
5. Tie among non-empty at/above threshold without preference → dispute. Rule 8 fires before the generic threshold-winner rule 19. Enable preferLargerResponses to resolve ties by size.
6. requiredParticipants shortfall is silent. No warning, no metric, no error. Operators cannot distinguish tag-quota shortfall from general upstream unavailability in current observability output.
7. fireAndForget goroutines survive graceful shutdown. context.WithoutCancel strips shutdown signals. Budget shutdown timeouts to account for in-flight fire-and-forget participants.
8. S3 uses PutObject, not append. With {dateByHour} in filePattern, flushes within the same hour overwrite the previous S3 object. Use {timestampMs} if each flush should create a new object.
9. S3 accessKeyID casing is load-bearing. YAML key must be accessKeyID (capital D). Lowercase accessKeyId silently falls back to the SDK credential chain.
10. File export requires absolute path. Relative paths are rejected at startup. (consensus/export.go:L37-41)
11. disputeLogLevel zero value coerces to warn. Set "trace" explicitly if trace-level dispute logging is required.
12. ErrConsensusDispute is retryable when any child error is retryable. A dispute caused partly by a retryable upstream error propagates retryability to the caller.
13. Misbehavior punishment only when majority consensus. shouldPunishUpstream requires consensusGroup.Count > validParticipants / 2. With 2 of 5 agreeing (2 > 2 fails), punishment is blocked. (consensus/executor.go:L1196)
14. ErrUpstreamsExhausted wrapping foreign consensus errors. Without a guard, HasErrorCode traversal would misclassify exhausted errors as ResponseTypeConsensusError, creating phantom voting groups. The explicit check at consensus/analysis.go:L376-386 prevents this.
15. Context cancellation after all executions does NOT produce LowParticipants. When the parent context is cancelled after every participant has completed but before outcome delivery, the non-blocking try-receive picks up the computed outcome rather than treating it as abandonment. (consensus/executor.go:L271-279)
16. Cold start for wait caps. AdaptiveDuration.Resolve falls back to min when no latency data exists. If you configure maxWaitOnResult: {quantile: 0.5, max: 1s} without a min, cold start returns 0 — no cap until data accumulates.
17. S3 startup validation. newS3MisbehaviorExporter calls HeadBucket synchronously. If the bucket is unreachable, S3 export is disabled and an error is logged — consensus continues normally.
18. omitting minParticipants in requiredParticipants is a startup error. Go leaves unset int at 0, which immediately fails the > 0 validation. The server will not start.
19. IsTie flag on response groups is per-type, per-count. responseGroup.IsTie is true when at least two groups of the same ResponseType share the same Count. Infrastructure error groups always have IsTie = false. This flag is used by the lowParticipants + acceptMostCommon rule to escalate non-empty ties to dispute instead of returning an arbitrary winner. (consensus/analysis.go:L121-138)
20. Race-free analysis struct. newConsensusAnalysis pre-populates all cached accessor fields before returning. After the analyzer sends the outcome to the caller, both goroutines may read the analysis concurrently (caller for metrics; analyzer for misbehavior tracking). Lazy-init under concurrent reads would be a data race. (consensus/analysis.go:L141-153)
21. preferHighestValueFor can be combined per-method with hash-based consensus. The map allows different handling per method — eth_getTransactionCount: ["result"] uses highest-value while eth_call falls through to normal hash-based consensus on the same failsafe entry. The rule only matches when at least one valid group has extractable numeric values; if extraction fails for all responses it falls through to subsequent rules.
22. ErrConsensusDispute and ErrConsensusLowParticipants error contracts. ErrConsensusDispute has HTTP method-level status 409 Conflict and JSON-RPC wire code -32603; ErrConsensusLowParticipants has HTTP method-level status 412 Precondition Failed and the same wire code. Wire HTTP status for POST JSON-RPC is 200 in both cases (errors are translated to JSON-RPC at the transport layer). Both errors carry errors.Join(per-participant errors) as their Cause; retryability propagates from children — if ANY child error is retryable, the whole dispute or low-participants error is retryable toward the network. ErrConsensusLowParticipants.DeepestMessage() includes per-participant info.

Observability

OTel trace spans:

Notable log messages: "consensus rule matched" (debug, per rule match), "consensus misbehavior detected - upstreams differ from consensus" (at disputeLogLevel, when misbehavingCount > 0; includes full participant breakdown with numbered keys upstream1, responseType1, agreesWithConsensus1, etc.), "fire-and-forget mode: remaining requests complete in background" (debug, on short-circuit when fireAndForget: true), "misbehaviour limit exhausted, punishing upstream" (warn), "upstream already in sitout, skipping" (debug, when punishment attempted but upstream already cordoned), "consensus caller abandoned; analysis continues in background" (warn), "panic in consensus analyzer" / "Panic in consensus participant" (error), "failed to append misbehavior record" (warn, when exporter errors), "failed to initialize S3 misbehavior exporter; export disabled" (error, S3 setup failure at init time — consensus continues normally), "uploaded misbehavior records to S3" (info, after each successful S3 PutObject), "failed to upload misbehavior records to S3" (error, on S3 PutObject failure).

Source code entry points

• consensus/consensus.go (opens in a new tab) — Consensus struct, NewConsensus, entry-point Run
• consensus/executor.go (opens in a new tab) — executeConsensus (fan-out, wait caps, context racing), runAnalyzer (collection loop, short-circuit, misbehavior tracking), executeParticipant, trackAndPunishMisbehavingUpstreams
• consensus/analysis.go (opens in a new tab) — newConsensusAnalysis (classify, hash, group), classifyAndHashResponse, errorToConsensusHash, LargestResult tracking
• consensus/rules.go (opens in a new tab) — consensusRules (24 rules) and shortCircuitRules (3 rules); all decision logic
• consensus/quota.go (opens in a new tab) — reorderForParticipantQuota, upstreamMatchesTag (glob matching)
• consensus/export.go (opens in a new tab) — misbehaviorExporter interface, fileMisbehaviorExporter, JSONL record structs
• consensus/export_s3.go (opens in a new tab) — s3MisbehaviorExporter, buffered per-key S3 uploads with background flush worker
• consensus/export_utils.go (opens in a new tab) — resolveFilePattern/resolveFilePatternWithDefaults, getInstanceID (cached singleton using INSTANCE_ID → POD_NAME → HOSTNAME → SHA256 hash of now+pid), sanitizeForFilename, parseS3Path
• consensus/utils.go (opens in a new tab) — extractFieldValues, parseNumericValue, valuesToKey, compareValueChains for preferHighestValueFor logic
• erpc/network_executor.go#L183-L188 (opens in a new tab) — Integration point: routing to consensus.Run vs. retry(hedge(runUpstreamSweep))
• common/config.go#L1591-L1658 (opens in a new tab) — ConsensusPolicyConfig, ConsensusRequiredParticipant, MisbehaviorsDestinationConfig, S3FlushConfig, PunishMisbehaviorConfig
• common/defaults.go#L2389-L2487 (opens in a new tab) — ConsensusPolicyConfig.SetDefaults and MisbehaviorsDestinationConfig.SetDefaults
• consensus/executor_test.go (opens in a new tab) — behavior-locking tests incl. context-cancel-after-execution and misbehavior tracking

Related pages

• Retry — wraps each consensus participant slot; each retry attempt is a fresh hedge race inside consensus.
• Hedge — also wraps each participant slot; hedges individual upstream attempts within a slot.
• Timeout — bounds the whole consensus race from outside.
• Selection & scoring — decides which upstreams are drawn for participant slots.
• Rate limiters — caps per-upstream cost from the extra fan-out requests.
• Survive provider outages — the reliability outcome consensus reinforces.