Integrity checks

How it works

eRPC's integrity is two separate planes — keep them distinct when configuring:

• Block-tip & availability enforcement lives on RequestDirectives (populated from directiveDefaults + X-ERPC-Enforce-* headers) and runs in method-specific hooks. It is on by default.
• Data-integrity validation is the integrity: config block (project + network). It compiles to a set of checks the EVM post-forward hook runs through a single engine. It is opt-in — with no config, the engine runs nothing.

Both surfaces produce the same failure mode: ErrEndpointContentValidation — retryable at network scope (try another upstream), not retryable at the same upstream, feeding the validation exhaustion bucket. On the wire, HTTP 200 with a JSON-RPC error body (code: -32603). common/errors.go:L2719-2747

Layer 1 — Block-tip & availability enforcement

Configured under networks[].directiveDefaults. On by default; toggle per-request via X-ERPC-Enforce-* headers / ?enforce-*= query params.

Highest-block enforcement — eth_blockNumber. Reads dirs.EnforceHighestBlock. If the response block is below the network-known highest, eRPC replaces the response with a synthetic JSON-RPC result containing the highest hex block number — no re-request is made. Applies to EVERY response source, including cache hits, so a stale value planted in a shared cache can never be served below the tip this instance knows. The tip is resolved request-aware: a use-upstream selector scopes it to the targeted subset. architecture/evm/eth_blockNumber.go:L31-123

Highest-block enforcement — eth_getBlockByNumber[latest/finalized]. Reads dirs.EnforceHighestBlock. If the returned block is behind the network-known highest, eRPC re-requests that block against a different upstream (SkipCacheRead=true, UseUpstream=!<id>); pickHighestBlock keeps the higher of the two — protecting against a corrupted state pointer. Cache responses skip the re-fetch (the read-side realtime age guard handles staleness instead). architecture/evm/eth_getBlockByNumber.go:L111-283

Stale-tip cache write guard. Under enforceHighestBlock, a realtime-finality response already behind the network-wide tip is never written to cache (the classic symptom it prevents: eth_blockNumber sawtoothing backwards on x-erpc-cache: HIT while one upstream lags). Fails open when pollers don't yet know a tip. architecture/evm/json_rpc_cache.go:L1075-1098

Block-range availability. Before forwarding eth_getLogs/trace_filter/arbtrace_filter to an upstream, CheckBlockRangeAvailability verifies fromBlock/toBlock are within the upstream's range; otherwise ErrUpstreamBlockUnavailable (retryable) routes to a different upstream. architecture/evm/block_range.go

Future-block empty-result guard. emptyResultBeyondConfidence returns an empty result truthfully (rather than retrying) when the block is beyond the confidence head (latest or finalized, per emptyResultConfidence) — preventing retry storms on not-yet-produced blocks. Fail-open on unknown head or block tags. architecture/evm/common.go:L55-89

Layer 1 config schema — networks[].directiveDefaults

networks[].evm.emptyResultConfidence (AvailabilityConfidence, default "blockHead"): "blockHead" retries empty results for blocks at/below the latest tip; "finalizedBlock" retries only at/below the finalized tip — use for archive workloads that legitimately return empty for unfinalized blocks. common/defaults.go:L2078

The deprecated networks[].evm.integrity (EvmIntegrityConfig: enforceHighestBlock/enforceGetLogsBlockRange/enforceNonNullTaggedBlocks) is auto-migrated into directiveDefaults during SetDefaults (explicit old values win; no warning). common/defaults.go:L1952-1964

Layer 2 — Data-integrity validation (the integrity: module)

Opt-in and off by default. With no integrity: block, the engine runs zero checks — safe to deploy this version and enable nothing. The block lives at the project level (applies to all networks) and the network level (overrides); the network block is merged over the project block at startup.

The front door: level

One knob picks a preset over the check catalog. Each level is a superset of the previous:

integrity:  level: authoritative              # off | intrinsic | corroborated | authoritative
  # Per-finality verdict for the reorg-sensitive checks only. At the tip you can't  # tell a node bug from a reorg, so unfinalized mismatches are recorded, not rejected.  invalidBehavior:    finalized: reject               # reject | soft-flag | off    unfinalized: soft-flag
  # Override individual checks by id (enable / disable / params / onFailure).  checks:    receiptVsBlock: { enabled: false }          # drop one authoritative check    bloomMatch:     { onFailure: soft-flag }     # record instead of reject    txHashUniqueness: { params: { strict: "true" } }  # the one check with a param
  # Cost cap on the canonical force-fetches the authoritative tier issues.  budget:    maxPerSecond: 50    maxConcurrent: 8

The check catalog

Every check is an independently-toggleable unit, grouped by family. A check only runs for the methods it applies to (e.g. receipt checks on eth_getBlockReceipts/eth_getTransactionReceipt, block checks on eth_getBlockByNumber/eth_getBlockByHash). All cryptographic recomputes are chain-safe: the known-field set is derived from the reference encoder (go-ethereum), so a chain with custom header/receipt fields, a system/deposit tx, or a hashes-only response is skipped, never false-flagged.

Methods column: block = eth_getBlockByNumber/eth_getBlockByHash; receipts = eth_getBlockReceipts; receipt = eth_getTransactionReceipt; logs = eth_getLogs; txByHash = eth_getTransactionByHash.

※ = reorg-sensitive (governed by invalidBehavior). All others are deterministic and always reject on violation. Note the division of labour on eth_getBlockByHash: blockByHashIdentity enforces that you got the block you asked for, while the continuity pair (which judges canonicality, a question a by-hash lookup never asked) deliberately does not run there — see Edge cases #7. Both stateful tiers share one reorg-aware per-network ChainView — a bounded number→hash pin plus a content-addressed header cache (window reorgWindow, default 32). corroborated populates it passively from blocks your own traffic already pulled (it never fetches a block sequence); authoritative additionally force-fetches a missing anchor once through the normal network path (cache-backed, recursion-guarded, deduped by the ChainView), capped by budget. A changed hash for a number is treated as a reorg: the new fork is adopted and its stale descendants are rolled back, so the pin always reflects one consistent fork — block and receipts can't disagree.

Per-finality verdict: invalidBehavior

Most checks are deterministic — a violation is corruption regardless of finality, so they always reject. A handful are reorg-sensitive (parentHashLinkage, hashStability, receiptVsBlock): they compare against another observation, and near the tip a disagreement may be a benign reorg rather than a bug. invalidBehavior decides what to do for those, by the block's finality (read from the upstream state poller):

• finalized → default reject (a finalized block can't reorg; a mismatch is corruption).
• unfinalized → default soft-flag (record a metric/log, still serve — it may be a reorg). off skips the check (and any force-fetch) on the hot tip entirely.

Rolling out safely: observeOnly

observeOnly: true runs every enabled check and reports everything, but no verdict may touch the response — a violation that would have been rejected is served anyway and recorded with the outcome would_reject. This is the way to turn integrity on for a network the first time: it surfaces bad upstream data and the module's own gaps on that chain at zero request risk, and

sum(rate(erpc_integrity_check_total{outcome="would_reject"}[5m])) by (network, check, upstream)

is precisely the client-facing cost enforcement would incur — read it before promoting.

Interpreting what you see, by how the would_rejects are spread:

observeOnly is absolute and deliberately outranks everything else: a per-check onFailure: reject, invalidBehavior, and any check a future release adds all cannot reject while it is set. That last property is the point — invalidBehavior: soft-flag cannot give you this, because the ~25 deterministic checks ignore invalidBehavior by design and always reject.

It is inherited like the rest of the block, which gives the natural rollout shape — a project-wide safety net, promoted one chain at a time:

projects:
  - id: main
    integrity:
      level: authoritative
      observeOnly: true          # every network observes by default
    networks:
      - architecture: evm
        evm: { chainId: 8453 }
        integrity:
          observeOnly: false     # base graduates to enforcement

off still means off: observeOnly never resurrects a check you disabled, nor its force-fetches.

Why continuity is by-number only

Continuity answers "what is the chain at height N", so it runs on eth_getBlockByNumber and not on eth_getBlockByHash. A by-hash lookup asks for one named block: its canonicality was never the question — retrieving orphaned-but-real blocks by hash is exactly how indexers unwind a reorg. Rejecting there would discard data the caller explicitly asked for, and because an orphan hash does not exist on the canonical fork, no failover can produce an alternative: the request simply fails.

What a by-hash lookup does require is identity — that you got the block you named — and that is enforced by its own check, blockByHashIdentity (intrinsic, deterministic). Do not rely on blockHashRecompute for this: it proves the returned header hashes to the hash it claims, i.e. that the block is real and self-consistent, but a node answering with an entirely different valid block satisfies it. Identity is what closes that gap.

Symmetrically, a by-hash response never moves the number→hash pin (it only populates the content-addressed header cache). If it did, one client fetching an orphan would adopt that orphan as canonical at its height and roll back the real fork's descendants, turning a private reorg-unwind into mass rejections of everyone else's by-number traffic.

All deterministic checks (hash recompute, roots, field shapes, identity) still apply to by-hash responses.

Per-request selection: profiles & headers

An operator can define named profiles and let callers pick one per request, gated by headerMode:

• off (default) — the X-ERPC-Integrity header / ?integrity= query param is ignored.
• profiles — a request may only select a named profile by word.
• full — a request may also set a bare level word (intrinsic/authoritative/…).

integrity:  level: intrinsic  headerMode: profiles  profiles:    strict:        { level: authoritative }    lenient:       { level: off }

X-ERPC-Integrity: strict        # selects the "strict" profile (headerMode: profiles|full)

Backward compatibility

The deprecated per-check directiveDefaults validation flags (validateLogsBloomMatch, enforceLogIndexStrictIncrements, validateTransactionsRoot, validateHeaderFieldLengths, validateTransactionFields, validateTransactionBlockInfo, validateTxHashUniqueness, validateTransactionIndex, validateLogFields, validateLogsBloomEmptiness) are translated into integrity.checks at config-load time — an explicit integrity: block wins per check. This is the only place the old flags are read; there is no legacy path at runtime. common/defaults.go:L2031-2075

The removed per-request X-ERPC-Validate-* headers, the validateReceiptTransactionMatch / receiptsCount* / validationExpectedBlock* directives, the GroundTruth* library fields, and the non-functional upstreams[].integrity block no longer exist — use the integrity: config (and X-ERPC-Integrity for per-request selection) instead.

Layer 2 config schema — integrity

Worked examples — real-world scenarios

1. Free structural hardening for an indexer (the common starting point). Catch malformed / cryptographically-inconsistent data with zero extra upstream calls — bad responses fail over to another upstream before they reach your database:

integrity:  level: intrinsic

2. One default for the whole project, stronger on a high-value network. A project-wide block applies to every network; a network block overrides it field-by-field:

integrity:  level: intrinsic           # project-wide defaultnetworks:  - evm: { chainId: 1 }    integrity:      level: authoritative   # mainnet gets force-fetch corroboration

3. Strongest guarantees, cost-bounded. authoritative force-fetches the canonical block to corroborate single receipts and recompute the receipts root. Cap the fetch rate, and skip corroboration on the volatile tip (where a mismatch is usually a reorg, not corruption):

integrity:  level: authoritative  invalidBehavior:    finalized: reject    unfinalized: off       # don't force-fetch / corroborate the hot tip  budget:    maxPerSecond: 50    maxConcurrent: 8

4. Cross-block / cache correctness. corroborated adds parent-hash linkage and hash-stability over blocks you've already seen — catching a node serving a block that doesn't link to the chain you were served before. Still no extra upstream calls:

integrity:  level: corroborated

5. Safe rollout — observe before you enforce. Set everything to soft-flag first: violations are recorded (a WARN log + the validation metrics bucket) but the response is still served, so you can measure your upstreams' data quality without failing traffic. Once it's quiet, flip finalized back to reject:

integrity:  level: intrinsic  invalidBehavior:    finalized: soft-flag     # record, don't reject — flip to 'reject' after rollout    unfinalized: soft-flag

invalidBehavior only governs the reorg-sensitive checks (the ※ rows). To soft-flag a deterministic check during rollout, give it onFailure: soft-flag — e.g. checks: { bloomMatch: { onFailure: soft-flag } }.

6. Non-standard chain — leave it off. ZK-rollups and chains with custom encodings: the recompute checks already skip what they can't model, but the simplest answer is to not enable the module for that network (it's off by default — this is only needed if you turned it on project-wide):

networks:  - evm: { chainId: 324 }     # zkSync Era    integrity:      level: off

7. Per-tenant strictness via profiles + header. Define named presets and let callers pick one per request (without letting them set arbitrary levels):

integrity:  level: intrinsic           # the default for callers who send no header  headerMode: profiles       # off | profiles | full  profiles:    strict:  { level: authoritative }    relaxed: { level: off }

X-ERPC-Integrity: strict       # or  ?integrity=strict

8. Add or drop a single check around a level. Levels are presets — fine-tune with checks:

integrity:  level: intrinsic  checks:    receiptVsBlock: { enabled: true }    # pull one authoritative check up to intrinsic    bloomMatch:     { enabled: false }   # …and drop one you don't want

Enabling an authoritative check (like receiptVsBlock) at a lower level turns on its force-fetch — set a budget too.

9. Migrating from the old validate* flags. If your config still sets the deprecated directiveDefaults.validateLogsBloomMatch: true (etc.), they're translated into integrity.checks automatically at startup — nothing breaks. To make it explicit (recommended), move them into an integrity block:

# old (still works, auto-translated at startup):#   directiveDefaults: { validateLogsBloomMatch: true, enforceLogIndexStrictIncrements: true }# new (explicit):integrity:  checks:    bloomMatch:         { enabled: true }    logIndexContiguity: { enabled: true }

Request/response behavior

• Violations produce ErrEndpointContentValidation — HTTP 200 with JSON-RPC error body {"code": -32603}. Retryable at network scope (try another upstream), not at the same upstream. common/errors.go:L2719-2747
• Because content-validation errors are excluded from consensus's preferLargerResponses logic, a corrupt-but-larger response can no longer dispute an honest agreeing majority. consensus/rules.go
• A soft-flag verdict serves the response and emits a recorded-mismatch log; only reject fails the response over to another upstream.

Best practices

• Start at level: intrinsic. It's free (no extra upstream calls) and catches malformed/self-inconsistent data plus the cryptographic recompute checks. Most teams never need more.
• Roll out with soft-flag, then flip to reject. Run invalidBehavior: { finalized: soft-flag, unfinalized: soft-flag } (and onFailure: soft-flag on any deterministic checks you're unsure about) first, watch the metrics for a few days, then enforce. This avoids failing real traffic on day one if an upstream has a quirk you didn't anticipate.
• Use authoritative only where data quality warrants it, and always set a budget — it force-fetches the canonical block (one fetch per cold block, cache-amortized). Pair it with invalidBehavior.unfinalized: off to skip the force-fetch on the hot tip.
• Leave invalidBehavior.unfinalized at soft-flag (or off) — rejecting unfinalized mismatches would reject benign reorgs. Only finalized data is safe to hard-reject on a cross-observation mismatch.
• It needs ≥2 upstreams to be useful. A reject is a failover signal (ErrEndpointContentValidation is retryable toward the network) — with a single upstream it just surfaces an error. Pair integrity with a retry policy.
• It's defense-in-depth with consensus. Consensus catches a minority bad upstream; integrity catches the case consensus can't see — when every serving upstream returns the same wrong value (shared client/indexer bug). High-value paths use both.
• Skip the module on non-standard chains (ZK-rollups with custom encodings). The recompute checks are chain-safe (they skip what they can't model), but if a chain's normal data looks "custom", just leave integrity off for it and keep Layer-1 enforcement.
• Profiles, not free-form, for untrusted callers — headerMode: profiles lets callers pick a named preset without setting arbitrary levels; headerMode: off (default) ignores the header entirely.
• Watch the rollout, then the steady state. A persistent stream of recorded-mismatch WARN logs / validation-bucket metrics for one upstream means that provider's data quality is poor — down-weight or remove it.

Edge cases & gotchas

1. Opt-in means opt-in. No integrity: block ⇒ zero data-integrity checks (Layer 1 enforcement still runs). Deploying this version changes nothing until you add the block.
2. Recompute checks skip rather than false-flag — almost always. A header/receipt with a field the reference encoder doesn't know, a system/deposit tx, or a hashes-only block response is skipped, never rejected. The one case the skip-guard can't catch is #6.
3. Continuity never fetches. parentHashLinkage/hashStability only compare against blocks already observed (bounded per-network number→hash store); an unseen parent skips the check.
4. authoritative force-fetches are recursion-guarded (marked internal, skipped by the engine) and budgeted; on budget exhaustion the corroboration check no-ops.
5. Project⊕network precedence. A project-wide integrity: applies to all networks; a network block overrides field-by-field (profiles unioned).
6. Some chains need recompute checks disabled — proven: HyperEVM (chainId 999). A chain whose eth_getBlock keeps system transactions in the header's transactionsRoot but omits them from the returned transactions list will systematically fail transactionsRootRecompute (and, by the same mechanism, receiptsRootRecompute) — even though every returned tx is valid. The skip-guard in #2 can't catch it: each individual tx hash verifies, only the Merkle root differs. This is a chain representation quirk, not corruption. Disable the affected check(s) on that network — the level preset stays, the one check is turned off:

   networks:
     - evm:
         chainId: 999          # HyperEVM
       integrity:
         level: authoritative
         checks:
           transactionsRootRecompute: { enabled: false }   # confirmed in production
           receiptsRootRecompute:     { enabled: false }   # same mechanism — disable if you run authoritative receipt corroboration
   Diagnostic rule of thumb: if erpc_integrity_violation_total{check="…Recompute"} rejects across every upstream of one chain, it's the check, not the data — disable that check for that chain (and consider opening an issue so it can be added to this list).
7. Continuity does not run on eth_getBlockByHash (see Why continuity is by-number only) — and a by-hash response never moves the pin. Earlier builds checked by-hash lookups too, which produced a large stream of rejects (mostly ending as integrity_failed client errors) whenever one upstream kept serving settled orphaned blocks to clients unwinding reorgs: the pin correctly said "not canonical", but since the request named the orphan hash, no upstream could satisfy the pinned fork.

Source code entry points

• architecture/evm/integrity/ (opens in a new tab) — the check engine, catalog, levels, decode, resolver, history.
• architecture/evm/hooks.go (opens in a new tab) — HandleUpstreamPostForward runs the engine (opt-in via resolveIntegrity).
• architecture/evm/integrity_config.go (opens in a new tab) — compiles integrity config → check set + reorg policy.
• common/config_integrity.go (opens in a new tab) — the IntegrityConfig schema + merge.
• common/defaults.go (opens in a new tab) — migrateLegacyIntegrityChecks: deprecated flag → integrity.checks translation.
• architecture/evm/eth_blockNumber.go (opens in a new tab) / eth_getBlockByNumber.go (opens in a new tab) / block_range.go (opens in a new tab) — Layer-1 block-tip & availability enforcement.

Observability

Data-integrity violations are counted by erpc_integrity_violation_total (labeled by the individual check id and verdict) and also surface as ErrEndpointContentValidation feeding the validation bucket of ErrUpstreamsExhausted; a soft-flag verdict additionally emits a recorded-mismatch WARN log rather than failing the response. The bundled Grafana dashboard (monitoring/grafana/dashboards/erpc.json (opens in a new tab)) ships a collapsed Data Integrity row that breaks these violations down by check, verdict, network, upstream, vendor and method, with a drill-down table.

Tracing. When tracing is enabled, each validation emits an Integrity.Validate span — its duration is the integrity overhead and the aux force-fetches nest under it. Attributes (simple mode): integrity.method, integrity.upstream, integrity.outcome (pass/reject/soft_flag), integrity.checks, integrity.rejected_check. In detailed tracing mode the span additionally records, verbatim and without redaction, the actual-vs-expected values of every violation (as integrity.reject / integrity.soft_flag events), each check's outcome, and — for caught requests — the response bodies (capped at 128 KB). That makes a caught request checkable by hand: the rejected attempt's Integrity.Validate span carries the original body (integrity.response) and the request-level Project.Forward span carries the corrected served body (integrity.served_response), so you can confirm the catch was real and the correction right. Bodies are recorded only on a violation (original) or a saved request (corrected) and only under the detailed-tracing gate, so it's zero-cost when tracing is off. Sample a few requests in detailed mode to audit a specific upstream.

Related pages

• Retry — integrity violations are retryable; retry routes them to a fresh upstream.
• Consensus — content-validation errors are excluded from agreement/preferLargerResponses.
• Timeout — bounds the re-fetch triggered by highest-block enforcement and authoritative force-fetch.
• Selection policies — upstream scoring uses stale-block metrics.
• Survive provider outages — the broader scenario this feature serves.