Selection & scoring

How it works

The entry point is policy.Engine, which owns a map of *Slot values keyed by (network, method, finality). Each slot has its own ticker goroutine; the request path reads slot.cache via atomic.Pointer.Load() — completely wait-free. On every tick the engine takes a metric snapshot from the health tracker, builds an EvalContext, acquires a pooled Sobek JS runtime, calls the compiled eval function, materializes the ordered result, and atomically swaps the cache. The whole cycle completes in under a millisecond for typical policy chains.

When Engine.RegisterNetwork is called at network bootstrap it:

1. Upgrades the placeholder evalFunc to default_policy.js (embedded at compile time).
2. Creates the wildcard ("*","*") slot and runs a synchronous initial tick — the first request always sees a populated cache.
3. Starts the slot's ticker goroutine.

The default evalScope: "network" creates one slot per network. Setting evalScope: "network-method" lazy-creates additional slots on first request for each distinct method, enabling per-method primary divergence. evalScope: "network-finality" splits on finality bucket; "network-method-finality" is the most granular. Narrow slots fall back to the wildcard slot cache during cold-start until their first tick completes.

Score inputs (health tracker). Every metric available to evalFunc comes from the health tracker's rolling window. The tracker maintains 10 atomic sub-buckets per (upstream, method, finality) tuple; one bucket rotates out every windowSize / 10, so data drips out with no cliff. A DDSketch per bucket (1% relative accuracy) produces p50/p70/p90/p95/p99 latency quantiles.

errorRate counts only genuine upstream-quality failures: connection refused, 5xx, genuine timeouts. EVM reverts, rate-limit 429s, client disconnects, and hedge cancellations are deliberately excluded so a misbehaving caller or aggressive hedger cannot inflate a healthy upstream's error rate.

throttledRate counts remote 429 responses specifically — vendor quota exhaustion. misbehaviorRate tracks upstreams returning semantically wrong data. Latency quantiles are populated only from successful responses — a fast-failing upstream that rejects connections in 1ms has empty quantile sketches and reports p70 = 0 (unknown quality, not ultra-fast). A well-written policy should guard latencyAbove with samplesAbove(N).

blockHeadLag and finalizationLag are atomic point-in-time state, not rolling counters. They persist across window rotations and stay non-zero until the upstream catches up to the network tip. blockHeadLagSeconds and finalizationLagSeconds multiply the block count by the tracker's EMA-estimated block time (alpha 0.1, minimum 3 samples before publishing). Until the EMA is warm, both seconds fields are 0 — pair blockSecondsLagAbove with blockNumberLagAbove for cold-start coverage.

The default policy chain. Omitting evalFunc applies the production-hardened default embedded at compile time:

(upstreams, ctx) =>
  upstreams
    .removeCordoned()
    .excludeIf(all(samplesAbove(10), errorRateAbove(0.7)))
    .excludeIf(all(samplesAbove(10), throttleRateAbove(0.4)))
    .excludeIf(any(
        all(samplesAbove(20), latencyAbove(3000), latencyDeviationAbove(3, { mode: 'majority' })),
        latencyAbove(10_000)
    ))
    .excludeIf(any(blockNumberLagAbove(16), blockSecondsLagAbove(30)))
    .whenEmpty(() => upstreams)
    .preferTag('!tier:fallback', { minHealthy: 1, fallback: 'tier:fallback' })
    .sortByScore(PREFER_FASTEST)
    .stickyPrimary({ hysteresis: 0.30, minSwitchInterval: '30s' })
    .probeExcluded({ sampleRate: 0.1, minSamples: 10, minSamplesWindow: '60s', maxConcurrent: 4, timeout: '10s' })

Each excludeIf step records per-leaf attribution into erpc_selection_exclusion_total{reason}. The whenEmpty safety net prevents fail-closed behavior during a widespread outage. stickyPrimary prevents flapping: the challenger must score more than 30% above the incumbent AND the last switch must be at least 30 seconds ago. probeExcluded shadow-mirrors 10% of real traffic to excluded upstreams so re-admission happens the moment their metrics recover.

Probe eligibility (which excluded upstreams get shadow traffic). Not every exclusion benefits from probing, so exclude-family steps record a per-upstream probe verdict: excludeIf(...) defaults to probe-eligible (the predicate reads traffic metrics that freeze without traffic — probes are how the upstream proves recovery), while excludeTag(...) and removeCordoned() default to probe-blocking (static/admin exclusions don't reverse via traffic data, so probing them only burns quota). Each accepts an override: excludeTag('tier:reserve', { probe: true }), excludeIf(blockNumberLagAbove(16), { probe: false }) — the latter is recommended for lag predicates, whose inputs the state poller refreshes regardless of traffic. Verdicts combine order-insensitively across the whole chain: every exclude step judges every upstream it would drop (including upstreams already dropped by earlier steps), and an excluded upstream is probed iff at least one probe-eligible step excludes it and no probe-blocking step does — so an upstream that is both erroring and tag-excluded is not probed, regardless of step order. Upstreams excluded only by untracked means (raw filter, take, …) default to probing. routing.probe: 'off' remains the per-upstream hard veto on top. The resolved verdict is visible per upstream in the decision output (ProbeEligible).

Scoring formula and presets. sortByScore ranks surviving upstreams by:

score(u) = overall / (1 + errorRate×w₁ + respLatency×w₂ + throttledRate×w₃ + blockHeadLag×w₄ + finalizationLag×w₅ + misbehaviors×w₆)

A clean upstream with no penalties scores 1.0; every unit of penalty divides the score. Three built-in weight presets:

Per-upstream routing.scoreMultipliers on the upstream config merge with the chosen preset. The default merge mode ("merge") lets individual multiplier keys override the base weight; overall multiplies the final score and is the simplest way to strongly prefer or demote a specific upstream without touching weight shapes.

Circuit-breaker interplay. The selection policy and the circuit breaker are complementary layers that act at different points and timescales. The policy runs per eval tick and excludes upstreams from the ranked list via excludeIf predicates over rolling-window metrics. The circuit breaker is a per-upstream state machine inside the upstream executor: it is checked only after the sweep has already dispatched to that upstream (selection runs first). When the breaker is open, TryAcquirePermit() fails the attempt immediately with ErrFailsafeCircuitBreakerOpen — no transport call is made — and the sweep advances to the next upstream in the policy's ordered list. So an upstream can sit at position 0 in the policy list with an open breaker: each request burns one cheap fast-fail attempt there before moving on, until the policy's own metrics demote it on a later tick. Neither mechanism reads the other's state (breaker-open does not cordon or exclude); they only interact through outcomes. If the split bothers you, align breaker thresholds with the policy's errorRateAbove so both layers agree on what "broken" means. Source: upstream/upstream_executor.go:L345-L366 (opens in a new tab).

shadowExcludeIf — dry-run exclusion rules. shadowExcludeIf(predicate) is the non-destructive counterpart of excludeIf. The predicate runs every tick; would-have-been-excluded upstreams stay in the ordered list but each trip increments erpc_selection_shadow_exclusion_total{upstream, reason} with the same leaf-slug attribution. Use it to measure a new rule's blast radius before promoting it to excludeIf.

includeIf — conditional re-admission (pool-level quantifiers). includeIf(target, condition) is the dual of excludeIf: it ADDS upstreams from the full universe back into the chain when an aggregate condition over the surviving pool holds — it never removes anyone. target comes first ("include these if condition"): a tag pattern ('tier:reserve', ! negation supported) or a selector object {id, tag, vendor, type, position} whose facets AND together; an object with no concrete facet ({}, {position:'head'}) is a deliberate no-op so it can never accidentally admit the entire universe. Admitted upstreams land at the tail by default (position: 'head' available in object form), are never duplicated by id, and are ranked by the subsequent sortByScore like everyone else. condition is a boolean or (pool, ctx) => boolean receiving the current chain — which is what enables pool-level ("network-level") questions via native array methods over the per-upstream predicate factories: p.every(blockSecondsLagAbove(30)), p.length < 2. Mind that native every is true on an empty array — guard with p.length > 0 && when the empty pool should be handled by a separate explicit p.length < 1 line. Malformed arguments and throwing conditions degrade to a no-op rather than sinking the eval. Source: internal/policy/stdlib/stdlib.js:L1023-L1116 (opens in a new tab); tests: internal/policy/stdlib/include_if_test.go (opens in a new tab).

Additional stdlib chain methods. Beyond excludeIf / includeIf, the JS stdlib installs the following chainable Array.prototype methods (source: internal/policy/stdlib/stdlib.js (opens in a new tab)):

• Identity/label filters: byId(id), excludeId(id), byTag(pat), excludeTag(pat), byVendor(name), excludeVendor(name), byType(t), where(fn), whereNot(fn).
• Health filters (higher-level wrappers): removeCordoned(), removeByErrorRate(threshold), removeByThrottling(threshold), removeByMisbehavior(threshold), removeByLag(blocks), removeByLatency(ms), keepHealthy(), removeByMinRequests(n).
• Tier selection: preferTag(pat, opts?), preferVendor(name, opts?).
• Diversity: spreadAcrossTags(prefix) — interleaves the sorted list so adjacent positions come from different tag-partitions (e.g. "region:").
• Safety net: whenEmpty(fn) — falls back to fn() if the array is empty.
• Conditional/finality branching: when(mask, fn), byFinality(handlers), if(cond, fn), unless(cond, fn), fallbackTo(fn), ensureMin(n, fn).

hasMatchingTag semantics (used by byTag, excludeTag, preferTag, includeIf string targets — internal/policy/stdlib/stdlib.js:L267-293 (opens in a new tab)):

• Positive pattern ("tier:main", "region:us-*"): matches if ANY tag on the upstream matches.
• Negated pattern ("!tier:fallback"): matches if NO tag matches the un-negated pattern.
• Array of patterns: positives are OR'd; negations are AND'd. Upstream matches iff (no positives OR at least one positive matches) AND (all negations hold).

preferTag step semantics (internal/policy/stdlib/stdlib.js:L854-865 (opens in a new tab)):

1. Filter to upstreams matching the primary pattern.
2. If ≥ minHealthy match → return that subset.
3. Else if fallback pattern is set → filter to the fallback pattern.
4. Else → return the input unchanged.

Upstream JS object shape (built by buildJSUpstreams at internal/policy/eval.go:L312 (opens in a new tab)):

UpstreamMetrics object shape (u.metrics and u.metricsAcrossMethods — internal/policy/eval.go:L283 (opens in a new tab)):

The block-time EMA used to compute blockHeadLagSeconds/finalizationLagSeconds has sanity bounds: values outside [10ms, 120s] are rejected and the internal EMA is reset to the last published value to prevent post-halt spikes.

EvalContext (ctx) shape (internal/policy/eval.go:L17 (opens in a new tab)):

Global JS helpers available inside every evalFunc:

• methodMatches(pat) — tests ctx.method against a pattern
• isFinalityRequest() — ctx.finality === 'finalized'
• Preset globals: PREFER_FASTEST, PREFER_FRESHEST, PREFER_LEAST_ERRORS
• Finality bit-flag constants: REALTIME (1), UNFINALIZED (2), FINALIZED (4), UNKNOWN (8) — used with .when(mask, fn) inside eval; NOT for scoreMultipliers.finality config fields

Exclusion attribution leaf slugs. excludeIf records which leaf predicate actually tripped per upstream. Bounded-cardinality slugs emitted by erpc_selection_exclusion_total{reason}:

error_rate_above, error_rate_below, throttle_rate_above, throttle_rate_below, misbehavior_rate_above, latency_p<Q>_above, latency_p<Q>_deviation_above, block_head_lag_above, finalization_lag_above, block_head_lag_seconds_above, finalization_lag_seconds_above, samples_below, samples_above (guard — not a failure slug), not_<child_slug>, custom.

Note: samplesAbove/samplesBelow predicates are marked isGuard=true so they do not appear as primary exclusion reasons when they do NOT trip; they only appear when they are the leaf that fires.

Log messages emitted by the selection engine:

• WARN "selection policy eval failed; retaining previous cache" — tick error; fields: network, method, tick_id, error.
• DEBUG "policy step" — one line per stdlib step when step-logging is enabled; fields: network, method, tick_id, idx, step, in, out, dropped, added, reordered, args. Disabled by default in production (zero overhead: no allocation when disabled).
• DEBUG "policy excluded upstream" — one per excluded upstream when step-logging is enabled; fields: network, method, tick_id, upstream, step, reason, leaf_reasons.

Legacy upstream tag keys. Upstream-level group: <value> and cohort: <value> YAML keys are rewritten as tier:<value> and cohort:<value> tags at config-load time by mergeLegacyLabelKeysIntoTags. They are not valid in new configs.

Config schema

selectionPolicy (networks[].selectionPolicy)

Full YAML path: projects[].networks[].selectionPolicy Config struct: SelectionPolicyConfig at common/config.go:L2352-2430 Defaults: common/defaults.go:L2512-2609

EvalScope slot cardinality:

Legacy fields (accepted, never survive to runtime):

upstream.routing (upstreams[].routing)

ScoreMultiplierConfig fields

Merge modes (set via sortByScore({ multipliers: 'merge' | 'override' | 'off' })):

• "merge" (default): per-upstream keys override matching base-weight keys; overall lifts final score multiplicatively.
• "override": configured upstreams rank by their weights only; unconfigured fall back to base.
• "off": ignore u.scoreMultipliers entirely.

probeExcluded options (JS)

Defaults from internal/policy/eval.go:L793-799 (opens in a new tab) and internal/policy/stdlib/stdlib.js:L999-1009 (opens in a new tab).

stickyPrimary options (JS)

Defaults from internal/policy/stdlib/stdlib.js:L725-727 (opens in a new tab).

latencyDeviationAbove options (JS)

Defaults from internal/policy/stdlib/stdlib.js:L1343-1358 (opens in a new tab).

Worked examples

All patterns below are distilled from real production fleets; comments explain the non-obvious choices.

1. Production default: the full default policy chain with scoreMetricsWindowSize. In production every project sets scoreMetricsWindowSize explicitly — this controls how fast exclusion decisions react to upstream changes. Shorter windows react faster but are noisier; 4m is the production-proven sweet spot. The default evalFunc is omitted here (eRPC loads default_policy.js automatically), so only the project-level window and upstreamDefaults.routing matter:

projects:  - id: main    # 4-minute rolling window for health tracker metrics — fast enough to react    # to a flapping upstream without being too jittery on cold-start pods.    scoreMetricsWindowSize: 4m    scoreMetricsMode: compact    networkDefaults:      # selectionPolicy omitted → eRPC loads the embedded default_policy.js      failsafe:        - matchMethod: "*"          hedge:            quantile: 0.95            maxCount: 1            minDelay: 500ms            maxDelay: 10s    upstreamDefaults:      routing:        scoreMultipliers:          # Realtime/unfinalized: heavily weight latency AND block-head lag —          # a lagging upstream on tip-of-chain traffic is worse than a slow one.          - finality: [realtime, unfinalized]            respLatency: 10            errorRate: 2            misbehaviors: 5            blockHeadLag: 15            finalizationLag: 2          # Finalized/unknown: latency matters more than lag for archive queries.          - finality: [unknown, finalized]            respLatency: 15            errorRate: 2            misbehaviors: 10            blockHeadLag: 10            finalizationLag: 2

2. Vendor demotion with overall multiplier (real provider pattern). Some vendors are less reliable or more expensive — instead of hard-excluding them, apply a fractional overall multiplier so they serve as implicit fallbacks. Production uses 0.2 for Alchemy and 0.3 for QuickNode across all networks:

upstreams:  - id: my-alchemy-node    endpoint: https://eth-mainnet.g.alchemy.com/v2/${ALCHEMY_API_KEY}    evm: { chainId: 1 }    routing:      scoreMultipliers:        # overall: 0.2 pushes this upstream to position 3–4 behind free-tier        # nodes. It still serves traffic when primaries are excluded; a hard        # exclude would waste the paid capacity entirely.        - overall: 0.2  - id: my-quicknode-node    endpoint: https://example.quiknode.pro/${QUICKNODE_API_KEY}/    evm: { chainId: 1 }    routing:      scoreMultipliers:        # 0.3 keeps QuickNode as a mid-tier — ahead of Alchemy (0.2) but        # behind free-tier primaries (no multiplier = score 1.0 baseline).        - overall: 0.3

3. Latency-sensitive API with preferred upstream and tier:fallback. A latency-sensitive use case (MEV, trading) uses one high-quality primary with a overall: 1.5 multiplier to keep it at position 0, and a paid archival node tagged tier:fallback that only activates when no primary survives:

selectionPolicy:  evalScope: network  evalFunc: |    (upstreams, ctx) =>      upstreams        .removeCordoned()        .excludeIf(all(samplesAbove(10), errorRateAbove(0.5)))        // Drop upstreams >8 blocks behind tip — stale state is dangerous for        // trading. Pair with blockNumberLagAbove for cold-start coverage since        // blockSecondsLagAbove is 0 until the block-time EMA warms (>=3 pairs).        .excludeIf(any(blockNumberLagAbove(8), blockSecondsLagAbove(15)))        .whenEmpty(() => upstreams)        .preferTag('!tier:fallback', { minHealthy: 1, fallback: 'tier:fallback' })        .sortByScore(PREFER_FASTEST)        // Lower hysteresis (0.20) than default: switch primaries faster when        // the challenger is 20%+ better — acceptable for low-latency routing.        .stickyPrimary({ hysteresis: 0.20, minSwitchInterval: '15s' })        .probeExcluded({ sampleRate: 0.1, minSamples: 10, timeout: '5s' })

4. Freshness-critical workload (indexer / subgraph node). A subgraph node or NFT indexer queries finalized blocks and needs the freshest data. PREFER_FRESHEST ranks by block-head lag and finalization lag instead of raw latency. Tighter lag guards (blockNumberLagAbove(3)) drop a node the moment it falls behind, and probeExcluded ensures re-admission is immediate when it catches up:

selectionPolicy:  evalScope: network  evalFunc: |    (upstreams, ctx) =>      upstreams        .removeCordoned()        .excludeIf(all(samplesAbove(10), errorRateAbove(0.7)))        // Finalization lag guard: drop upstreams that haven't finalized recent        // blocks — crucial for indexers reading finalized state only.        .excludeIf(any(blockNumberLagAbove(3), finalizationLagAbove(2)))        .whenEmpty(() => upstreams)        // PREFER_FRESHEST: weights blockHeadLag×15 and finalizationLag×8 vs        // PREFER_FASTEST's respLatency×15. Use this when stale data is worse        // than slow data.        .sortByScore(PREFER_FRESHEST)        .stickyPrimary({ hysteresis: 0.30, minSwitchInterval: '30s' })        .probeExcluded({ sampleRate: 0.1, minSamples: 10, timeout: '10s' })

5. Auditing a new exclusion rule without risk (shadowExcludeIf). Before tightening a throttle-rate guard on production, run it in shadow mode: the predicate fires and increments a metric, but no upstream is actually dropped. Watch erpc_selection_shadow_exclusion_total{reason="throttle_rate_above"} for a few hours, then promote to excludeIf when the fire rate looks acceptable:

(upstreams, ctx) =>
  upstreams
    .removeCordoned()
    .excludeIf(all(samplesAbove(10), errorRateAbove(0.7)))
    // Shadow dry-run: fires the metric without excluding — safe to deploy first.
    // Check erpc_selection_shadow_exclusion_total{reason="throttle_rate_above"}.
    .shadowExcludeIf(all(samplesAbove(10), throttleRateAbove(0.2)))
    .whenEmpty(() => upstreams)
    .sortByScore(PREFER_FASTEST)
    .stickyPrimary({ hysteresis: 0.30, minSwitchInterval: '30s' })
    .probeExcluded({ sampleRate: 0.1, minSamples: 10, timeout: '10s' })

6. Reserve tier — break-glass inclusion (includeIf). Keep a pricier or rate-limited provider (tagged tags: ["tier:reserve"]) out of normal rotation, and bring it in only when the serving pool is collectively unfit — every survivor lagging or slow, or none left. Because includeIf adds rather than evicts, a single degraded primary never pulls the reserve in by itself; once admitted, sortByScore ranks reserve and survivors together. Keep both halves of the pattern: excludeTag declares the tier is out of normal rotation, includeIf declares the only conditions under which it returns — reading the pair top-to-bottom documents why the reserve exists. Pair with routing.probe: off on the reserve upstreams to also keep shadow-probe traffic off them while benched:

(upstreams, ctx) =>
  upstreams
    .removeCordoned()
    .excludeIf(all(samplesAbove(10), errorRateAbove(0.7)))
    // Reserve stays out of normal rotation...
    .excludeTag('tier:reserve')
    // ...and returns ONLY when survivors are collectively lagging or slow.
    // The `length > 0 &&` guard matters: native `every` is true on an empty
    // pool, so the empty case is handled by its own explicit line below.
    .includeIf('tier:reserve', (p) => p.length > 0 && p.every(blockSecondsLagAbove(30)))
    .includeIf('tier:reserve', (p) => p.length > 0 && p.every(latencyAbove(2000, 99)))
    .includeIf('tier:reserve', (p) => p.length < 1)
    .sortByScore(PREFER_FASTEST)
    .stickyPrimary({ hysteresis: 0.30, minSwitchInterval: '30s' })

Request/response behavior

• The ordered upstream list is set on req via req.SetUpstreams(upsList); the failsafe layer (hedge/retry) walks through it sequentially, never selecting the same upstream twice per execution. req.ConsumedUpstreams and req.ErrorsByUpstream sync.Maps track what has been tried. erpc/networks.go:L1023-L1074 (opens in a new tab)
• If policyEngine == nil or GetOrdered returns an empty list (engine not yet ticked), the network falls back to raw upstream registry order. erpc/networks.go:L1032-1038 (opens in a new tab)
• RegisterNetwork fires a synchronous initial tick before returning, so in practice the first request always sees a populated cache. internal/policy/engine.go:RegisterNetwork:L305 (opens in a new tab)
• On any tick error (timeout, throw, invalid_return), the previous cache is retained unchanged — routing continues on the last-good order. internal/policy/slot.go:L250-L263 (opens in a new tab)
• An OpenTelemetry span "PolicyEngine.GetOrdered" wraps the GetOrdered call in the request path. Attributes: upstreams.count, upstreams.sorted (detailed tracing only).

Best practices

• Pair samplesAbove(N) with every rate predicate. errorRateAbove(0.7) on an upstream with 2 samples (1 failure → 50% rate) will exclude it spuriously. all(samplesAbove(10), errorRateAbove(0.7)) waits for a statistically meaningful window. The default policy uses samplesAbove(10) for error/throttle and samplesAbove(20) before the latency-deviation check.
• Use blockNumberLagAbove(N) as the primary lag guard, not blockSecondsLagAbove. The seconds variant is 0 until the block-time EMA warms (requires ≥3 block pairs). On cold start, blockNumberLagAbove(16) fires correctly; blockSecondsLagAbove(30) silently no-ops.
• Keep evalScope: "network" unless you genuinely need per-method ranking. Each additional scope dimension multiplies ticker goroutines and JS eval cycles. If you need method-specific scoreMultipliers, set evalScope: "network-method" — without it, method-specific entries silently no-op.
• Don't set routing.probe: "off" on upstreams you care about recovering. With probing off, an excluded upstream accumulates no shadow samples and only recovers when structural signals (lag clearing, etc.) naturally drive re-admission. Probing is what makes re-admission fast.
• Let whenEmpty(() => upstreams) stay in every policy. Without it, a simultaneous spike in error rates across all upstreams returns ErrUpstreamsExhausted rather than degraded-but-working traffic. Fail-open is almost always better than fail-closed for RPC routing.
• Use overall multipliers rather than custom weight shapes for simple up/down-ranking. A scoreMultipliers: [{overall: 0.1}] on a backup upstream cleanly demotes it without changing the scoring formula for everyone else.
• stickyPrimary hysteresis of 0.30 is conservative; lower it for faster recovery. At 0.30, a challenger must score 30% better than the incumbent. For setups with very similar upstreams, 0.15 may be more responsive without causing flapping.

Edge cases & gotchas

1. scoreMultipliers.method silently no-ops in evalScope: "network". WildcardMatch("eth_getLogs", "*") returns false. Entries with a specific Method field are only applied in per-method or per-method-finality slots. internal/policy/eval.go:resolveScoreMultipliers:L442 (opens in a new tab)

2. blockHeadLagSeconds / finalizationLagSeconds are 0 until block-time EMA warms. The EMA requires ≥3 (blockNumber, blockTimestamp) pairs with differing timestamps. Predicates like blockSecondsLagAbove(30) silently no-op until then. Use blockNumberLagAbove(16) as the primary guard. internal/policy/eval.go:L89-96 (opens in a new tab)

3. whenEmpty(() => upstreams) prevents fail-closed. If every excludeIf rule fires simultaneously (project-wide outage), the safety net restores the full unfiltered set. The network sends traffic to degraded upstreams rather than returning ErrUpstreamsExhausted. internal/policy/default_policy.js:L46 (opens in a new tab)

4. No time-based re-admission. An excluded upstream stays excluded until its tracker metrics actually improve. probeExcluded accelerates re-admission by feeding fresh samples; arbitrary time passage does not. erpc/networks_selection_policy_test.go:TestNetworkPolicy_NoTimeBasedReadmit_OnlyMetricsHeal (opens in a new tab)

5. routing.probe: "off" permanently excludes from probing. The upstream accumulates no shadow samples and stays excluded until structural signals (e.g. lag clearing) naturally drive re-admission, or an operator manually uncordons. common/config.go:L784-788 (opens in a new tab)

6. stickyPrimary with method-disagreement escape hatch. If the previous primary is excluded in THIS method's slot but not others, the slot returns the current sort order WITHOUT updating the shared sticky register. Other slots that still have the upstream in their survivor set keep it as primary. internal/policy/stdlib/stdlib.js:L795-802 (opens in a new tab)

7. scoreMultipliers.finality YAML integer vs JS bit-flag confusion. The config field uses the Go DataFinalityState enum ("finalized"=0, "unfinalized"=1, "realtime"=2, "unknown"=3). The JS globals FINALIZED=4, REALTIME=1, UNFINALIZED=2, UNKNOWN=8 are bitmasks for .when(mask, fn) inside evalFunc bodies — they are not valid in the YAML config field. Setting finality: [4] in YAML fails to match any request. common/data.go:L10-L27 (opens in a new tab), internal/policy/stdlib/install.go:L74-77 (opens in a new tab)

8. latencyDeviationAbove is constructed fresh per tick. The closure captures topByMethod at factory time. Do NOT put latencyDeviationAbove in a variable and reuse across ticks — it would use stale peer data. The default policy is safe because the chain is a stateless expression evaluated fresh each tick. internal/policy/stdlib/stdlib.js:L1228-1324 (opens in a new tab)

9. Bug #907 regression. With per-finality tracking enabled, block-head-lag writes must fan out to both the specific-finality bucket AND the {*, All} rollup. Pre-fix, blockNumberLagAbove read the starved rollup and silently no-oped. Fixed and covered by TestNetworkPolicy_RealPoll_LaggingUpstreamExcluded. erpc/networks_selection_policy_realpoll_test.go (opens in a new tab)

10. Alphabetical tiebreak in sortByScore. When two upstreams have identical scores (e.g. both clean on cold start), the lexicographically earlier ID wins. Test assertions are order-sensitive until metrics diverge. internal/policy/stdlib/stdlib.js:L656 (opens in a new tab)

11. upstreamDefaults.routing is all-or-nothing. If upstream.routing is non-nil (even empty), the default routing block is NOT applied. erpc/eval_multipliers_test.go:TestApplyDefaults_InheritsRouting (opens in a new tab)

12. p70 = 0 means unknown quality, not ultra-fast. Fast-failing upstreams that reject connections in 1ms have empty quantile sketches. A predicate like latencyAbove(500) sees 0 < 500 = false and does NOT exclude such an upstream. Pair latency predicates with samplesAbove(N) or treat p70 == 0 as a warning signal. internal/policy/eval.go:L89-96 (opens in a new tab)

13. probeExcluded is a no-op array transform. It does NOT modify the upstream list. Its only effect is depositing __probeConfig for the Go-side Prober. Probe traffic never blocks the request path; channel overflow increments erpc_selection_probe_dropped_total and drops the probe silently. The probe feed channel size is 256 (probeFeedBufferSize = 256 in internal/policy/prober.go:L102 (opens in a new tab)). internal/policy/stdlib/stdlib.js:L999-1010 (opens in a new tab)

14. Step-log is disabled by default in production. Per-step chain trail logging (DEBUG "policy step") is only active when Engine.SetStepLogEnabled(true) is called — by the simulator or when log level is DEBUG. In production there is zero overhead beyond one function-call indirection per stdlib step; no log entries and no per-step allocations are created. internal/policy/engine.go:L679-695 (opens in a new tab)

15. ErrUpstreamExcludedByPolicy error code exists but has no production call site. Defined at common/errors.go:L1823-L1836 (opens in a new tab); error code string: "ErrUpstreamExcludedByPolicy". The health tracker silently ignores it (does NOT penalize the upstream's error rate). The policy engine achieves exclusion structurally — upstreams absent from the ordered list are never attempted by NextUpstream — rather than by returning this error. The error type is forward-compatible scaffolding.

16. Idle slot eviction sweep runs every 1 minute. The engine's sweepIdleSlots fires every engineSweepInterval = 1 minute. A narrow slot with idleEvictionAfter = 1 h may persist for up to 1 h + 1 min after its last access. The health tracker's own idle eviction threshold is separate and shorter: DefaultIdleEvictionAfter = 30 minutes for per-(upstream, method, finality) metric entries. internal/policy/engine.go:engineSweepInterval:L208 (opens in a new tab)

Observability

erpc_selection_position value semantics (assigned at internal/policy/slot.go:L477 (opens in a new tab)):

• 0 — primary upstream (first element of the JS result array); serves the next request when no failover occurs.
• 1 — first runner-up; tried if the primary fails or is consumed by a concurrent hedge.
• 2, 3, … — third, fourth choices in ranked order.
• -1 — excluded from rotation; all excluded upstreams share -1 regardless of how many or which predicate dropped them.

An upstream can have position=0 in the wildcard slot and position=1 in a per-method slot simultaneously (positions are per-slot).

erpc_selection_eval_errors_total kind values (classified at internal/policy/slot.go:L460-L467 (opens in a new tab)):

• "timeout" — JS eval exceeded evalTimeout (default 100 ms); previous cache retained.
• "throw" — uncaught JS exception (runtime error, user-thrown error).
• "invalid_return" — eval completed but returned a non-array, null/undefined, entries without id, or unknown upstream IDs.

Note: "fallback_default" appears in the metric Help string and in simulator types as a comment, but no production code path currently emits this value. Treat it as forward-compatible documentation only.

erpc_selection_shadow_exclusion_total vs erpc_selection_exclusion_total semantics:

• erpc_selection_exclusion_total — upstream was removed from the ordered list; erpc_selection_position becomes -1.
• erpc_selection_shadow_exclusion_total — upstream was NOT removed and remains in rotation; shadowExcludeIf predicate fired for monitoring only. Same leaf-slug attribution as excludeIf.

Key dashboarding tip: erpc_selection_position{upstream="X"} == -1 sustained over time means X is stuck excluded — check erpc_selection_exclusion_total{upstream="X"} for the leaf reason slug, then erpc_selection_probe_requests_total{upstream="X"} to confirm probing is running.

Source code entry points

• internal/policy/engine.go (opens in a new tab) — Engine struct; RegisterNetwork, GetOrdered, GetExcluded, PublishRequest; idle slot eviction; probe config reconciliation.
• internal/policy/slot.go (opens in a new tab) — tickOnce full eval cycle; snapshotMetrics; materializeOrder; atomic cache swap at L308-311 (opens in a new tab).
• internal/policy/eval.go (opens in a new tab) — runEval; buildJSUpstreams; readUpstreamMetrics; resolveScoreMultipliers; extractOrderedResult.
• internal/policy/default_policy.js (opens in a new tab) — embedded default policy chain; upgraded from placeholder at register time.
• internal/policy/stdlib/stdlib.js (opens in a new tab) — all chainable Array.prototype methods; predicate factories; sortByScore; stickyPrimary; probeExcluded; latencyDeviationAbove algorithm.
• internal/policy/prober.go (opens in a new tab) — Prober; shadow-mirror request dispatch; windowCounter for minSamples floor; per-upstream in-flight cap.
• erpc/networks.go:L1023-L1074 (opens in a new tab) — request path: GetOrdered call, raw-registry fallback, PublishRequest to probe bus.
• common/defaults.go:L2512-2609 (opens in a new tab) — SelectionPolicyConfig.SetDefaults; all field defaults.
• telemetry/metrics.go:L174-291 (opens in a new tab) — all MetricSelection* Prometheus vars.
• erpc/networks_selection_policy_test.go (opens in a new tab) — integration tests: slow/erroring/lagging/throttled exclusion, fallback tier, safety-net, sticky primary, no-time-based readmit.

Related pages

• Hedge — uses the ordered upstream list; each hedge leg gets the next upstream in selection order.
• Retry — walks the same ordered list sequentially on each retry attempt.
• Circuit breaker — complementary per-upstream state machine checked after selection dispatches to an upstream; open breakers fail fast and the sweep advances down the ranked list.
• Rate limiters — caps traffic per upstream independent of selection scoring.
• Upstream config — where routing.scoreMultipliers and routing.probe are set.
• Survive provider outages — the outcome this feature primarily enables.