# Cordoning

> Source: https://docs.erpc.cloud/operation/cordoning
> Pull any upstream out of routing instantly with one admin call — no metric window to wait for, no config redeploy required.
> Format: machine-readable markdown export of the docs page above.
> All collapsible AI sections are inlined and fully expanded.

# Cordoning

When a vendor starts acting up — latency spikes, wrong responses, a quota incident — you
want it gone from routing *right now*, not after a 15-minute error-rate window closes.
One admin RPC call cordons the upstream instantly. Another call brings it back just as
fast. No config change, no redeploy, no restart.

## Agent reference

Copy one of these prompts into your AI agent session (Claude Code, Cursor, …) — each one
points the agent at this page's machine-readable reference so it can do the work correctly:

**Prompt Example #1: pull a degraded upstream out of rotation right now**

```text
One of my eRPC upstreams is having a vendor incident and I need to remove it from routing
immediately without redeploying. Show me the admin API calls to cordon it, verify the
change, and restore it when the incident is resolved. My admin endpoint is configured in
my eRPC config. Read the full reference first:
https://docs.erpc.cloud/operation/cordoning.llms.txt
```

**Prompt Example #2: cordon a single broken RPC method**

```text
A specific upstream in my eRPC setup is timing out only on eth_getLogs but is fine for
everything else. Show me how to cordon just that method without pulling the whole upstream
out of rotation, and explain why erpc_listCordoned won't show it. Work with my existing eRPC config.
Reference: https://docs.erpc.cloud/operation/cordoning.llms.txt
```

**Prompt Example #3: alert and dashboard on cordon state**

```text
Set up Prometheus alerts and dashboard panels so I can see which eRPC upstreams are
currently cordoned, how long they've been cordoned, and how many times they've been
cordoned in the past 24 hours. Explain the erpc_upstream_cordoned gauge vs the event
counter. Reference: https://docs.erpc.cloud/operation/cordoning.llms.txt
```

---

### Cordoning — full agent reference

### How it works

Two kinds of cordon exist and they never overwrite each other:

- **Operator cordons** (`erpc_cordonUpstream` with `method: "*"`, the default) are one
  shared counter per upstream — the same primitive that shares block heights across pods.
  Key `<clusterKey>/v2/operatorCordon/<projectId>/<upstreamId>/<hash>`, value = unix-ms
  when cordoned, `0` when lifted. Shared state does the rest: pub/sub propagation on
  `redis`/`postgresql` (milliseconds), background reconcile on `dynamodb` (seconds), and
  an initial fetch when the upstream bootstraps, so a restarted or scaled-out pod restores
  the cordon before it serves anything. Writes are local-first and pushed in the
  background, like every other shared counter; if the store is unreachable the cordon
  applies on this pod and reaches peers when the store returns. With the `memory` driver
  (the default when `database.sharedState` is omitted) the counter is in-process only.
- **Automatic cordons** are set by in-process detectors (consensus sit-out, SVM health,
  EVM chain-identity mismatch) on the health tracker's `(upstream, method, finality=All)`
  cell, as are method-scoped operator cordons (`method: "eth_getLogs"`). They are per pod.
  While the shared operator cordon is held, a detector cannot lift the `"*"` cell — a
  consensus penalty ending never releases an operator cordon.

Routing, `erpc_listCordoned` and the health check all read one lookup,
`Tracker.CordonedReason(upstream, method)`: the wildcard `"*"` cell first, then the
method cell. The policy reads it per upstream rather than from the slot's metrics bucket,
so a wildcard cordon shadows every method slot at every selection-policy `evalScope`.

The state feeds the selection policy through the `cordonedReason` field exposed on each
upstream's JS metrics object. The built-in default policy calls `.removeCordoned()` as its
first chain step, so a cordon takes effect within one eval interval (default 15 s) after
it reaches a replica. A custom `evalFunc` must call `.removeCordoned()` explicitly or the
cordon has no routing effect.

`erpc_uncordonUpstream` lifts the operator cordon for the whole fleet; on every replica
that also lifts an automatic wildcard cordon (they share the `"*"` cell) — the operator
call is the override for a detector verdict. Uncordoning a method never lifts a wildcard
cordon.

The reason string is kept only on the replica that took the call; peers and restarted
pods report `"operator cordon (set on another replica)"`. Re-cordoning an already
cordoned upstream keeps the original start, whichever replica serves it, so
`erpc_upstream_cordon_duration_seconds` measures the whole incident.

Use cordons for outages measured in minutes to days. For permanent exclusion, remove the
upstream from config or set `ignoreMethods: ["*"]` and redeploy.

### Config schema

Cordoning has no config-file fields. It is controlled entirely at runtime via admin RPC.
The selection policy's `evalFunc` implicitly controls whether cordon state is respected —
it requires `.removeCordoned()` in the chain. For `evalFunc` and `evalInterval` config
fields, see [Selection &amp; scoring](/config/projects/selection-policies.llms.txt).

### Worked examples

**1. Incident response: cordon a vendor immediately.**
You notice `erpc_upstream_error_rate` spiking on `alchemy-eth-1` before the error window
is wide enough to trigger automatic exclusion. Cordon it now:

```bash
curl -X POST http://localhost:4000/admin \
  -H 'Content-Type: application/json' \
  -d '{
    "jsonrpc": "2.0", "id": 1,
    "method": "erpc_cordonUpstream",
    "params": [{"projectId": "main", "upstream": "alchemy-eth-1",
                "reason": "vendor incident #12345"}]
  }'
```

```json
{"jsonrpc":"2.0","id":1,"result":{
  "projectId": "main",
  "upstream": "alchemy-eth-1",
  "method": "*",
  "cordoned": true,
  "reason": "vendor incident #12345"
}}
```

Omitting `method` cordons the upstream for all methods (`"*"`).

**2. Method-scoped cordon: isolate a broken RPC method.**
A vendor is fine for most calls but `eth_getLogs` is timing out past 30 s. Cordon just
that method while the rest of the vendor's capacity keeps working:

```bash
curl -X POST http://localhost:4000/admin \
  -H 'Content-Type: application/json' \
  -d '{
    "jsonrpc": "2.0", "id": 2,
    "method": "erpc_cordonUpstream",
    "params": [{"projectId": "main", "upstream": "drpc-eth-1",
                "method": "eth_getLogs", "reason": "p95 > 30 s"}]
  }'
```

A wildcard cordon overrides method-scoped cordons: an upstream with both `"*"` and
`"eth_getLogs"` cordons is excluded for all methods. Uncordoning a specific method does
not lift a wildcard cordon.

**3. List and restore.**
During an incident, list all whole-upstream cordons, then restore when the vendor
confirms recovery:

```bash
# list whole-upstream cordons in a project
curl -X POST http://localhost:4000/admin \
  -H 'Content-Type: application/json' \
  -d '{"jsonrpc":"2.0","id":3,"method":"erpc_listCordoned",
       "params":[{"projectId":"main"}]}'
```

```json
{
  "projectId": "main",
  "cordoned": [
    {"upstream": "alchemy-eth-1", "reason": "vendor incident #12345"}
  ]
}
```

`erpc_listCordoned` only returns upstreams with a `"*"`-scope cordon (operator or
automatic). Method-scoped cordons are invisible to it; read the `erpc_upstream_cordoned`
gauge labels to enumerate them.

```bash
# restore the upstream
curl -X POST http://localhost:4000/admin \
  -H 'Content-Type: application/json' \
  -d '{"jsonrpc":"2.0","id":4,"method":"erpc_uncordonUpstream",
       "params":[{"projectId":"main","upstream":"alchemy-eth-1",
                  "reason":"vendor confirmed resolved"}]}'
```

**4. Custom evalFunc — explicit `.removeCordoned()` required.**
If you have replaced `selectionPolicy.evalFunc` with a custom function, you must call
`.removeCordoned()` explicitly near the start of the chain. Without it the
`cordonedReason` field on the upstream's metrics object is populated, but no step actually
drops the upstream from the ordered list — the cordon has no routing effect.

Source: [`internal/policy/default_policy.js:L1`](https://github.com/erpc/erpc/blob/main/internal/policy/default_policy.js#L1) — `.removeCordoned()` is the first call.

### Request/response behavior

- `erpc_cordonUpstream` `method` param defaults to `"*"` when absent; `reason` defaults
  to `"admin: manual cordon"`. `"*"` sets the shared operator cordon; any other method
  sets a cordon on this replica only. [[`erpc/admin.go:L656-694`](https://github.com/erpc/erpc/blob/main/erpc/admin.go#L656-L694)]
- `erpc_uncordonUpstream` `reason` defaults to `"admin: manual uncordon"` — a distinct
  string from the cordon default; it is used only for the local automatic cordon it lifts.
- `erpc_listCordoned` only returns upstreams where `CordonedReason("*")` returns
  `cordoned=true` — method-scoped cordons are not listed.
- Routing changes on the next policy eval tick (default 15 s) after the cordon reaches a
  replica; shared-state propagation to peers is milliseconds on `redis`/`postgresql` and
  seconds on `dynamodb`.
- A cordoned upstream is placed at position `-1` in `erpc_selection_position` once
  `.removeCordoned()` drops it.
- All admin RPCs require `admin.auth` configured; missing `admin:` block returns
  `"admin is not enabled for this project"` (401); present `admin:` but absent
  `admin.auth:` returns `"admin auth not configured"` (401).
  [[`erpc/admin.go:L26-30`](https://github.com/erpc/erpc/blob/main/erpc/admin.go#L26-L30)]

### Best practices

- **Cordon early, not late.** The selection policy's metric-driven exclusion needs sample
  accumulation; cordoning is zero-lag. When you see a vendor degrading in dashboards,
  cordon it immediately rather than waiting for `errorRateAbove(0.7)` to trigger.
- **Always supply a reason string.** The reason appears in `erpc_upstream_cordoned` gauge
  labels and in `erpc_listCordoned` output. A reason like `"incident-#12345 alchemy"` makes
  incident timelines clear and helps correlate with duration histograms post-incident.
- **Prefer method-scoped cordons when feasible.** If a vendor is broken only for
  `eth_getLogs` but healthy for `eth_call`, a method-scoped cordon keeps the vendor in
  rotation for the methods it can serve — reducing pressure on remaining upstreams.
- **Remember the 15 s propagation delay.** The tracker is updated immediately, but the
  ordered-list cache is rebuilt on the next eval tick. Build runbooks around this delay
  (e.g. wait 20 s before verifying traffic shifted).
- **Use a remote `database.sharedState` connector for fleets.** Operator cordons reach
  every replica and survive restarts only through `redis`, `postgresql` or `dynamodb`;
  `memory` keeps them in-process.
- **Do not use cordon for permanent exclusions.** For a vendor you want permanently
  removed, update the config; a shared cordon outlives the pod that set it.
- **Custom evalFunc operators: `.removeCordoned()` must be explicit.** The default policy
  includes it; a custom `evalFunc` that omits it will silently ignore all admin cordons.

### Edge cases &amp; gotchas

1. **Uncordoning a method does not lift a wildcard cordon.** `IsCordoned` checks the
   operator cordon and the `"*"` cell first, so the upstream stays out of rotation for all
   methods as long as either is active. Source: [`health/tracker.go:L900-939`](https://github.com/erpc/erpc/blob/main/health/tracker.go#L900-L939).
2. **Automatic and method-scoped cordons are per pod; whole-upstream operator cordons are
   per fleet.** A consensus sit-out, chain-identity cordon or `method: "eth_getLogs"` cordon
   on one replica is not visible on others. An `erpc_uncordonUpstream` for `"*"` reaches
   every replica and lifts automatic wildcard cordons there too.
3. **The reason travels only with the pod that took the call.** Peers and restarted pods
   show `"operator cordon (set on another replica)"` in `erpc_listCordoned` and in the
   `erpc_upstream_cordoned` `reason` label.
4. **Custom evalFunc without `.removeCordoned()` ignores all cordons.** `cordonedReason`
   is populated on the JS upstream object, but no routing effect occurs unless
   `.removeCordoned()` (or equivalent logic) appears in the chain.
5. **Routing effect is delayed by up to `evalInterval`.** The tracker is updated
   immediately, but the ordered-list cache used by the request path is rebuilt on the next
   eval tick. Requests in flight can still reach the upstream for up to one eval interval.
6. **State-poller is unaffected by cordon.** The EVM state poller for a cordoned upstream
   continues running, keeping its latest/finalized block numbers fresh. When uncordoned,
   the upstream's metrics are current and re-admission scoring works immediately.
7. **Shadow mirroring is unaffected by cordon.** Shadow upstreams receive async-mirrored
   traffic regardless of cordon state; cordon only affects routing of real requests.
8. **`erpc_cordonUpstream` is idempotent on the same upstream.** Repeated calls update the
   local `reason` but do not reset the shared value or fire multiple `cordon` event
   counter increments, whichever replica serves them.
   Source: [`upstream/upstream.go:L1526-1536`](https://github.com/erpc/erpc/blob/main/upstream/upstream.go#L1526-L1536).
9. **The default policy's `.whenEmpty(() => upstreams)` safety net can re-admit a cordoned
   upstream.** If every other upstream is health-excluded (for example far behind the
   network head), the raw set is used rather than failing closed.
10. **`erpc_listCordoned` requires `admin.auth` configured.** If the `admin:` section is
    present but `admin.auth:` is absent, the request returns `"admin auth not configured"`.
    If the `admin:` section is entirely missing, it returns `"admin is not enabled for this
    project"`. Both are 401 responses.
    Source: [`erpc/admin.go:L26-30`](https://github.com/erpc/erpc/blob/main/erpc/admin.go#L26-L30).

### Observability

| Metric | Type | Labels | When it fires |
|---|---|---|---|
| `erpc_upstream_cordoned` | gauge | project, vendor, network, upstream, category (=method), reason | Set to 1 on cordon, 0 on uncordon; persists until uncordon or process restart |
| `erpc_upstream_cordon_event_total` | counter | project, network, upstream, action | Edge transitions only: OFF→ON (`action="cordon"`) and ON→OFF (`action="uncordon"`); repeated cordons do not increment |
| `erpc_upstream_cordon_duration_seconds` | histogram | project, network, upstream | Observed once per uncordon on each replica; value = `now − CordonedAtMs` as seen by that replica; buckets 1 s … 86400 s |
| `erpc_selection_position{upstream=…}` | gauge | project, network, method, upstream | Set to `-1` for every excluded (including cordoned) upstream after each eval tick |
| `erpc_selection_rejection_total{step="removeCordoned"}` | counter | project, network, method, upstream, step | Per tick × upstream dropped by the `.removeCordoned()` step |

Log messages (DEBUG level, `health/tracker.go`):
- `"cordoning upstream to disable routing"` / `"uncordoning upstream to enable routing"` — every automatic or method-scoped `Cordon` / `Uncordon` call.

### Source code entry points

- [`upstream/upstream.go:L1503-1567`](https://github.com/erpc/erpc/blob/main/upstream/upstream.go#L1503-L1567) — `CordonAdmin` / `UncordonAdmin`, the shared counter (`operatorCordonVar`) that flips the `"*"` cell, and the `Uncordon` guard
- [`health/tracker.go:L842-939`](https://github.com/erpc/erpc/blob/main/health/tracker.go#L842-L939) — `Cordon` / `Uncordon`, `IsCordoned`, `CordonedReason`
- [`data/shared_state_variable.go`](https://github.com/erpc/erpc/blob/main/data/shared_state_variable.go) — `CounterInt64SharedVariable`: the shared primitive (rollback threshold `0` accepts every change, including back to `0`)
- [`erpc/admin.go:L590-736`](https://github.com/erpc/erpc/blob/main/erpc/admin.go#L590-L736) — `erpc_cordonUpstream`, `erpc_uncordonUpstream`, `erpc_listCordoned` handlers
- [`internal/policy/eval.go`](https://github.com/erpc/erpc/blob/main/internal/policy/eval.go) — `cordonedReason` exposed to the JS eval context via `Tracker.CordonedReason`
- [`internal/policy/default_policy.js:L1`](https://github.com/erpc/erpc/blob/main/internal/policy/default_policy.js#L1) — `.removeCordoned()` as first default policy step

### Related pages

- [Admin API](/operation/admin.llms.txt) — authentication setup required before any cordon call will succeed.
- [Selection &amp; scoring](/config/projects/selection-policies.llms.txt) — the `evalFunc` chain where `.removeCordoned()` must appear; also `evalInterval` that controls propagation delay.
- [Survive provider outages](/use-cases/survive-provider-outages.llms.txt) — the incident-response outcome cordoning helps achieve.
- [Upstreams](/config/projects/upstreams.llms.txt) — permanent exclusion via `ignoreMethods: ["*"]` or config removal, the alternative when cordon is not the right tool.

---


## Navigation (machine-readable surface)

- Up: [All pages index](https://docs.erpc.cloud/llms.txt)
- Root index of every page: [llms.txt](https://docs.erpc.cloud/llms.txt) · everything in one file: [llms-full.txt](https://docs.erpc.cloud/llms-full.txt)

### Sibling pages

- [Admin API](https://docs.erpc.cloud/operation/admin.llms.txt) — A built-in operator control plane — inspect topology, cordon sick upstreams without restarts, and manage API keys, all over a secure JSON-RPC 2.0 endpoint.
- [Batching & multiplexing](https://docs.erpc.cloud/operation/batch.llms.txt) — Send one request, get back a merged response — eRPC parallelises inbound batch arrays, re-batches calls to supporting upstreams, and collapses identical in-flight requests so each unique call hits the network exactly once.
- [CLI & env vars](https://docs.erpc.cloud/operation/cli.llms.txt) — Start, validate, or inspect your eRPC config from the command line — then deploy with confidence knowing exactly what the engine will run.
- [Directives](https://docs.erpc.cloud/operation/directives.llms.txt) — Send an HTTP header or query param and change routing, caching, validation, or consensus for exactly that one request — no restarts, no config changes.
- [Healthcheck](https://docs.erpc.cloud/operation/healthcheck.llms.txt) — One endpoint that tells Kubernetes exactly when your pod is ready, draining, or broken — with eight probe strategies from "any upstream alive" to live chain-ID verification.
- [Monitoring & metrics](https://docs.erpc.cloud/operation/monitoring.llms.txt) — Every subsystem in eRPC — upstreams, cache, rate limits, consensus, hedging — emits Prometheus metrics. One scrape target, full visibility, zero instrumentation work.
- [Production checklist](https://docs.erpc.cloud/operation/production.llms.txt) — Go live confidently — a short list of settings that separate a hardened eRPC deployment from a dev-mode one.
- [Tracing & logging](https://docs.erpc.cloud/operation/tracing.llms.txt) — Every request, cache lookup, and upstream call becomes a searchable span — shipped to any OTel backend. Secrets never leave the process.
- [URL structure](https://docs.erpc.cloud/operation/url.llms.txt) — One URL pattern routes every chain — domain and network aliases let you publish clean, memorable endpoints without touching your app code.