Shared state

How it works

Registry and key space. NewSharedStateRegistry wraps one Connector and a clusterKey prefix. Every counter is stored under <clusterKey>/<counter-family>/<upstreamUniqueKey> — for example prod-main/latestBlock/projectX/evm/1/alchemy. The registry holds a sync.Map of live counterInt64 instances; calling GetCounterInt64 with the same key always returns the same object (idempotent LoadOrStore). On first call, two async bootstrap tasks start: one seeds the local counter with a single remote Get, and one calls WatchCounterInt64 to open a long-lived change stream from the connector.

Write path. TryUpdate is synchronous and local-only — it atomically advances the in-memory counter via a CAS loop and then schedules a background push without touching the network. scheduleBackgroundPushCurrent coalesces concurrent pushes: a single goroutine per counter (1) publishes the current state via connector.PublishCounterInt64 for fast peer notification, (2) acquires the distributed lock within lockMaxWait, (3) reads remote state and adopts it if the remote value is newer, then writes local state if local is newer. If the lock cannot be acquired within lockMaxWait, the goroutine falls back to publish-only propagation and reschedules if another update arrived while it was running.

Foreground latency budget. TryUpdateIfStale is used by the EVM state poller to debounce expensive RPC calls. It launches the upstream RPC in a background goroutine and waits at most updateMaxWait for the result. If the RPC finishes within that window the fresh value is returned synchronously; otherwise the stale local value is returned immediately and the RPC completes asynchronously. The foreground path never acquires the distributed lock.

Rollback handling. All counter updates enforce a rollback policy. latestBlock, finalizedBlock, and servedLatestBlock/servedFinalizedBlock use ignoreRollbackOf = 1024 — decreases up to 1024 blocks are silently discarded as noise from lagging upstreams; larger gaps (genuine reorgs or provider resets) are accepted and trigger the OnLargeRollback callback. earliestBlock uses ignoreRollbackOf = 0 — all decreases are accepted because node pruning genuinely shifts the historical anchor.

Timestamp vector clock and conflict resolution. Each counter state is serialised as {"v":<int64>,"t":<unix_ms>,"b":"<instance_id>"}. The t field doubles as both a vector clock for conflict resolution and a staleness indicator. allocateUpdatedAtMs ensures local timestamps are strictly monotonic even under concurrent goroutines (a CAS spin-loop). Remote state is only adopted when st.UpdatedAt > currentTs, providing idempotency and protection against out-of-order network delivery. When a remote update has a newer timestamp but a value that would imply a small rollback (within ignoreRollbackOf), advanceTimestampPast is called to advance the local timestamp past the remote's so the local (higher) value wins the next reconciliation cycle.

Transport differences by connector. All five connectors implement the same GetCounterInt64 / PublishCounterInt64 / WatchCounterInt64 interface; propagation latency differs significantly:

• Redis — push via PUBLISH counter:<fullKey> (e.g. counter:prod-main/latestBlock/...) and subscribe via a single shared PSubscribe("counter:*") connection that matches all counters regardless of cluster; subscribers filter by exact key inside runMessageLoop. Reconnects automatically with exponential backoff (1 s → 30 s with jitter) and a 5-minute periodic poll fallback; pollAllKeys() is triggered after each reconnect to close the gap during the reconnection window. Subscriber channels are buffered (size 1, drop-on-full) so slow consumers never block the message loop. Source: data/redis_pubsub_manager.go:L155-460, data/redis.go:L591
• PostgreSQL — push via pg_notify using a channel name derived by sanitizeChannelName(fmt.Sprintf("counter_%s", key)) to strip characters illegal in PostgreSQL NOTIFY channel names; LISTEN-based subscription with a 30 s polling fallback. Source: data/postgresql.go:L620-718
• DynamoDB — no push primitive. PublishCounterInt64 is a documented no-op; propagation relies on polling at statePollInterval (default 5 s). Cross-pod convergence latency is bounded by the poll interval, not a push event. Source: data/dynamodb.go:L723-829
• Memory — both WatchCounterInt64 and PublishCounterInt64 are no-ops; in-process only, no cross-pod coordination. Source: data/memory.go:L195-207
• gRPC — both WatchCounterInt64 and PublishCounterInt64 return an error; the gRPC connector cannot participate in shared state.

Deadlock-prevention invariant. The foreground paths (TryUpdate and TryUpdateIfStale) never acquire the distributed lock. Only the background scheduleBackgroundPushCurrent goroutine acquires it, and only after any local update is complete. This strict lock-ordering rule was introduced to fix a deadlock where the old design could have scheduleBackgroundPushCurrent hold the distributed lock and wait for updateMu, while TryUpdateIfStale held updateMu and waited for the distributed lock. The test TestLockOrderSimulation_OldBuggyBehavior documents the old behavior.

Counter families. Four families of counters are materialised during normal operation, all stored under <clusterKey>/<family>/<...> in the backing store:

1. latestBlock/<projectId>/<arch>/<networkId>/<upstreamId> — latest block per upstream (ignoreRollbackOf=1024).
2. finalizedBlock/<projectId>/<arch>/<networkId>/<upstreamId> — finalized block per upstream (ignoreRollbackOf=1024).
3. earliestBlock/<projectId>/<arch>/<networkId>/<upstreamId>/<probeType> — earliest available block per upstream per probe type (ignoreRollbackOf=0; all decreases accepted because node pruning genuinely shifts the historical anchor).
4. servedLatestBlock/<networkId>/<grp:<hash>> and servedFinalizedBlock/<networkId>/<grp:<hash>> — cross-pod monotonic served-tip per tag-group.

The per-upstream key component is derived via common.UniqueUpstreamKey(up) which hashes the upstream's project/network/upstream triple to prevent collisions between upstreams with the same ID in different networks or projects. Source: common/upstream.go:L62-65

Instance ID. Each counter state carries an UpdatedBy field ("b" in JSON) identifying which pod wrote it. It is resolved from the first non-empty value of: env INSTANCE_ID → env POD_NAME → env HOSTNAME → os.Hostname() → "unknown". This is diagnostic-only and does not affect routing. Source: data/shared_state_registry.go:L82-98

SuggestLatestBlock fast path. When the eRPC proxy observes a block number in an upstream response, it immediately calls SuggestLatestBlock(blockNumber) which calls TryUpdate if the new value is greater than the current local value. This avoids a full polling round-trip and immediately triggers background propagation, advancing the shared counter without waiting for the next polling tick. Source: architecture/evm/evm_state_poller.go:L459-481

IsStale semantics. IsStale(d) returns true if updatedAtUnixMs <= 0 (counter never written) or if time.Since(time.UnixMilli(updatedAtUnixMs)) > d. It is the gate inside TryUpdateIfStale before attempting a refresh. Source: data/shared_state_variable.go:L37-43

OnValue / OnLargeRollback callbacks. Registered at counter construction time; called synchronously from triggerValueCallback within both processNewValue (local write path) and processNewState (remote sync path) after a successful CAS. The EVM state poller registers OnValue to push the new block number into the health tracker for metrics and OnLargeRollback to record large rollback events. Source: architecture/evm/evm_state_poller.go:L126-139

Poll timeout in EVM state poller. The state poller computes its per-tick context timeout as lockTtl + 15s (minimum 30 s) to allow time for acquiring the distributed lock plus network operations. Source: architecture/evm/evm_state_poller.go:L174-187

Bootstrap task retry. If WatchCounterInt64 fails at startup (connector not ready, network error), initCounterSync marks the watch task as failed and the Initializer retries it in the background. During this window the counter operates in local-only mode. Source: data/shared_state_registry.go:L168-222

Served-tip partitions. Beyond per-upstream counters, eRPC also maintains network-wide servedLatestBlock and servedFinalizedBlock counters, plus per-tag-group partitions for tag-aware routing. The partition key is "grp:" + hex(sha256(sorted_upstream_ids)[:8]) so equivalent selectors map to the same key. A cap of 16 partitions per network prevents cardinality explosion from pathological configs. Once the cap is reached, new tag-group selectors fall back to the stateless subnet-minimum.

Fallback when absent. erpc/erpc.go:L49-L66 synthesises an in-memory registry if sharedState == nil; the same GetCounterInt64 API works identically but no cross-pod propagation occurs. Init failures are non-fatal: erpc/init.go logs a warning and continues with a synthesised in-memory registry. A broken Redis config therefore produces degraded per-pod behavior, not a hard startup failure.

Config schema

All fields are under database.sharedState in erpc.yaml. The block is optional; its absence causes a silent fallback to a process-local in-memory registry. Struct at common/config.go:L269-286, defaults at common/defaults.go:L789-828, validation at common/validation.go:L263-309.

Top-level clusterKey (common/config.go:L40): defaults to "erpc-default". Propagated to database.sharedState.clusterKey if that field is empty.

DynamoDB-specific statePollInterval (common/config.go:L441): default 5s. Controls the polling rate inside WatchCounterInt64 since DynamoDB has no pub/sub primitive.

Worked examples

All patterns below are distilled from real production fleets; comments explain the non-obvious choices.

1. Minimal Redis config with an explicit cluster key (recommended baseline). Every production deployment sets a unique clusterKey so that staging and production instances sharing one Redis instance don't contaminate each other's block-number state. The explicit connector.id avoids the auto-assignment gotcha where shared-state connectors get "redis" instead of "shared-state-redis":

database:  sharedState:    # unique per deployment — prevents cross-pod block-height contamination    # if two clusters share the same Redis instance (e.g. prod + staging)    clusterKey: "prod-us-east-1"    connector:      # always set id explicitly: auto-assigned id here is "redis", not "shared-state-redis"      id: "shared-state-redis"      driver: redis      redis:        uri: "${SHARED_STATE_REDIS_URL}"

2. Multi-region fleet with relaxed lock and wide foreground wait. When pods span multiple regions, the Redis round-trip from any one pod may vary. This production multi-region config raises lockTtl to 30 s to accommodate the longer distributed-lock cycle, sets lockMaxWait to 200 ms to fail fast and fall back to publish-only propagation rather than blocking, and raises updateMaxWait to 2 s to give the foreground refresh more time before falling through to a stale value:

database:  sharedState:    clusterKey: "fly-prod-aggregator-shard0"    # long TTL covers one full Get+Set cycle at cross-region Redis latency;    # must stay > fallbackTimeout (default 3s), so 30s is generous    lockTtl: "30s"    # fail fast on lock contention — proceed with publish-only; never block requests    lockMaxWait: "200ms"    # cross-region Redis RTT can be 50–200ms; give the refresh goroutine time to finish    # synchronously before falling through to the stale value    updateMaxWait: "2s"    connector:      driver: redis      redis:        uri: "redis://erpc-redis.internal:6379"

3. Same Redis, two logical clusters (staging + prod). When you can't provision separate Redis instances, use distinct clusterKey values. Counter keys are prefixed with the cluster key so staging/latestBlock/... and prod/latestBlock/... never collide, even though both clusters publish to the same Redis PUBLISH channel namespace. The fallbackTimeout and lockTtl defaults are left in place since in-cluster Redis latency is well under 3 s:

# staging erpc.yamldatabase:  sharedState:    # distinct from the production clusterKey — same Redis, no cross-contamination    clusterKey: "staging"    connector:      driver: redis      redis:        uri: "${SHARED_STATE_REDIS_URL}"---# production erpc.yamldatabase:  sharedState:    clusterKey: "prod"    connector:      driver: redis      redis:        uri: "${SHARED_STATE_REDIS_URL}"

4. DynamoDB for AWS-only deployments. When Redis or PostgreSQL aren't available, DynamoDB is a valid fallback, but PublishCounterInt64 is a no-op — propagation relies entirely on polling. Cross-pod convergence latency is bounded by statePollInterval (default 5 s) rather than a push event. Acceptable for batch indexing workloads that tolerate slightly stale block-height views, but not recommended for latency-sensitive RPC fleets:

database:  sharedState:    clusterKey: "prod-main"    connector:      driver: dynamodb      dynamodb:        region: "us-east-1"        table: "erpc-shared-state"        # DynamoDB has no pub/sub — all propagation is via polling;        # lower values cost more DynamoDB read capacity; default 5s is the floor        statePollInterval: 5s

5. No shared state (single-pod or dev). Omitting the block entirely gives you a silent per-process in-memory registry — no config needed, same API, no cross-pod propagation:

# No database.sharedState block — eRPC synthesises an in-memory registry automatically
database: {}

Request/response behavior

The shared-state layer has no effect on JSON-RPC request or response shapes visible to clients. Its impact is routing-internal:

• Block-number reads (latestBlock, finalizedBlock) used by the upstream selector and tag-aware routing come from the in-process atomic copy — zero latency on the request path.
• Block-number refreshes are debounced via TryUpdateIfStale; the foreground caller waits at most updateMaxWait (default 50 ms). If the refresh completes within that window the fresh value influences the current request; otherwise the stale value is used and the refresh completes asynchronously for the next request.
• Served-tip counters (servedLatestBlock, servedFinalizedBlock) are updated after a response is selected and do not add latency. They gate the "never serve older than seen" invariant that prevents block-number regressions across requests in a scaled fleet.
• A broken or unreachable shared-state connector degrades to per-pod local state — no error is returned to the client; routing quality degrades silently.

Best practices

• Always set a unique clusterKey per logical deployment (e.g. "prod-v2", "staging"). The default "erpc-default" is shared by every unconfigured instance pointing at the same store — cross-contamination breaks block-height monotonicity.
• Always set connector.id explicitly (id: "shared-state-redis"). The auto-assigned id for shared-state connectors is string(driver) (e.g. "redis"), not the "shared-state-redis" pattern used by cache connectors — copying connector blocks between scopes silently changes the id.
• Prefer Redis or PostgreSQL over DynamoDB for shared state. DynamoDB has no pub/sub; convergence latency is 5 s by default vs. milliseconds for Redis pubsub.
• Do not set fallbackTimeout low to try to speed up request latency — fallbackTimeout governs background I/O only. The foreground latency knob is updateMaxWait.
• Keep lockTtl > fallbackTimeout (the default 4 s > 3 s relationship). Validation enforces this, but understand it: lockTtl must be long enough for one full Get+Set cycle under the lock.
• In AWS deployments with Redis, set pool_size in the Redis URI (?pool_size=10) to avoid connection exhaustion under bursty background push traffic.
• Monitor erpc_upstream_block_head_large_rollback in production. A sustained non-zero value means an upstream is experiencing genuine reorgs or provider resets — something worth routing around.

Edge cases & gotchas

1. Value=0 is valid, UpdatedAt=0 is not. Genesis block (earliestBlock=0) is a legitimate counter value. Uninitialised state is signalled by UpdatedAt <= 0, never by Value == 0. Source: data/connector.go:L32-34
2. Stale lock expiry is silent. If a remote operation takes longer than lockTtl, the lock key expires. Unlock returns "lock was already expired" logged at DEBUG — expected and benign. Setting lockTtl < fallbackTimeout is rejected by validation to prevent this in every call. Source: common/validation.go:L292-295
3. Remote-ahead adoption happens in background. A fresh pod can have its local counter bumped forward by a peer before the pod's first RPC poll completes. Source: data/shared_state_variable.go:L686-691
4. Small-rollback rejection advances local timestamp. When a remote update has a newer UpdatedAt but a value within the rollback threshold, advanceTimestampPast is called to ensure the local (higher) value wins the next reconciliation. Source: data/shared_state_variable.go:L211-218
5. TryUpdate must not acquire updateMu. Holding updateMu during TryUpdate would block a concurrent TryUpdateIfStale caller already holding the mutex that could in turn wait for background operations — a classic deadlock. The old design had this bug; test TestLockOrderSimulation_OldBuggyBehavior documents it. Source: data/shared_state_variable_deadlock_test.go:L95-212
6. continueAsyncRefresh goroutine leak protection. If the downstream RPC ignores its context and never returns, a bounded timer of fallbackTimeout + 1s kills the helper. Source: data/shared_state_variable.go:L511
7. Double-checked staleness. TryUpdateIfStale checks IsStale before and after acquiring updateMu to prevent a thundering herd where N goroutines all see stale and all fire a refresh after the first one already updated the value. Source: data/shared_state_variable.go:L399-413
8. Redis pubsub channel buffered=1, drop-on-full. If the registry consumer falls behind, messages are silently dropped. The 5-minute periodic pollingLoop is the recovery path. Source: data/redis_pubsub_manager.go:L450-457
9. DynamoDB PublishCounterInt64 is a no-op. Cross-pod latency is bounded by statePollInterval (default 5 s). Using DynamoDB means significantly higher convergence latency than Redis or PostgreSQL. Source: data/dynamodb.go:L825-829
10. WatchCounterInt64 channel closed on shutdown. RedisPubSubManager.stop() closes all subscriber channels on app shutdown; the registry goroutine observing the channel receives ok=false and marks the watch task as failed. This is benign since the app context is also cancelled at that point. Source: data/shared_state_registry.go:L200-207
11. clusterKey propagation order. Explicit database.sharedState.clusterKey wins over the inherited top-level clusterKey. Source: common/defaults.go:L803-805
12. Served-tip partition cap at 16 per network. Once reached, new tag-group selectors fall back to stateless subnet-minimum rather than getting a monotonic counter. Source: erpc/networks.go:L80-84
13. init.go treats shared-state failure as a warning, not fatal. A broken Redis config produces degraded per-pod behavior, not a hard startup failure. Source: erpc/init.go:L93-97
14. database.sharedState nil means SetDefaults is never called on it. The synthesised config at erpc/erpc.go:L49-L66 exists only in local scope and does not write back to cfg.Database.SharedState. Source: common/defaults.go:L837-840
15. Connector id auto-assignment differs between shared-state and other scopes. Shared-state connector id defaults to string(driver) (e.g. "redis"), not "shared-state-redis". Source: common/defaults.go:L799-801

Observability

OTel trace spans:

• CounterInt64.TryUpdate — every foreground local-update call; attributes: key, new_value (when detailed tracing enabled).
• CounterInt64.TryUpdateIfStale — attributes: key, staleness_ms, skipped_not_stale, skipped_not_stale_after_mutex, role, result_path (got_result vs timeout), async_refresh, foreground_remote_io_disabled.
• CounterInt64.TryUpdateIfStale.AcquireMutex — measures mutex wait time inside TryUpdateIfStale.
• CounterInt64.TryUpdateIfStale.ExecuteRefresh — spans the actual RPC call inside the refresh goroutine; attribute timeout_ms.
• RedisConnector.PublishCounterInt64 — spans each publish call; attributes key, value, updated_at, updated_by (when detailed tracing enabled).
• PostgreSQLConnector.PublishCounterInt64 — same pattern as Redis span above.

Key log messages (component sharedState):

• "no remote initial value found for counter" (DEBUG) — normal first-run; counter seeds from zero.
• "fetched initial value for counter" (DEBUG) — successful seed from remote on first bootstrap.
• "received new value from shared state sync" (DEBUG) — pubsub/watch update received and processed.
• "failed to setup counter sync" (ERROR) — watch setup failed; will retry via initializer.
• "lock held by another instance, proceeding with local lock" (DEBUG) — normal under load; background push proceeds without distributed lock.
• "lock acquisition timed out waiting for other instance" (WARN) — lock contention above lockMaxWait; falling back to publish-only propagation.
• "lock expired during operations (expected behavior)" (DEBUG) — expected when operations take near lockTtl.
• "published counter value to remote" (DEBUG) — successful remote publish.
• "pubsub reconnected successfully" (INFO) — Redis pubsub recovery.
• "counter value increased (remote)" (TRACE) — value advanced by remote pubsub event.
• "small rollback ignored (remote)" (TRACE) — rollback inside threshold, discarded.
• "large rollback applied (remote)" (TRACE) — real reorg or provider reset accepted.

Source code entry points

• data/shared_state_registry.go:L38-L80 (opens in a new tab) — NewSharedStateRegistry: registry construction, clusterKey prefix, sync.Map of live counters
• data/shared_state_variable.go:L364-L707 (opens in a new tab) — TryUpdate, TryUpdateIfStale, scheduleBackgroundPushCurrent, lock ordering, rollback handling, timestamp CAS helpers
• data/shared_state_registry.go:L102-L222 (opens in a new tab) — GetCounterInt64: idempotent counter creation, bootstrap tasks, WatchCounterInt64 goroutine
• data/redis_pubsub_manager.go:L155-L460 (opens in a new tab) — RedisPubSubManager: single PSubscribe("counter:*"), reconnect backoff, periodic poll fallback, copy-on-write subscriber slice
• erpc/networks.go:L55-L96 (opens in a new tab) — served-tip partition key derivation and 16-partition cap
• erpc/erpc.go:L49-L66 (opens in a new tab) — in-memory fallback synthesis when sharedState == nil
• common/defaults.go:L789-L828 (opens in a new tab) — SharedStateConfig.SetDefaults; all default values including connector id override
• data/shared_state_sources.go (opens in a new tab) — named constants for the source field in logs and traces (UpdateSourceRemoteSync, UpdateSourceTryUpdate, etc.)
• data/shared_state_variable_deadlock_test.go:L95-L212 (opens in a new tab) — lock-ordering correctness tests

Related pages

• Connectors — the backing store options (Redis, PostgreSQL, DynamoDB, memory) that power shared state.
• Tag-aware routing — uses servedLatestBlock/servedFinalizedBlock partition counters that shared state keeps monotonic across pods.
• EVM state poller — the consumer that calls TryUpdateIfStale per upstream; registers OnValue/OnLargeRollback callbacks.
• Survive provider outages — routing quality under failover depends on consistent block-height awareness across pods.
• Deployment — horizontal scaling guidance where shared state becomes necessary.