EVM fallback data source

EvmFallbackDataSourceBuilder builds a data source that drives an ordered list of downstream sources — Portal datasets, JSON-RPC endpoints, or custom sources — and fails over between them automatically. While the most-preferred source is healthy it is the only one consuming data; when it degrades, the next healthy source takes over at a batch boundary, and when a higher-preference source recovers, the fallback switches back up. The built source is a drop-in replacement for a single EVM Portal stream. For a deployment-oriented walkthrough, see High availability.

Install

npm i @subsquid/squid-sdk @subsquid/evm-rpc @subsquid/evm-normalization

The two peer packages are needed only if an rpc downstream source is used.

Complete example

import {run} from '@subsquid/squid-sdk/processor'
import {EvmFallbackDataSourceBuilder} from '@subsquid/squid-sdk/evm/fallback'

const dataSource = new EvmFallbackDataSourceBuilder()
  .setDownstreamSources([
    {type: 'portal', url: 'https://portal.sqd.dev/datasets/ethereum-mainnet'},
    {type: 'rpc', url: process.env.RPC_URL!, network: 'ethereum-mainnet'},
  ])
  .setFields({log: {topics: true, data: true}})
  .addLog({
    where: {address: [CONTRACT], topic0: [TRANSFER]},
    range: {from: 10_000_000},
  })
  .build()

run(dataSource, db, async (ctx) => {
  // Handler code is oblivious to which source is active.
})

Downstream sources

setDownstreamSources([...]) takes plain tagged config objects, most-preferred first:

{type: 'portal', url, ...} — a SQD Network Portal dataset; same options as setPortal.
{type: 'rpc', url, network?, ...} — a JSON-RPC endpoint; same options as setRpc, including the per-network preset.
{type: 'custom', buildSource} — the escape hatch: a function that receives the shared field selection and requests and returns any EVM data source.

The field selection and query are defined once on the fallback builder and applied to every source, so all sources fetch identical data and indexer output does not depend on which one is active. Each entry takes an optional name used in metrics and logs, defaulting to `${type}-${index}` (portal-0, rpc-1, …).

An rpc standby without a network preset or explicit overrides is not parity-verified: after a failover, its output may differ from the Portal dataset. Set network to a supported slug or chainId.

How failover works

Each source has a trinary health state: healthy, unhealthy, or unknown. unknown lets the first batch ship before any probe completes — a fresh fallback starts streaming immediately. The active source is failed over when:

it errors on a batch request or liveness check (livenessFailThreshold consecutive failures flip it unhealthy);
it lags: its height falls more than maxLagBlocks behind an independent chain-head reference — the maximum head of the other eligible sources. The lag trigger arms only after the tip is first reached, so a long historical backfill never false-fires;
it stalls: its next-batch request stays outstanding longer than maxStalenessMs (time parked waiting for your handler to consume a batch does not count).

An unhealthy source waits out cooldownMs, returns to unknown, and must pass livenessRecoverThreshold consecutive probes (with capability confirmed) to become healthy again. With preferPrimary: 'eager' (the default), a recovered higher-preference source is promoted at the next batch boundary; 'onFailureOnly' switches only on failure. When every source is down, the fallback polls indefinitely by default (the no-downtime goal). Set allDownTimeoutMs to a finite value to throw AllSourcesDownError instead.

Policy reference

setPolicy({...}) accepts a FallbackPolicy; all fields are optional:

Field	Default	Meaning
`preferPrimary`	`'eager'`	`'eager'` switches up to a recovered higher-preference source; `'onFailureOnly'` never switches except on failure
`allDownTimeoutMs`	`null`	`null` polls forever when all sources are down; a finite value throws `AllSourcesDownError` after that long
`allDownPollMs`	`1000`	Backoff between all-sources-down poll attempts
`cooldownMs`	`30_000`	Wait before an `unhealthy` source is re-probed
`livenessFailThreshold`	`2`	Consecutive failed probes that flip a source `unhealthy`
`livenessRecoverThreshold`	`3`	Consecutive passes required to become `healthy`
`maxLagBlocks`	`10`	Failover when the active source lags this far behind the independent head reference; `null` disables
`maxStalenessMs`	`180_000`	Failover when a batch request stays outstanding this long; `null` disables
`freshnessTickMs`	`1000`	How often the staleness clock is checked
`headTtlMs`	`5000`	Max age of a cached per-source head used for the lag reference
`headPollTimeoutMs`	`500`	Time-box on each head poll, so a sick standby cannot stall a healthy active source; `null` disables
`capabilityLookahead`	`16`	Blocks ahead of the indexing frontier a capability probe targets
`capabilityTipMargin`	`16`	Blocks below the chain tip a probe is clamped to

Capability probe

setCapabilityProbe(probe) — default true. A source counts as healthy only once it confirms it can actually serve the configured data at the indexing frontier. This catches reachable-but-incapable nodes — traces or debug_ methods disabled, pruned state — before a switch-up promotes them. Pass false to govern health by liveness alone, or {timeoutMs} (default 30 000 ms) to tune the probe. The probe targets capabilityLookahead blocks ahead of the last committed height, clamped to capabilityTipMargin blocks below the tip so a caught-up source probes a settled block.

Metrics

When the built source runs under run() or Processor with Prometheus enabled, six gauges are registered automatically — no wiring needed:

Gauge	Labels	Meaning
`sqd_fallback_active`	`source`	1 for the active source, 0 for standbys
`sqd_fallback_source_health`	`source`, `state`, `check`, `reason`, `code`	Health per source; the `unhealthy` row carries the cause
`sqd_fallback_lag_blocks`	—	Blocks behind the independent chain-head reference
`sqd_fallback_staleness_ms`	—	How long the active source’s batch request has been outstanding
`sqd_fallback_chain_stalled`	—	1 when every source is stuck at the same head (chain problem, not source problem)
`sqd_fallback_switches`	—	Cumulative switch count

For a custom registry or metric prefix, use fallbackMetricsSink(source, prefix?) from @subsquid/squid-sdk/fallback. Registration is idempotent per registry and prefix, so calling it alongside the automatic registration is safe.

Errors

AllSourcesDownError (importable from @subsquid/squid-sdk/evm/fallback or @subsquid/squid-sdk/fallback) is thrown only when allDownTimeoutMs is set to a finite value and expires. With the default policy the fallback never throws for availability reasons — it polls until a source recovers.

Overview

Pipes SDK

Squid SDK

Alternative clients

Install

Complete example

Downstream sources

How failover works

Policy reference

Capability probe

Metrics

Errors

​Install

​Complete example

​Downstream sources

​How failover works

​Policy reference

​Capability probe

​Metrics

​Errors

Install

Complete example

Downstream sources

How failover works

Policy reference

Capability probe

Metrics

Errors