High availability

A squid built on a single Portal dataset or a single RPC endpoint stops indexing when that endpoint has an outage. The EVM fallback data source removes that single point of failure: it drives an ordered list of sources and fails over — and back — automatically, with no operator action and no gaps in output.

Portal primary, RPC standby

The recommended setup uses the Portal as the primary (fast, free, validated) and a JSON-RPC endpoint as the standby:

import {run, PrometheusServer} from '@subsquid/squid-sdk/processor'
import {EvmFallbackDataSourceBuilder} from '@subsquid/squid-sdk/evm/fallback'

const dataSource = new EvmFallbackDataSourceBuilder()
  .setDownstreamSources([
    {type: 'portal', name: 'portal', url: 'https://portal.sqd.dev/datasets/ethereum-mainnet'},
    {type: 'rpc', name: 'standby', url: process.env.RPC_URL!, network: 'ethereum-mainnet'},
  ])
  .setFields({log: {topics: true, data: true}})
  .addLog({where: {address: [CONTRACT], topic0: [TRANSFER]}, range: {from: 10_000_000}})
  .build()

const prometheus = new PrometheusServer()
prometheus.setPort(3000)

run(dataSource, db, handler, {prometheus})

Install @subsquid/evm-rpc and @subsquid/evm-normalization alongside @subsquid/squid-sdk — they are the optional peers that power the RPC source.

Always set network on the RPC standby to a supported slug or chainId. Without it the standby’s output is not parity-verified against the Portal dataset, and a failover could change what your squid indexes.

What you’ll see on a Portal outage: the fallback logs the failed health check, marks portal unhealthy, and continues the stream from standby at the next batch boundary. When the Portal recovers and passes its probes, indexing switches back up automatically.

Tuning

The full policy has 13 knobs; four matter for a typical deployment:

maxLagBlocks (default 10) — how far the active source may fall behind the other sources’ head before failover. Lower = fresher data, more switch churn; raise it on chains with uneven block production.
maxStalenessMs (default 3 min) — how long a batch request may hang before failover. Lower it if your latency budget is tight.
cooldownMs (default 30 s) — how long a failed source stays benched. Raise it for flappy endpoints to avoid switch thrash.
preferPrimary (default 'eager') — switch back up as soon as the primary recovers, or ('onFailureOnly') stay on the standby until it fails. Eager keeps you on the cheapest/best source; on-failure-only minimizes switches.

What to watch

With Prometheus enabled, the fallback registers its gauges automatically. Alert on:

sqd_fallback_active{source="standby"} == 1 for longer than your Portal-outage tolerance — you are running on the standby;
sqd_fallback_source_health{state="unhealthy"} — a source is down (the reason/code labels say why);
rate(sqd_fallback_switches[15m]) above ~1 — switch thrash, usually a flappy source or an over-tight maxLagBlocks;
sqd_fallback_chain_stalled == 1 — every source is stuck at the same head: the chain has stalled, and failing over would not help.

See the metrics reference for the full gauge list.

Overview

Pipes SDK

Squid SDK

Alternative clients

Portal primary, RPC standby

Tuning

What to watch

​Portal primary, RPC standby

​Tuning

​What to watch

Portal primary, RPC standby

Tuning

What to watch