Skip to main content

RPC ingestion and reorgs

Starting with the ArrowSquid release, the processor can ingest data either from a SQD Network gateway or directly from an RPC endpoint. If both a network gateway and a RPC endpoint are provided, the processor will use network data until it reaches the highest block available there, then index the few remaining blocks using the RPC endpoint. This allows squids to combine low sync times with near real-time chain data access. It is, however, possible to use just the RPC endpoint (e.g. for local development). On EVM it is also possible to use just the gateway (e.g. for non-realtime analytics).

RPC ingestion can create a heavy load on node endpoints. The load is typically short in duration when a SQD Network gateway is used, and the total number of requests is low. However, the request rate may be sufficient to trigger HTTP 429 responses. Use private endpoints and rate limit your requests with the rateLimit option (EVM/Substrate).

Indexing unfinalized blocks​

When ingesting from RPC, Squid SDK can index blocks before they are finalized, enabling real-time use cases. If a blockhain reorganization happens, processor will roll back any changes to the database made due to orphaned blocks, then re-run its batch handler on consensus blocks.

To take advantage of this feature, a squid processor must

  1. have a node RPC endpoint as one of its data sources and
  2. use a Database with unfinalized blocks support (e.g. TypeormDatabase) in its processor.run() call, and
  3. not disable RPC ingestion explicitly with setRpcDataIngestionSettings({ disabled: true }).

With this, the processor will consider some blocks to be "hot":

All blocks with fewer confirmations than the number set by the setFinalityConfirmations() setting

The processor will periodically (interval setting for EVM, Substrate) poll the RPC endpoint for changes in consensus. When the consensus changes, it will re-run the batch handler with the new consensus data and ask the Database to adjust its state. The Database then must roll back the changes made due to orphaned blocks and apply the new changes. With this, the state of the Database reflects the current blockchain consensus at all times.