Skip to main content
Starting with the ArrowSquid release, the processor can ingest data either from a SQD Network gateway or directly from an RPC endpoint. If both a network gateway and a RPC endpoint are provided, the processor will use network data until it reaches the highest block available there, then index the few remaining blocks using the RPC endpoint. This allows squids to combine low sync times with near real-time chain data access. It is, however, possible to use just the RPC endpoint. RPC ingestion can create a heavy load on node endpoints. The load is typically short in duration when a SQD Network gateway is used, and the total number of requests is low. However, the request rate may be sufficient to trigger HTTP 429 responses. Use private endpoints and rate limit your requests with the rateLimit option of setRpcEndpoint().

​
Indexing unfinalized blocks

When ingesting from RPC, Squid SDK can index blocks before they are finalized, enabling real-time use cases. If a blockchain reorganization happens, the processor will roll back any changes to the database made due to orphaned blocks, then re-run its batch handler on consensus blocks. To take advantage of this feature, a squid processor must
  1. have a node RPC endpoint (setRpcEndpoint()) as one of its data sources and
  2. use a Database with unfinalized blocks support (e.g. TypeormDatabase) in its processor.run() call, and
  3. not disable RPC ingestion explicitly with setRpcDataIngestionSettings({ disabled: true }).
With this, the processor will consider some blocks to be “hot”: The processor will periodically (interval setting) poll the RPC endpoint for changes in consensus. When the consensus changes, it will re-run the batch handler with the new consensus data and ask the Database to adjust its state. The Database then must roll back the changes made due to orphaned blocks and apply the new changes. With this, the state of the Database reflects the current blockchain consensus at all times.