Skip to main content
A running squid is a small pipeline of ordinary processes. This page walks through what they are and how data moves between them.

​
The processes

  • The processor — the main Node.js process. It pulls blockchain data, runs your transformation code, and writes results through a store. All indexing logic lives here.
  • A database — usually PostgreSQL (via the TypeORM store); file datasets and BigQuery are alternatives.
  • A GraphQL server (optional) — a separate process that serves the indexed data from PostgreSQL.

​
The data flow

The processor is configured with a data source — where blocks come from and which data items and fields to fetch (see the Fuel data source). Filtering happens upstream: the SQD Network sends only the requested items with only the requested fields, in chunks covering thousands of blocks. Each chunk arrives at your batch handler as a context object:
  • ctx.blocks — the block slice with the requested data items;
  • ctx.store — the persistence interface, bound to the current database transaction;
  • a head flag telling you whether the squid has caught up with the chain.
The handler decodes and transforms the batch — arbitrary TypeScript, including external API calls — and persists results through the store, ideally with one write per table per batch. The store commits the data and the current block position in the same database transaction, which is what makes squids restart-safe: on restart, indexing resumes exactly where the last committed batch ended.

​
Keeping up with the chain

Historical data is served from the SQD Network, which stays some distance behind the tip. The remaining stretch is covered by a complementary real-time source, so squids index unfinalized blocks with minimal latency. Blocks near the tip are treated as hot: if the chain reorganizes, the store rolls back the affected changes and the processor re-runs the replaced blocks — your handler code doesn’t deal with forks.

​
Serving the data

Squids that persist to PostgreSQL can expose the data as a GraphQL API. The schema-first flow: you define entities in a schema file, generate TypeORM model classes from it, and the OpenReader server derives a full-featured API from the same schema. Other serving options, including Hasura, work from the same database. Next: the Quickstart runs this whole pipeline from a template, and Make an indexer walks the development flow in detail.