Make an indexer

This page is a definitive end-to-end guide into practical squid development. It uses templates to simplify the process. Check out Build an indexer from scratch for a more educational barebones approach.

Feel free to also use the template-specific sqd scripts defined in commands.json to simplify your workflow. See the sqd CLI cheatsheet for a short intro.

Prepare the environment

Node.js 20 or newer
Git
Squid CLI
Docker (if your squid will store its data to PostgreSQL)

Understand your technical requirements

Consider your business requirements and find out

How the data should be delivered. Options:
- PostgreSQL with an optional GraphQL API - can be real-time
- file-based dataset - local or on S3
- Google BigQuery
What data should be delivered
Which EVM networks you’ll be indexing - see the supported networks page Note that networks without a Portal dataset can still be indexed with the EVM RPC data source.
What exact data should be retrieved from blockchain(s)
Whether you need to mix in any off-chain data

Example requirements

DEX analytics on Polygon

Suppose you want to train a prototype ML model on all trades done on Uniswap Polygon since the v3 upgrade.

A delay of a few hours typically won’t matter for training, so you may want to deliver the data as files for easier handling.
The output could be a simple list of swaps, listing pair, direction and token amounts for each.
Polygon is an EVM chain.
All the required data is contained within Swap events emitted by the pair pool contracts. Uniswap deploys these dynamically, so you will also have to capture PoolCreated events from the factory contract to know which Swap events are coming from Uniswap and map them to pairs.
No off-chain data will be necessary for this task.

NFT ownership on Ethereum

Suppose you want to make a website that shows the image and ownership history for ERC721 NFTs from a certain Ethereum contract.

For this application it makes sense to deliver a GraphQL API.
Output data might have Token, Owner and Transfer database tables / entities, with e.g. Token supplying all the fields necessary to show ownership history and the image.
Ethereum is an EVM chain.
Data on token mints and ownership history can be derived from Transfer(address,address,uint256) EVM event logs emitted by the contract. To render images, you will also need token metadata URLs that are only available by querying the contract state with the tokenURI(uint256) function.
You’ll need to retrieve the off-chain token metadata (usually from IPFS).

Start from a template

Although it is possible to compose a squid from individual packages, in practice it is usually easier to start from a template.

Templates for the PostgreSQL+GraphQL data destination

A minimal template intended for developing EVM squids. Indexes ETH burns.
sqd init my-squid-name -t evm

A starter squid for indexing ERC20 transfers.

sqd init my-squid-name -t https://github.com/subsquid-labs/squid-erc20-template

Classic example Subgraph after a migration to SQD.
sqd init my-squid-name -t gravatar
A template showing how to combine data from multiple chains. Indexes USDC transfers on Ethereum and Binance.
sqd init my-squid-name -t multichain

Templates for storing data in files

USDC transfers -> local CSV

sqd init my-squid-name -t https://github.com/subsquid-labs/file-store-csv-example

USDC transfers -> local Parquet

sqd init my-squid-name -t https://github.com/subsquid-labs/file-store-parquet-example

USDC transfers -> CSV on S3

sqd init my-squid-name -t https://github.com/subsquid-labs/file-store-s3-example

Templates for the Google BigQuery data destination

USDC transfers -> BigQuery dataset

sqd init my-squid-name -t https://github.com/subsquid-labs/squid-bigquery-example

After retrieving the template, prepare it for a local run:

Install dependencies

cd my-squid-name
npm i

Test the template locally. The procedure varies depending on the data sink:

PostgreSQL+GraphQL
Filesystem dataset
BigQuery

Start PostgreSQL

docker compose up -d

Build the squid

npm run build

Apply the database migrations

npx squid-typeorm-migration apply

Start the squid processor

node -r dotenv/config lib/main.js

You should see output that contains lines like these:

04:11:24 INFO  sqd:batch-processor prometheus metrics are served on port 45829
04:11:27 INFO  sqd:batch-processor 6051219 / 18079056, rate: 16781 blocks/sec, mapping: 770 blocks/sec, 544 items/sec, eta: 12m

Start the GraphQL server

In a separate terminal, run:

npx squid-graphql-server

Visit the GraphiQL console to verify that the GraphQL API is running.

When done, shut down and erase your database with docker compose down.

Configure the S3 destination (optional)

For the S3 template only, set the credentials and prepare a bucket as described in the template README. Skip this step when writing to a local folder.

Build the squid

npm run build

Start the squid processor

node -r dotenv/config lib/main.js

The output should contain lines like these:

04:11:24 INFO  sqd:batch-processor prometheus metrics are served on port 45829
04:11:27 INFO  sqd:batch-processor 6051219 / 18079056, rate: 16781 blocks/sec, mapping: 770 blocks/sec, 544 items/sec, eta: 12m

After the processor starts writing data, a local destination should contain a ./data directory like this:

./data/
├── 0000000000-0007242369
│   └── transfers.tsv
├── 0007242370-0007638609
│   └── transfers.tsv
...
└── status.txt

The bottom-up development cycle

The advantage of this approach is that the code remains buildable at all times, making it easier to catch issues early.

Regenerate task-specific utilities

Generate contract or runtime utilities for the data you need. See the detailed guidance.

Configure data requests

Choose the data source, filters, related data, and fields. See the detailed guidance.

Decode and normalize the data

Turn each batch into the records your application needs. See the detailed guidance.

Enrich the data when needed

Optionally add external data or direct chain state queries. See the detailed guidance.

Prepare the store

Define the destination schema, tables, and migrations. See the detailed guidance.

Persist transformed data

Write each processed batch efficiently to the selected destination. See the detailed guidance.

Regenerate the task-specific utilities

Retrieve JSON ABIs for all contracts of interest (e.g. from Etherscan), taking care to get ABIs for implementation contracts and not proxies where appropriate. Assuming that you saved the ABI files to ./abi, you can then regenerate the utilities with

npx squid-evm-typegen ./src/abi ./abi/*.json --multicall

Or if you would like the tool to retrieve the ABI from Etherscan in your stead, you can run e.g.

npx squid-evm-typegen \
  src/abi \
  0xdAC17F958D2ee523a2206206994597C13D831ec7#usdt

The utility classes will become available at src/abi. See also Generating utility modules.

Configure the data requests

Data requests are defined with a DataSourceBuilder from @subsquid/evm-stream, customarily at src/main.ts. Edit the builder configuration to

Use a data source appropriate for your chain and task.
- Point the builder at the Portal dataset of your network with setPortal(). Portal access is public and the stream covers real-time data all the way to the unfinalized chain head - no API key or RPC endpoint required.
- For networks without a Portal dataset, use the EVM RPC data source instead.
Request all event logs, transactions, execution traces and state diffs that your task requires, with any necessary related data (e.g. parent transactions for event logs).
Select all data fields necessary for your task (e.g. gasUsed for transactions). There are no default optional fields - list every field your handler reads.

See the EVM Portal stream reference for more info and the examples page for a representative set of examples.

Decode and normalize the data

Next, change the batch handler to decode and normalize your data. In templates, the batch handler is defined at the run() call in src/main.ts as an inline function. Its sole argument ctx contains:

at ctx.blocks: all the requested data for a batch of blocks
at ctx.store: the means to save the processed data
at ctx.isHead: a boolean indicating whether the batch is at the current chain head

This structure (reference) is common for all Portal-native processors; the structure of ctx.blocks items varies. Note that there is no ctx.log - when your handler needs a Logger, create one explicitly with createLogger() from @subsquid/logger. Each item in ctx.blocks contains the data for the requested logs, transactions, traces and state diffs for a particular block, plus some info on the block itself. See the EVM batch context reference. Use the .decode methods from the contract ABI utilities to decode events and transactions, e.g.

import {run} from '@subsquid/batch-processor'
import * as erc20abi from './abi/erc20'

run(dataSource, db, async ctx => {
  for (let block of ctx.blocks) {
    for (let log of block.logs) {
      if (log.topics[0]===erc20abi.events.Transfer.topic) {
        let {from, to, value} = erc20abi.events.Transfer.decode(log)
      }
    }
  }
})

Mix in external data and chain state call output (optional)

If you need external (i.e. non-blockchain) data in your transformation, take a look at the External APIs and IPFS page. If any of the onchain data you need is unavailable from the processor or inconvenient to retrieve with it, you can use direct chain queries.

Prepare the store

At src/main.ts, change the Database object definition to accept your output data. The methods for saving data will be exposed by ctx.store within the batch handler.

PostgreSQL+GraphQL
Filesystem dataset
BigQuery

Define the schema of the database (and the core schema of the OpenReader GraphQL API if it is used) at schema.graphql.
Regenerate the TypeORM model classes with
npx squid-typeorm-codegen
The classes will become available at src/model.
Compile the models code with
npm run build
Ensure that the squid has access to a blank database. The easiest way to do so is to start PostgreSQL in a Docker container with
docker compose up -d
If the container is running, stop it and erase the database with
docker compose down
before issuing a docker compose up -d. The alternative is to connect to an external database. See this section to learn how to specify the connection parameters.

Regenerate a migration with

rm -r db/migrations

npx squid-typeorm-migration generate

You can now use the async functions ctx.store.upsert() and ctx.store.insert(), as well as various TypeORM lookup methods to access the database.See the Writing to PostgreSQL guide and the typeorm-store reference for more info.

Filesystem dataset writing, as performed by the @subsquid/file-store package and its extensions, stores the data into one or more flat tables. The exact table definition format depends on the output file format.

Decide on the file format you’re going to use:
- Parquet
- CSV
- JSON/JSONL
If your template does not have any of the necessary packages, install them.

Define any tables you need at the tables field of the Database constructor argument:

import { Database } from '@subsquid/file-store'

const dbOptions = {
  tables: {
    FirstTable: new Table(/* ... */),
    SecondTable: new Table(/* ... */),
    // ...
  },
  // ...
}

run(dataSource, new Database(dbOptions), async ctx => { // ...

Define the destination filesystem via the dest field of the Database constructor argument. Options:
- local folder - use LocalDest from @subsquid/file-store
- S3-compatible file storage service - install @subsquid/file-store-s3 and use S3Dest

Once you’re done you’ll be able to enqueue data rows for saving using the write() and writeMany() methods of the context store-provided table objects:

ctx.store.FirstTable.writeMany(/* ... */)
ctx.store.SecondTable.write(/* ... */)

The store will write the files automatically as soon as the buffer reaches the size set by the chunkSizeMb field of the Database constructor argument, or at the end of the batch if a call to setForceFlush() was made anywhere in the batch handler.See the filesystem datasets guide and the file-store reference.

Persist the transformed data to your data sink

Once your data is decoded, optionally enriched with external data and transformed the way you need it to be, it is time to save it.

PostgreSQL+GraphQL
Filesystem dataset
BigQuery

For each batch, create all the instances of all TypeORM model classes at once, then save them with the minimal number of calls to upsert() or insert(), e.g.:

import { EntityA, EntityB } from './model'

run(dataSource, new TypeormDatabase({supportHotBlocks: true}), async ctx => {
  const aEntities: Map<string, EntityA> = new Map() // id -> entity instance
  const bEntities: EntityB[] = []

  for (let block of ctx.blocks) {
    // fill aEntities and bEntities
  }

  await ctx.store.upsert([...aEntities.values()])
  await ctx.store.insert(bEntities)
})

It will often make sense to keep the entity instances in maps rather than arrays to make it easier to reuse them when defining instances of other entities with relations to the previous ones. The process is described in more detail in step 2 of the BAYC tutorial.If you perform any database lookups, try to do so in batches and make sure that the entity fields that you’re searching over are indexed.See also the patterns and anti-patterns sections of the Batch processing guide.

You can enqueue the transformed data for writing whenever convenient without any sizeable impact on performance.At low output data rates (e.g. if your entire dataset is in tens of Mbytes or under) take care to call ctx.store.setForceFlush() when appropriate to make sure your data actually gets written.

The top-down development cycle

The bottom-up development cycle described above is convenient for initial squid development and for trying out new things, but it has the disadvantage of not having the means of saving the data ready at hand when initially writing the data decoding/transformation code. That makes it necessary to come back to that code later, which is somewhat inconvenient, for example when adding new squid features incrementally. The alternative is to do the same steps in a different order:

Update the store

Define the destination schema, tables, and migrations before changing the processor.

Regenerate utility classes

Regenerate the task-specific utilities if the new feature requires them.

Update the processor configuration

Request the additional data and fields.

Decode and normalize the added data

Transform the new batch items into the required records.

Retrieve external data when needed

Optionally enrich the records with external data or chain state.

Add the persistence code

Write the transformed records to the prepared store.

GraphQL options

Store your data to PostgreSQL, then consult Serving GraphQL for options.

Scaling up

If you’re developing a large squid, make sure to use batch processing throughout your code. A common mistake is to make handlers for individual event logs or transactions; for updates that require data retrieval that results in lots of small database lookups and ultimately in poor syncing performance. Collect all the relevant data and process it at once. A simple architecture of that type is discussed in the BAYC tutorial. You should also check the Cloud best practices page even if you’re not planning to deploy to SQD Cloud - it contains valuable performance-related tips. Many issues commonly arising when developing larger squids are addressed by the third party @belopash/typeorm-store package. Consider using it. For complete examples of complex squids take a look at the Giant Squid Explorer and Thena Squid repos.

Next steps

Learn about batch processing.
Learn how squids deal with unfinalized blocks.
Use external APIs and IPFS in your squid.
See how squids should be set up for the multichain setting.
Deploy your squid on your own infrastructure or to SQD Cloud.

Overview

Pipes SDK

Squid SDK

Alternative clients

Prepare the environment

Understand your technical requirements

Example requirements

Start from a template

The bottom-up development cycle

Regenerate the task-specific utilities

Configure the data requests

Decode and normalize the data

Mix in external data and chain state call output (optional)

Prepare the store

Persist the transformed data to your data sink

The top-down development cycle

GraphQL options

Scaling up

Next steps

​Prepare the environment

​Understand your technical requirements

​Example requirements

​Start from a template

​The bottom-up development cycle

​Regenerate the task-specific utilities

​Configure the data requests

​Decode and normalize the data

​Mix in external data and chain state call output (optional)

​Prepare the store

​Persist the transformed data to your data sink

​The top-down development cycle

​GraphQL options

​Scaling up

​Next steps

Prepare the environment

Understand your technical requirements

Example requirements

Start from a template

The bottom-up development cycle

Regenerate the task-specific utilities

Configure the data requests

Decode and normalize the data

Mix in external data and chain state call output (optional)

Prepare the store

Persist the transformed data to your data sink

The top-down development cycle

GraphQL options

Scaling up

Next steps