Build an Indexer from Scratch

This guide demonstrates how SDK packages can be combined into a working indexer (called a squid). This page goes through all the technical details to make the squid architecture easier to understand. If you want to get to a working indexer ASAP, bootstrap from a template.

Prerequisites

Before you begin, ensure you have:

Node.js 20.x or newer - Download here
Docker - Download here

Project: USDT Transfers API

In this tutorial, you’ll build an indexer that tracks USDT transfers on Ethereum, saves the data to PostgreSQL, and serves it as a GraphQL API.

Required packages

From the requirements above, you’ll need these packages:

@subsquid/evm-stream - for retrieving Ethereum data from a SQD Network Portal
@subsquid/batch-processor - for running the data source against a batch handler
@subsquid/evm-objects - for enriching the retrieved blocks with item IDs and navigation helpers
@subsquid/typeorm-store, @subsquid/typeorm-codegen, and @subsquid/typeorm-migration - for saving data to PostgreSQL

Optional packages

@subsquid/evm-typegen - for decoding Ethereum data and useful constants such as event topic0 values
@subsquid/evm-abi and @subsquid/evm-codec - runtime dependencies imported by the code generated by @subsquid/evm-typegen
@subsquid/graphql-server / OpenReader - for serving GraphQL API

Build the Indexer

Initialize the project

Create a new folder and initialize a Node.js project:

npm init

Create a .gitignore file:

.gitignore

node_modules
lib

Install dependencies

Install the required packages:

npm i dotenv typeorm @subsquid/evm-stream @subsquid/batch-processor @subsquid/evm-objects @subsquid/typeorm-store @subsquid/typeorm-migration @subsquid/graphql-server @subsquid/evm-abi @subsquid/evm-codec

Install development dependencies:

npm i typescript @subsquid/typeorm-codegen @subsquid/evm-typegen --save-dev

Configure TypeScript

Create a tsconfig.json file with minimal configuration:

tsconfig.json

{
  "compilerOptions": {
    "rootDir": "src",
    "outDir": "lib",
    "module": "commonjs",
    "target": "es2020",
    "esModuleInterop": true,
    "skipLibCheck": true,
    "experimentalDecorators": true,
    "emitDecoratorMetadata": true
  }
}

Define the data schema

Create a schema.graphql file to define your data model:

schema.graphql

type Transfer @entity {
  id: ID!
  from: String! @index
  to: String! @index
  value: BigInt!
}

Generate TypeORM classes from the schema:

npx squid-typeorm-codegen

The TypeORM classes are now available at src/model/index.ts.

Set up the database

Create environment variables in a .env file:

.env

DB_NAME=squid
DB_PORT=23798

Create a docker-compose.yaml file for PostgreSQL:

docker-compose.yaml

services:
  db:
    image: postgres:15
    environment:
      POSTGRES_DB: "${DB_NAME}"
      POSTGRES_PASSWORD: postgres
    ports:
      - "${DB_PORT}:5432"

Start the database:

docker compose up -d

Compile the TypeORM classes:

npx tsc

Generate and apply the database migration:

npx squid-typeorm-migration generate
npx squid-typeorm-migration apply

Generate ABI utilities

Create an abi folder for contract ABIs:

mkdir abi

Find the USDT contract ABI on Etherscan and save it to ./abi/usdt.json.Generate TypeScript utility classes from the ABI:

npx squid-evm-typegen src/abi ./abi/*

The utility classes are now available in src/abi/usdt/, with an index.ts entry point. Typegen v5 removes a stale src/abi/usdt.ts file from the older single-file layout.

Create the data source and the batch handler

Create src/main.ts with the following code blocks:

Imports

main.ts

import { run } from "@subsquid/batch-processor";
import { DataSourceBuilder } from "@subsquid/evm-stream";
import { augmentBlock } from "@subsquid/evm-objects";
import { TypeormDatabase } from "@subsquid/typeorm-store";
import * as usdtAbi from "./abi/usdt";
import { Transfer } from "./model";

Data source configuration

main.ts

const dataSource = new DataSourceBuilder()
  .setPortal("https://portal.sqd.dev/datasets/ethereum-mainnet")
  .setFields({
    log: {
      topics: true,
      data: true,
    },
  })
  .addLog({
    where: {
      address: ["0xdac17f958d2ee523a2206206994597c13d831ec7"],
      topic0: [usdtAbi.events.Transfer.topic],
    },
  })
  .build();

The public Portal needs no RPC endpoint of your own, and the stream covers everything from historical blocks to the real-time chain head. Note that field selection is explicit: there are no default optional fields, so we list every log field the handler reads (the decoder needs topics and data).

Database configuration

main.ts

const db = new TypeormDatabase({ supportHotBlocks: true });

With supportHotBlocks: true the store can consume unfinalized blocks and roll the data back automatically on chain reorganizations.

Batch handler

main.ts

run(dataSource, db, async (ctx) => {
  const blocks = ctx.blocks.map(augmentBlock);
  const transfers: Transfer[] = [];
  for (let block of blocks) {
    for (let log of block.logs) {
      let { from, to, value } = usdtAbi.events.Transfer.decode(log);
      transfers.push(
        new Transfer({
          id: log.id,
          from,
          to,
          value,
        })
      );
    }
  }
  await ctx.store.insert(transfers);
});

augmentBlock() from @subsquid/evm-objects enriches the raw block items with unique IDs (used here as entity IDs) and navigation helpers — see Context interfaces.

Supplying a TypeormDatabase to run() causes ctx.store to be a PostgreSQL-compatible Store object.

Run the indexer

Compile the project:

npx tsc

Start the indexer:

node -r dotenv/config lib/main.js

The indexer is now processing USDT transfers from the Ethereum blockchain.

Start the GraphQL server

Update your .env file to include the GraphQL server port:

.env

DB_NAME=squid
DB_PORT=23798
GRAPHQL_SERVER_PORT=4350

In a separate terminal, start the GraphQL server:

npx squid-graphql-server

The GraphQL API with GraphiQL is available at localhost:4350/graphql.

Next Steps

Congratulations! You’ve built a complete blockchain indexer from scratch.

View Complete Code

See the final code for this tutorial

Development Flow

Learn about the general development workflow

The commands in this tutorial are often abbreviated as custom sqd commands in production squids. Use the commands.json file from the EVM template as a starter.

Overview

Pipes SDK

Squid SDK

Alternative clients

Build an Indexer from Scratch

Build an Indexer from Scratch

Prerequisites

Project: USDT Transfers API

Required packages

Optional packages

Build the Indexer

Imports

Data source configuration

Database configuration

Batch handler

Next Steps

View Complete Code

Development Flow

​Build an Indexer from Scratch

​Prerequisites

​Project: USDT Transfers API

​Required packages

​Optional packages

​Build the Indexer

​Imports

​Data source configuration

​Database configuration

​Batch handler

​Next Steps

View Complete Code

Development Flow

Build an Indexer from Scratch

Prerequisites

Project: USDT Transfers API

Required packages

Optional packages

Build the Indexer

Imports

Data source configuration

Database configuration

Batch handler

Next Steps