> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sqd.dev/llms.txt
> Use this file to discover all available pages before exploring further.
# Make an indexer
> Follow the Squid SDK workflow from data access and schema design to SQD Cloud deployment.
This page is a definitive end-to-end guide into practical squid development. It uses templates to simplify the process. Check out [Build an indexer from scratch](../examples-tutorials/build-indexer-from-scratch) for a more educational barebones approach.
Feel free to also use the template-specific `sqd` scripts defined in [`commands.json`](/en/cloud/reference/cli/commands-json) to simplify your workflow. See the [sqd CLI cheatsheet](./advanced/cli-cheatsheet) for a short intro.
## Prepare the environment
* Node.js 20 or newer
* Git
* [Squid CLI](/en/cloud/reference/cli/installation)
* Docker (if your squid will store its data to PostgreSQL)
See also the [Environment setup](./advanced/development-environment-setup) page.
## Understand your technical requirements
Consider your business requirements and find out
1. How the data should be delivered. Options:
* [PostgreSQL](./writing-to-postgres) with an optional [GraphQL API](./serving-graphql) - can be real-time
* [file-based dataset](./other-data-destinations#filesystem-datasets) - local or on S3
* [Google BigQuery](./other-data-destinations#bigquery)
2. What data should be delivered
3. Which Substrate networks you'll be indexing - see the [supported networks](/en/data/substrate) page
Note that you can use SQD via [RPC ingestion](./advanced/unfinalized-blocks) even if your network is not listed.
4. What exact data should be retrieved from blockchain(s)
5. Whether you need to mix in any [off-chain data](./advanced/external-apis-ipfs)
#### Example requirements
Suppose you want to create a BigQuery dataset with Kusama native tokens transfers.
1. The delivery format is BigQuery.
2. A single table with `from`, `to` and `amount` columns may suffice.
3. Kusama is a Substrate chain.
4. The required data is available from `Transfer` events emitted by the `Balances` pallet. Take a look at our [Substrate data sourcing miniguide](./substrate/data-sourcing-miniguide) for more info on how to figure out which pallets, events and calls are necessary for your task.
5. No off-chain data will be necessary for this task.
Start from a template
Although it is possible to [compose a squid from individual packages](../examples-tutorials/build-indexer-from-scratch), in practice it is usually easier to start from a template.
* Native events emitted by Substrate-based chains
```bash theme={"system"}
sqd init my-squid-name -t substrate
```
* ink! smart contracts
```bash theme={"system"}
sqd init my-squid-name -t ink
```
* Frontier EVM contracts on Astar and Moonbeam
```bash theme={"system"}
sqd init my-squid-name -t frontier-evm
```
After retrieving the template, prepare it for a local run:
```bash theme={"system"}
cd my-squid-name
npm i
```
The gateway-based Substrate templates above require an [SQD Network data API key](/en/sdk/migration/gateway-api-key). Add it to the template's existing `.env` file:
```bash title=".env" theme={"system"}
SQD_API_KEY=
```
`SubstrateBatchProcessor` reads `SQD_API_KEY` automatically when `.setGateway()` receives a URL.
The data API key is separate from the optional SQD Cloud deployment key used by `sqd auth`.
Test the template locally. The procedure varies depending on the data sink:
```bash theme={"system"}
docker compose up -d
```
```bash theme={"system"}
npm run build
```
```bash theme={"system"}
npx squid-typeorm-migration apply
```
```bash theme={"system"}
node -r dotenv/config lib/main.js
```
You should see output that contains lines like these:
```text theme={"system"}
04:11:24 INFO sqd:processor processing blocks from 6000000
04:11:24 INFO sqd:processor using archive data source
04:11:24 INFO sqd:processor prometheus metrics are served at port 45829
04:11:27 INFO sqd:processor 6051219 / 18079056, rate: 16781 blocks/sec, mapping: 770 blocks/sec, 544 items/sec, eta: 12m
```
In a separate terminal, run:
```bash theme={"system"}
npx squid-graphql-server
```
Visit the [GraphiQL console](http://localhost:4350/graphql) to verify that the GraphQL API is running.
When done, shut down and erase your database with `docker compose down`.
The bottom-up development cycle
The advantage of this approach is that the code remains buildable at all times, making it easier to catch issues early.
Generate contract or runtime utilities for the data you need. [See the detailed guidance](#typegen).
Choose the data source, filters, related data, and fields. [See the detailed guidance](#processor-config).
Turn each batch into the records your application needs. [See the detailed guidance](#batch-handler-decoding).
Optionally add external data or direct chain state queries. [See the detailed guidance](#external-data).
Define the destination schema, tables, and migrations. [See the detailed guidance](#store).
Write each processed batch efficiently to the selected destination. [See the detailed guidance](#batch-handler-persistence).
Regenerate the task-specific utilities
Configure the [Substrate typegen](../reference/substrate-typegen) at `typegen.json`, then regenerate the runtime-specific utilities with
```bash theme={"system"}
npx squid-substrate-typegen ./typegen.json
```
See the [Substrate typegen reference](../reference/substrate-typegen) for the full config format, and the [Substrate tutorial](../examples-tutorials/substrate-tutorial) for a worked example. If your squid indexes [ink! contracts](./substrate/ink), also generate the contract utilities with the [ink! typegen](../reference/substrate-typegen#ink-contracts).
These squids use both Substrate typegen *and* EVM typegen. To generate all the required utilities, [configure the Substrate part](../reference/substrate-typegen), then save all relevant JSON ABIs to `./abi`, then run
```bash theme={"system"}
npx squid-evm-typegen ./src/abi ./abi/*.json --multicall
```
followed by
```bash theme={"system"}
npx squid-substrate-typegen ./typegen.json
```
See the [Frontier EVM guide](./substrate/frontier-evm) for details.
Configure the data requests
Data requests are [customarily](../reference/project-structure) defined at `src/processor.ts`. Edit the definition of `const processor` to
1. Use a data source appropriate for your chain and task
* [Use](../reference/substrate-processor/general#set-gateway) a [SQD Network gateway](/en/data/substrate) whenever it is available. [RPC](../reference/substrate-processor/general#set-rpc-endpoint) is still required in this case.
* For networks without a gateway use just the RPC.
2. Request all [events](../reference/substrate-processor/data-requests#events) and [calls](../reference/substrate-processor/data-requests#calls) that your task requires, with any necessary related data (e.g. parent extrinsics).
3. If your squid indexes any of the following:
* an [ink! contract](./substrate/ink)
* an EVM contract running on the [Frontier EVM pallet](./substrate/frontier-evm)
* [Gear messages](./substrate/gear)
then you can use some of the [specialized data requesting methods](../reference/substrate-processor/data-requests#specialized-setters) to retrieve data more selectively.
4. [Select all data fields](../reference/substrate-processor/field-selection) necessary for your task (e.g. `fee` for extrinsics).
See the [Substrate processor reference](../reference/substrate-processor/general) for more info. Processor config examples can be found in the tutorials:
* [general Substrate](../examples-tutorials/substrate-tutorial)
* [ink!](../examples-tutorials/ink)
* [Frontier EVM](../examples-tutorials/frontier-evm)
Decode and normalize the data
Next, change the batch handler to decode and normalize your data.
In templates, the batch handler is defined at the [`processor.run()`](../reference/substrate-processor/architecture#processorrun) call in `src/main.ts` as an inline function. Its sole argument `ctx` contains:
* at `ctx.blocks`: all the requested data for a batch of blocks
* at `ctx.store`: the means to save the processed data
* at `ctx.log`: a [`Logger`](../reference/logger)
* at `ctx.isHead`: a boolean indicating whether the batch is at the current chain head
* at `ctx._chain`: the means to access RPC for [state calls](#external-data)
This structure ([reference](../reference/substrate-processor/architecture#batch-context)) is common for all processors; the structure of `ctx.blocks` items varies.
Each item in `ctx.blocks` contains the data for the requested events, calls and, if requested, any related extrinsics; it also has some info on the block itself. See the [Substrate batch context reference](../reference/substrate-processor/context-interfaces).
Use the `.is()` and `.decode()` functions to decode the data for each runtime version, e.g. like this:
```ts theme={"system"}
import {events} from './types'
processor.run(db, async ctx => {
for (let block of ctx.blocks) {
for (let event of block.events) {
if (event.name == events.balances.transfer.name) {
let rec: {from: string; to: string; amount: bigint}
if (events.balances.transfer.v1020.is(event)) {
let [from, to, amount] = events.balances.transfer.v1020.decode(event)
rec = {from, to, amount}
}
else if (events.balances.transfer.v1050.is(event)) {
let [from, to, amount] = events.balances.transfer.v1050.decode(event)
rec = {from, to, amount}
}
else if (events.balances.transfer.v9130.is(event)) {
rec = events.balances.transfer.v9130.decode(event)
}
else {
throw new Error('Unsupported spec')
}
}
}
}
})
```
You can also decode the data of certain pallet-specific events and transactions with specialized tools:
* use the utility classes made with the [ink! typegen](../reference/substrate-typegen#ink-contracts) to [decode events emitted by ink! contracts](./substrate/ink)
* use the [`@subsquid/frontier` utils](../reference/frontier) and the EVM typegen to decode event logs and transactions of [Frontier EVM contracts](./substrate/frontier-evm)
Mix in external data and chain state call output (optional)
If you need external (i.e. non-blockchain) data in your transformation, take a look at the [External APIs and IPFS](./advanced/external-apis-ipfs) page.
If any of the onchain data you need is unavailable from the processor or inconvenient to retrieve with it, you can use [direct storage queries](../reference/substrate-processor/architecture#storage-queries).
Prepare the store
At `src/main.ts`, change the [`Database`](../reference/data-stores/store-interface) object definition to accept your output data. The methods for saving data will be exposed by `ctx.store` within the batch handler.
1. Define the schema of the database (and the [core schema of the OpenReader GraphQL API](../reference/openreader/api/intro) if it is used) at [`schema.graphql`](../reference/schema-files/schema-files-codegen).
2. Regenerate the TypeORM model classes with
```bash theme={"system"}
npx squid-typeorm-codegen
```
The classes will become available at `src/model`.
3. Compile the models code with
```bash theme={"system"}
npm run build
```
4. Ensure that the squid has access to a blank database. The easiest way to do so is to start PostgreSQL in a Docker container with
```bash theme={"system"}
docker compose up -d
```
If the container is running, stop it and erase the database with
```bash theme={"system"}
docker compose down
```
before issuing a `docker compose up -d`.
The alternative is to connect to an external database. See [this section](../reference/data-stores/typeorm-store#database-connection-parameters) to learn how to specify the connection parameters.
5. Regenerate a migration with
```bash theme={"system"}
rm -r db/migrations
```
```bash theme={"system"}
npx squid-typeorm-migration generate
```
You can now use the async functions [`ctx.store.upsert()`](../reference/data-stores/typeorm-store#upsert) and [`ctx.store.insert()`](../reference/data-stores/typeorm-store#insert), as well as various [TypeORM lookup methods](../reference/data-stores/typeorm-store#typeorm-methods) to access the database.
See the [Writing to PostgreSQL guide](./writing-to-postgres) and the [`typeorm-store` reference](../reference/data-stores/typeorm-store) for more info.
Filesystem dataset writing, as performed by the `@subsquid/file-store` package and its extensions, stores the data into one or more flat tables. The exact table definition format depends on the output file format.
1. Decide on the file format you're going to use:
* [Parquet](../reference/data-stores/file-store/parquet)
* [CSV](../reference/data-stores/file-store/csv)
* [JSON/JSONL](../reference/data-stores/file-store/json)
If your template does not have any of the necessary packages, install them.
2. Define any tables you need at the `tables` field of the `Database` constructor argument:
```ts theme={"system"}
import { Database } from '@subsquid/file-store'
const dbOptions = {
tables: {
FirstTable: new Table(/* ... */),
SecondTable: new Table(/* ... */),
// ...
},
// ...
}
processor.run(new Database(dbOptions), async ctx => { // ...
```
3. Define the destination filesystem via the `dest` field of the `Database` constructor argument. Options:
* local folder - use `LocalDest` from `@subsquid/file-store`
* S3-compatible file storage service - install `@subsquid/file-store-s3` and use [`S3Dest`](../reference/data-stores/file-store/s3-dest)
Once you're done you'll be able to enqueue data rows for saving using the `write()` and `writeMany()` methods of the context store-provided table objects:
```ts theme={"system"}
ctx.store.FirstTable.writeMany(/* ... */)
ctx.store.SecondTable.write(/* ... */)
```
The store will write the files automatically as soon as the buffer reaches the size set by the `chunkSizeMb` field of the `Database` constructor argument, or at the end of the batch if a call to [`setForceFlush()`](./other-data-destinations#filesystem-datasets) was made anywhere in the batch handler.
See the [filesystem datasets guide](./other-data-destinations#filesystem-datasets) and the [`file-store` reference](../reference/data-stores/file-store).
Follow the [BigQuery guide](./other-data-destinations#bigquery).
Persist the transformed data to your data sink
Once your data is [decoded](#batch-handler-decoding), optionally [enriched with external data](#external-data) and transformed the way you need it to be, it is time to save it.
For each batch, create all the instances of all TypeORM model classes at once, then save them with the minimal number of calls to `upsert()` or `insert()`, e.g.:
```ts theme={"system"}
import { EntityA, EntityB } from './model'
processor.run(new TypeormDatabase(), async ctx => {
const aEntities: Map = new Map() // id -> entity instance
const bEntities: EntityB[] = []
for (let block of ctx.blocks) {
// fill aEntities and bEntities
}
await ctx.store.upsert([...aEntities.values()])
await ctx.store.insert(bEntities)
})
```
It will often make sense to keep the entity instances in maps rather than arrays to make it easier to reuse them when defining instances of other entities with [relations](../reference/schema-files/entity-relations) to the previous ones.
If you perform any [database lookups](../reference/data-stores/typeorm-store#typeorm-methods), try to do so in batches and make sure that the entity fields that you're searching over are [indexed](../reference/schema-files/indexes-constraints).
See also the [patterns](./advanced/batch-processing#implementation-patterns) and [anti-patterns](./advanced/batch-processing#anti-patterns-to-avoid) sections of the Batch processing guide.
You can enqueue the transformed data for writing whenever convenient without any sizeable impact on performance.
At low output data rates (e.g. if your entire dataset is in tens of Mbytes or under) take care to call [`ctx.store.setForceFlush()`](./other-data-destinations#filesystem-datasets) when appropriate to make sure your data actually gets written.
You can enqueue the transformed data for writing whenever convenient without any sizeable impact on performance. The actual data writing will happen automatically at the end of each batch.
## The top-down development cycle
The [bottom-up development cycle](#bottom-up-development) described above is convenient for initial squid development and for trying out new things, but it has the disadvantage of not having the means of saving the data ready at hand when initially writing the data decoding/transformation code. That makes it necessary to come back to that code later, which is somewhat inconvenient, for example when adding new squid features incrementally.
The alternative is to do the same steps in a different order:
[Define the destination schema, tables, and migrations](#store) before changing the processor.
[Regenerate the task-specific utilities](#typegen) if the new feature requires them.
[Request the additional data and fields](#processor-config).
[Transform the new batch items](#batch-handler-decoding) into the required records.
Optionally [enrich the records with external data or chain state](#external-data).
[Write the transformed records](#batch-handler-persistence) to the prepared store.
## GraphQL options
[Store your data to PostgreSQL](./writing-to-postgres), then consult [Serving GraphQL](./serving-graphql) for options.
## Scaling up
If you're developing a large squid, make sure to use [batch processing](./advanced/batch-processing) throughout your code.
A common mistake is to make handlers for individual event logs or transactions; for updates that require data retrieval that results in lots of small database lookups and ultimately in poor syncing performance. Collect all the relevant data and process it at once.
You should also check the [Cloud best practices page](/en/cloud/resources/best-practices) even if you're not planning to deploy to [SQD Cloud](/en/cloud/overview) - it contains valuable performance-related tips.
Many issues commonly arising when developing larger squids are addressed by the third party [`@belopash/typeorm-store` package](https://github.com/belopash/squid-typeorm-store). Consider using it.
For complete examples of complex squids take a look at the [Giant Squid Explorer](https://github.com/subsquid-labs/giant-squid-explorer) and [Thena Squid](https://github.com/subsquid-labs/thena-squid) repos.
## Next steps
* Learn about [batch processing](./advanced/batch-processing).
* Learn how squids deal with [unfinalized blocks](./advanced/unfinalized-blocks).
* [Use external APIs and IPFS](./advanced/external-apis-ipfs) in your squid.
* See how squids should be set up for the [multichain setting](./advanced/multichain-indexing).
* Deploy your squid [on your own infrastructure](/en/sdk/self-hosting) or to [SQD Cloud](/en/cloud/overview).