> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sqd.dev/llms.txt
> Use this file to discover all available pages before exploring further.
# Make an indexer
> Follow the Squid SDK workflow from data access and schema design to SQD Cloud deployment.
This page is a definitive end-to-end guide into practical squid development. It uses templates to simplify the process. Check out [Build an indexer from scratch](../examples-tutorials/build-indexer-from-scratch) for a more educational barebones approach.
Feel free to also use the template-specific `sqd` scripts defined in [`commands.json`](/en/cloud/reference/cli/commands-json) to simplify your workflow. See the [sqd CLI cheatsheet](./advanced/cli-cheatsheet) for a short intro.
## Prepare the environment
* Node.js 20 or newer
* Git
* [Squid CLI](/en/cloud/reference/cli/installation)
* Docker (if your squid will store its data to PostgreSQL)
See also the [Environment setup](./advanced/development-environment-setup) page.
## Understand your technical requirements
Consider your business requirements and find out
1. How the data should be delivered. Options:
* [PostgreSQL](./writing-to-postgres) with an optional [GraphQL API](./serving-graphql) - can be real-time
* [file-based dataset](./other-data-destinations#filesystem-datasets) - local or on S3
* [Google BigQuery](./other-data-destinations#bigquery)
2. What data should be delivered
3. Which Tron networks you'll be indexing - see the [supported networks](/en/data/all-networks) page
4. What exact data should be retrieved from blockchain(s)
5. Whether you need to mix in any [off-chain data](./advanced/external-apis-ipfs)
Start from a template
Although it is possible to [compose a squid from individual packages](../examples-tutorials/build-indexer-from-scratch), in practice it is usually easier to start from a template.
* A starter squid indexing Tron USDT transfers:
```bash theme={"system"}
sqd init my-squid-name -t https://github.com/subsquid-labs/tron-example
```
After retrieving the template, prepare it for a local run:
```bash theme={"system"}
cd my-squid-name
npm i
```
The Tron template requires an [SQD Network data API key](/en/sdk/migration/gateway-api-key). Add it to the template's existing `.env` file:
```bash title=".env" theme={"system"}
SQD_API_KEY=
```
The template passes the key to the processor's `.setGateway()` call.
The data API key is separate from the optional SQD Cloud deployment key used by `sqd auth`.
Test the template locally. The procedure varies depending on the data sink:
```bash theme={"system"}
docker compose up -d
```
```bash theme={"system"}
npm run build
```
```bash theme={"system"}
npx squid-typeorm-migration apply
```
```bash theme={"system"}
node -r dotenv/config lib/main.js
```
You should see output that contains lines like these:
```text theme={"system"}
04:11:24 INFO sqd:processor processing blocks from 6000000
04:11:24 INFO sqd:processor using archive data source
04:11:24 INFO sqd:processor prometheus metrics are served at port 45829
04:11:27 INFO sqd:processor 6051219 / 18079056, rate: 16781 blocks/sec, mapping: 770 blocks/sec, 544 items/sec, eta: 12m
```
In a separate terminal, run:
```bash theme={"system"}
npx squid-graphql-server
```
Visit the [GraphiQL console](http://localhost:4350/graphql) to verify that the GraphQL API is running.
When done, shut down and erase your database with `docker compose down`.
The bottom-up development cycle
The advantage of this approach is that the code remains buildable at all times, making it easier to catch issues early.
Generate contract or runtime utilities for the data you need. [See the detailed guidance](#typegen).
Choose the data source, filters, related data, and fields. [See the detailed guidance](#processor-config).
Turn each batch into the records your application needs. [See the detailed guidance](#batch-handler-decoding).
Optionally add external data or direct chain state queries. [See the detailed guidance](#external-data).
Define the destination schema, tables, and migrations. [See the detailed guidance](#store).
Write each processed batch efficiently to the selected destination. [See the detailed guidance](#batch-handler-persistence).
Regenerate the task-specific utilities
Tron event logs are fully EVM-compatible, so decoding utilities are generated from contract ABIs with `squid-evm-typegen`, same as for EVM chains. See the [USDT transfers tutorial](../examples-tutorials/indexing-usdt-transfers) for a worked example.
Configure the data requests
Edit the processor definition in `src/main.ts` to request the logs, transactions and internal transactions your task requires, and to select the data fields you need. See the [Tron processor reference](../reference/tron-processor/general) for all available options.
Decode and normalize the data
Next, change the batch handler to decode and normalize your data.
In templates, the batch handler is defined at the [`processor.run()`](../reference/tron-processor/context-interfaces) call in `src/main.ts` as an inline function. Its sole argument `ctx` contains:
* at `ctx.blocks`: all the requested data for a batch of blocks
* at `ctx.store`: the means to save the processed data
* at `ctx.log`: a [`Logger`](../reference/logger)
* at `ctx.isHead`: a boolean indicating whether the batch is at the current chain head
This structure ([reference](../reference/tron-processor/context-interfaces)) is common for all processors; the structure of `ctx.blocks` items varies.
Each item in `ctx.blocks` contains the requested logs, transactions and internal transactions for a particular block, plus some info on the block itself. See the [Tron batch context reference](../reference/tron-processor/context-interfaces). Decode the EVM-compatible event logs with typegen-generated ABI utilities, as shown in the [USDT transfers tutorial](../examples-tutorials/indexing-usdt-transfers).
Mix in external data and chain state call output (optional)
If you need external (i.e. non-blockchain) data in your transformation, take a look at the [External APIs and IPFS](./advanced/external-apis-ipfs) page.
If any of the onchain data you need is unavailable from the processor or inconvenient to retrieve with it, you can query the network directly from within the batch handler using any suitable client library.
Prepare the store
At `src/main.ts`, change the [`Database`](../reference/data-stores/store-interface) object definition to accept your output data. The methods for saving data will be exposed by `ctx.store` within the batch handler.
1. Define the schema of the database (and the [core schema of the OpenReader GraphQL API](../reference/openreader/api/intro) if it is used) at [`schema.graphql`](../reference/schema-files/schema-files-codegen).
2. Regenerate the TypeORM model classes with
```bash theme={"system"}
npx squid-typeorm-codegen
```
The classes will become available at `src/model`.
3. Compile the models code with
```bash theme={"system"}
npm run build
```
4. Ensure that the squid has access to a blank database. The easiest way to do so is to start PostgreSQL in a Docker container with
```bash theme={"system"}
docker compose up -d
```
If the container is running, stop it and erase the database with
```bash theme={"system"}
docker compose down
```
before issuing a `docker compose up -d`.
The alternative is to connect to an external database. See [this section](../reference/data-stores/typeorm-store#database-connection-parameters) to learn how to specify the connection parameters.
5. Regenerate a migration with
```bash theme={"system"}
rm -r db/migrations
```
```bash theme={"system"}
npx squid-typeorm-migration generate
```
You can now use the async functions [`ctx.store.upsert()`](../reference/data-stores/typeorm-store#upsert) and [`ctx.store.insert()`](../reference/data-stores/typeorm-store#insert), as well as various [TypeORM lookup methods](../reference/data-stores/typeorm-store#typeorm-methods) to access the database.
See the [Writing to PostgreSQL guide](./writing-to-postgres) and the [`typeorm-store` reference](../reference/data-stores/typeorm-store) for more info.
Filesystem dataset writing, as performed by the `@subsquid/file-store` package and its extensions, stores the data into one or more flat tables. The exact table definition format depends on the output file format.
1. Decide on the file format you're going to use:
* [Parquet](../reference/data-stores/file-store/parquet)
* [CSV](../reference/data-stores/file-store/csv)
* [JSON/JSONL](../reference/data-stores/file-store/json)
If your template does not have any of the necessary packages, install them.
2. Define any tables you need at the `tables` field of the `Database` constructor argument:
```ts theme={"system"}
import { Database } from '@subsquid/file-store'
const dbOptions = {
tables: {
FirstTable: new Table(/* ... */),
SecondTable: new Table(/* ... */),
// ...
},
// ...
}
processor.run(new Database(dbOptions), async ctx => { // ...
```
3. Define the destination filesystem via the `dest` field of the `Database` constructor argument. Options:
* local folder - use `LocalDest` from `@subsquid/file-store`
* S3-compatible file storage service - install `@subsquid/file-store-s3` and use [`S3Dest`](../reference/data-stores/file-store/s3-dest)
Once you're done you'll be able to enqueue data rows for saving using the `write()` and `writeMany()` methods of the context store-provided table objects:
```ts theme={"system"}
ctx.store.FirstTable.writeMany(/* ... */)
ctx.store.SecondTable.write(/* ... */)
```
The store will write the files automatically as soon as the buffer reaches the size set by the `chunkSizeMb` field of the `Database` constructor argument, or at the end of the batch if a call to [`setForceFlush()`](./other-data-destinations#filesystem-datasets) was made anywhere in the batch handler.
See the [filesystem datasets guide](./other-data-destinations#filesystem-datasets) and the [`file-store` reference](../reference/data-stores/file-store).
Follow the [BigQuery guide](./other-data-destinations#bigquery).
Persist the transformed data to your data sink
Once your data is [decoded](#batch-handler-decoding), optionally [enriched with external data](#external-data) and transformed the way you need it to be, it is time to save it.
For each batch, create all the instances of all TypeORM model classes at once, then save them with the minimal number of calls to `upsert()` or `insert()`, e.g.:
```ts theme={"system"}
import { EntityA, EntityB } from './model'
processor.run(new TypeormDatabase(), async ctx => {
const aEntities: Map = new Map() // id -> entity instance
const bEntities: EntityB[] = []
for (let block of ctx.blocks) {
// fill aEntities and bEntities
}
await ctx.store.upsert([...aEntities.values()])
await ctx.store.insert(bEntities)
})
```
It will often make sense to keep the entity instances in maps rather than arrays to make it easier to reuse them when defining instances of other entities with [relations](../reference/schema-files/entity-relations) to the previous ones.
If you perform any [database lookups](../reference/data-stores/typeorm-store#typeorm-methods), try to do so in batches and make sure that the entity fields that you're searching over are [indexed](../reference/schema-files/indexes-constraints).
See also the [patterns](./advanced/batch-processing#implementation-patterns) and [anti-patterns](./advanced/batch-processing#anti-patterns-to-avoid) sections of the Batch processing guide.
You can enqueue the transformed data for writing whenever convenient without any sizeable impact on performance.
At low output data rates (e.g. if your entire dataset is in tens of Mbytes or under) take care to call [`ctx.store.setForceFlush()`](./other-data-destinations#filesystem-datasets) when appropriate to make sure your data actually gets written.
You can enqueue the transformed data for writing whenever convenient without any sizeable impact on performance. The actual data writing will happen automatically at the end of each batch.
## The top-down development cycle
The [bottom-up development cycle](#bottom-up-development) described above is convenient for initial squid development and for trying out new things, but it has the disadvantage of not having the means of saving the data ready at hand when initially writing the data decoding/transformation code. That makes it necessary to come back to that code later, which is somewhat inconvenient, for example when adding new squid features incrementally.
The alternative is to do the same steps in a different order:
[Define the destination schema, tables, and migrations](#store) before changing the processor.
[Regenerate the task-specific utilities](#typegen) if the new feature requires them.
[Request the additional data and fields](#processor-config).
[Transform the new batch items](#batch-handler-decoding) into the required records.
Optionally [enrich the records with external data or chain state](#external-data).
[Write the transformed records](#batch-handler-persistence) to the prepared store.
## GraphQL options
[Store your data to PostgreSQL](./writing-to-postgres), then consult [Serving GraphQL](./serving-graphql) for options.
## Scaling up
If you're developing a large squid, make sure to use [batch processing](./advanced/batch-processing) throughout your code.
A common mistake is to make handlers for individual event logs or transactions; for updates that require data retrieval that results in lots of small database lookups and ultimately in poor syncing performance. Collect all the relevant data and process it at once.
You should also check the [Cloud best practices page](/en/cloud/resources/best-practices) even if you're not planning to deploy to [SQD Cloud](/en/cloud/overview) - it contains valuable performance-related tips.
Many issues commonly arising when developing larger squids are addressed by the third party [`@belopash/typeorm-store` package](https://github.com/belopash/squid-typeorm-store). Consider using it.
For complete examples of complex squids take a look at the [Giant Squid Explorer](https://github.com/subsquid-labs/giant-squid-explorer) and [Thena Squid](https://github.com/subsquid-labs/thena-squid) repos.
## Next steps
* Learn about [batch processing](./advanced/batch-processing).
* Learn how squids deal with [unfinalized blocks](./advanced/unfinalized-blocks).
* [Use external APIs and IPFS](./advanced/external-apis-ipfs) in your squid.
* See how squids should be set up for the [multichain setting](./advanced/multichain-indexing).
* Deploy your squid [on your own infrastructure](/en/sdk/self-hosting) or to [SQD Cloud](/en/cloud/overview).