Processor in action

An end-to-end idiomatic squid built on the batch processor can be inspected in the gravatar template repository and also learned from more elaborate examples. In order to illustrate the concepts covered in the development guide, here we highlight the key steps, put together a data source configuration and a data handling definition.

Pre-requisites: NodeJS, Git, Docker, Squid CLI, any of the EVM templates.

1. Model the target schema and generate entity classes

Create or edit schema.graphql to define the target entities and relations. Consult the schema reference. Update the entity classes, start a fresh database and regenerate migrations:

npx squid-typeorm-codegen

docker compose down

docker compose up -d

rm -r db/migrations

npx squid-typeorm-migration generate

Apply the migrations with

npx squid-typeorm-migration apply

2. Generate Typescript ABI modules

Use evm-typegen to generate the facade classes, for example like this:

npx squid-evm-typegen src/abi 0x2E645469f354BB4F5c8a05B3b30A929361cf77eC#Gravity --clean

3. Configuration

See the EVM Portal stream reference for more details.

import {DataSourceBuilder} from '@subsquid/evm-stream'
import {events} from './abi/Gravity'

const GRAVITY_ADDRESS = '0x2e645469f354bb4f5c8a05b3b30a929361cf77ec'

const dataSource = new DataSourceBuilder()
  .setPortal('https://portal.sqd.dev/datasets/ethereum-mainnet')
  .setBlockRange({ from: 6_175_243 })
  // there are no default fields -
  // list everything the handler reads
  .setFields({
    log: {
      topics: true,
      data: true
    }
  })
  // fetch logs emitted by the Gravity contract
  // matching either `NewGravatar` or `UpdatedGravatar`
  .addLog({
    where: {
      address: [GRAVITY_ADDRESS],
      topic0: [
        events.NewGravatar.topic,
        events.UpdatedGravatar.topic,
      ]
    }
  })
  .build()

4. Iterate over the batch items and group events

The following code snippet illustrates a typical data transformation in a batch. The strategy is to

Augment the raw blocks with augmentBlock() to get item IDs and navigation helpers
Iterate over the blocks and their logs
Decode each log using a suitable facade class
Enrich and transform the data
Upsert arrays of entities in batches using ctx.store.save()

The run() call then looks as follows:

import {run} from '@subsquid/batch-processor'
import {augmentBlock} from '@subsquid/evm-objects'
import {TypeormDatabase} from '@subsquid/typeorm-store'
import {Gravatar} from './model'

run(dataSource, new TypeormDatabase({supportHotBlocks: true}), async (ctx) => {
  // storing the new/updated entities in
  // an in-memory identity map
  const gravatars: Map<string, Gravatar> = new Map()
  // iterate over the data batch stored in ctx.blocks
  const blocks = ctx.blocks.map(augmentBlock)
  for (const block of blocks) {
    for (const log of block.logs) {
      // decode the log data
      const { id, owner, displayName, imageUrl } = extractData(log)
      // transform and normalize to match the target entity (Gravatar)
      const gravatarId = '0x' + id.toString(16)
      gravatars.set(gravatarId, new Gravatar({
        id: gravatarId,
        owner,
        displayName,
        imageUrl
      }))
    }
  }
  // Upsert the entities that were updated.
  // Note that store.save() automatically updates
  // the existing entities and creates new ones.
  // It splits the data into suitable chunks to
  // guarantee an adequate performance.
  await ctx.store.save([...gravatars.values()])
})

In the snippet above, we decode both NewGravatar and UpdatedGravatar with a single helper function that uses the generated events facade module. Decoder output types follow the ABI: the uint256 gravatar id comes out as a bigint (which we then format as a hex string to use as the entity ID) and the owner address as a hex string:

function extractData(log: {topics: string[], data: string}): {
  id: bigint,
  owner: string,
  displayName: string,
  imageUrl: string
} {
  if (log.topics[0] === events.NewGravatar.topic) {
    return events.NewGravatar.decode(log)
  }
  if (log.topics[0] === events.UpdatedGravatar.topic) {
    return events.UpdatedGravatar.decode(log)
  }
  throw new Error('Unsupported topic')
}

5. Run the processor and store the transformed data into the target database

Build the code, then run the processor:

npm run build
node -r dotenv/config lib/main.js

In a separate terminal window, run

npx squid-graphql-server

Inspect the GraphQL API at http://localhost:4350/graphql.

Overview

Pipes SDK

Squid SDK

Alternative clients

1. Model the target schema and generate entity classes

2. Generate Typescript ABI modules

3. Configuration

4. Iterate over the batch items and group events

5. Run the processor and store the transformed data into the target database

​1. Model the target schema and generate entity classes

​2. Generate Typescript ABI modules

​3. Configuration

​4. Iterate over the batch items and group events

​5. Run the processor and store the transformed data into the target database

1. Model the target schema and generate entity classes

2. Generate Typescript ABI modules

3. Configuration

4. Iterate over the batch items and group events

5. Run the processor and store the transformed data into the target database