Field selection

`setFields(options)`

Set the fields to be retrieved for data items of each supported type. The options object has the following structure:

{
  log?:         // field selector for logs
  transaction?: // field selector for transactions
  stateDiff?:   // field selector for state diffs
  trace?:       // field selector for traces
  block?:       // field selector for block headers
}

Every field selector is a collection of boolean fields, typically (with a notable exception of trace field selectors) mapping one-to-one to the fields of data items within the batch context iterables. Defining a field of a field selector of a given type and setting it to true will cause the data source to populate the corresponding field of all data items of that type. Here is a definition of a data source that requests gas and value fields for transactions:

const builder = new DataSourceBuilder()
  .setFields({
    transaction: {
      gas: true,
      value: true
    }
  })

There are no default optional fields: apart from a small set of always-present fields (listed per item type below), a field that is not requested with setFields() will not be fetched and will be absent from the TypeScript types. Request every field your batch handler reads.

Same fields will be available for all data items of any given type, including the items accessed via nested references. Suppose we used the builder defined above to subscribe to some transactions as well as some logs, and for each log we requested a parent transaction:

const dataSource = builder
  .addLog({
    where: {
      // some log filters
    },
    include: {
      transaction: true
    }
  })
  .addTransaction({
    where: {
      // some transaction filters
    }
  })
  .build()

After populating the convenience reference fields with augmentBlock() from @subsquid/evm-objects, gas and value fields would be available both within the transaction items of the transactions iterable of block data and within the transaction items that provide parent transaction information for the logs:

run(dataSource, database, async ctx => {
  let blocks = ctx.blocks.map(augmentBlock)
  for (let block of blocks) {
    for (let txn of block.transactions) {
      let txnGas = txn.gas // OK
    }
    for (let log of block.logs) {
      let parentTxnGas = log.getTransaction().gas // also OK!
    }
  }
})

Fields that were not requested are missing from the data and from the item types, so reading them fails to compile:

const dataSource = new DataSourceBuilder()
  .setFields({
    transaction: {
      value: true
    }
  })
  .addTransaction({
    where: {
      // some transaction filters
    }
  })
  .build()

run(dataSource, database, async ctx => {
  for (let block of ctx.blocks) {
    for (let txn of block.transactions) {
      let txnHash = txn.hash // ERROR: no such field
    }
  }
})

Requesting only the fields you need improves sync performance, as unrequested fields are not fetched from the SQD Network Portal.

Data item types and field selectors

Most IDEs support smart suggestions to show the possible field selectors. For VS Code, press Ctrl+Space.

Here we describe the data item types as functions of the field selectors. Unless otherwise mentioned, each data item type field maps to the eponymous field of its corresponding field selector. Item fields are divided into two categories:

Fields that are always present regardless of the setFields() call.
Fields that must be requested with setFields(). E.g. a transactionHash field will only be available in logs if the log field selector sets transactionHash: true.

The raw items do not have id, block or relation fields such as log.transaction — these are added by augmentBlock() from @subsquid/evm-objects, see Block data.

Logs

Log data items may have the following fields:

Log {
  // always present
  logIndex: number
  transactionIndex: number

  // can be requested with field selectors
  transactionHash: string
  address: string
  data: string
  topics: string[]
}

Transactions

Transaction data items may have the following fields:

Transaction {
  // always present
  transactionIndex: number

  // can be requested with field selectors
  hash: string
  from: string
  to?: string
  gas: bigint
  gasPrice: bigint
  maxFeePerGas?: bigint
  maxPriorityFeePerGas?: bigint
  input: string
  nonce: number
  value: bigint
  v: bigint
  r: string
  s: string
  yParity?: number
  chainId?: number
  authorizationList?: Array<{
    chainId: number
    nonce: number
    address: string
    yParity: number
    r: string
    s: string
  }>
  gasUsed: bigint
  cumulativeGasUsed: bigint
  effectiveGasPrice: bigint
  contractAddress?: string
  type: number
  status: number
  sighash?: string
  // limited availability (see below)
  l1Fee?: bigint
  l1FeeScalar?: number
  l1GasPrice?: bigint
  l1GasUsed?: bigint
  l1BlobBaseFee?: bigint
  l1BlobBaseFeeScalar?: number
  l1BaseFeeScalar?: number
}

status field contains the value returned by eth_getTransactionReceipt: 1 for successful transactions, 0 for failed ones and undefined for chains and block ranges not compliant with the post-Byzantinum hard fork EVM specification (e.g. 0-4,369,999 on Ethereum). type field is populated similarly. For example, on Ethereum 0 is returned for Legacy txs, 1 for EIP-2930 and 2 for EIP-1559. Other networks may have a different set of types. authorizationList is populated for EIP-7702 transactions (type 4). l1* fields can only be requested for networks from this list. Requesting them for other networks may cause HTTP 500 responses.

State diffs

StateDiff data items may have the following fields:

StateDiff {
  // always present
  transactionIndex: number
  address: string
  key: 'balance' | 'code' | 'nonce' | string

  // can be requested with field selectors
  kind: '=' | '+' | '*' | '-'
  prev?: string | null
  next?: string | null
}

The meaning of the kind field values is as follows:

'=': no change has occurred;
'+': a value was added;
'*': a value was changed;
'-': a value was removed.

Requesting the kind field also narrows the item type into a discriminated union: e.g. for a diff with kind: '+' the compiler knows that prev is empty and next holds the added value. The values of the key field are regular hexadecimal contract storage key strings or one of the special keys 'balance' | 'code' | 'nonce' denoting ETH balance, contract code and nonce value associated with the state diff.

Traces

Field selection for trace data items is somewhat more involved because their action and result fields may contain different subfields depending on the value of the type field. The retrieval of each one of these subfields is configured independently, with a selector name obtained by prefixing the capitalized subfield name with the trace type (and Result for result subfields). For example, to ensure that all traces of 'call' type contain the .action.gas field, the data source must be configured as follows:

dataSource.setFields({
  trace: {
    callGas: true
  }
})

The full Trace type with all its possible (sub)fields looks like this:

Trace {
  // always present
  transactionIndex: number
  traceAddress: number[]
  type: 'create' | 'call' | 'suicide' | 'reward'

  // can be requested with field selectors
  error: string | null      // error: true
  revertReason?: string     // revertReason: true
  subtraces: number         // subtraces: true
  // if (type==='create')
  action: {
                  // request the subfields with
    from: string  // createFrom: true
    value: bigint // createValue: true
    gas: bigint   // createGas: true
    init: string  // createInit: true
  }
  result?: {
    gasUsed: bigint  // createResultGasUsed: true
    code: string     // createResultCode: true
    address: string  // createResultAddress: true
  }
  // if (type==='call')
  action: {
    callType: string // callCallType: true
    from: string     // callFrom: true
    to: string       // callTo: true
    value?: bigint   // callValue: true
    gas: bigint      // callGas: true
    input: string    // callInput: true
    sighash?: string // callSighash: true
  }
  result?: {
    gasUsed: bigint // callResultGasUsed: true
    output: string  // callResultOutput: true
  }
  // if (type==='suicide')
  action: {
    address: string        // suicideAddress: true
    refundAddress: string  // suicideRefundAddress: true
    balance: bigint        // suicideBalance: true
  }
  // if (type==='reward')
  action: {
    author: string // rewardAuthor: true
    value: bigint  // rewardValue: true
    type: string   // rewardType: true
  }
}

The action and result objects are omitted from the item type when none of their subfields are requested. After augmentBlock() every trace also exposes getTransaction(), getParent() and children. The corresponding transaction and parent properties are optional because related data must be requested explicitly — see addTrace().

Block headers

BlockHeader data items may have the following fields:

BlockHeader {
  // always present
  number: number
  height: number // deprecated alias of `number`
  hash: string
  parentHash: string

  // can be requested with field selectors
  timestamp: number // Unix milliseconds
  nonce: string
  sha3Uncles: string
  logsBloom: string
  transactionsRoot: string
  stateRoot: string
  receiptsRoot: string
  mixHash: string
  miner: string
  difficulty: bigint
  totalDifficulty: bigint
  extraData: string
  size: bigint
  gasLimit: bigint
  gasUsed: bigint
  baseFeePerGas: bigint
  // limited availability (see below)
  l1BlockNumber: number
}

Use number for the block height; height remains available only as a deprecated alias. The stream converts the raw block time from seconds to milliseconds. Pass timestamp directly to new Date(timestamp) without multiplying it by 1000. The l1BlockNumber field can only be requested for networks from this list. Requesting it for other networks may cause HTTP 500 responses.

A complete example

Complete example

import {run} from '@subsquid/batch-processor'
import {DataSourceBuilder} from '@subsquid/evm-stream'
import {createLogger} from '@subsquid/logger'
import {TypeormDatabase} from '@subsquid/typeorm-store'
import * as gravatarAbi from './abi/gravatar'
import * as erc721abi from './abi/erc721'

const gravatarRegistryContract = '0x2e645469f354bb4f5c8a05b3b30a929361cf77ec'
const gravatarTokenContract = '0xac5c7493036de60e63eb81c5e9a440b42f47ebf5'

const dataSource = new DataSourceBuilder()
  .setPortal('https://portal.sqd.dev/datasets/ethereum-mainnet')
  .setBlockRange({from: 6_000_000})
  .addLog({
    where: {
      address: [
        gravatarRegistryContract
      ],
      topic0: [
        gravatarAbi.events.NewGravatar.topic,
        gravatarAbi.events.UpdatedGravatar.topic,
      ]
    }
  })
  .addTransaction({
    where: {
      to: [
        gravatarTokenContract
      ],
      sighash: [
        erc721abi.functions.setApprovalForAll.sighash
      ]
    },
    range: {from: 15_500_000}
  })
  .setFields({
    log: {
      topics: true,
      data: true,
    },
    transaction: {
      from: true,
      input: true,
      to: true
    }
  })
  .build()

const log = createLogger('sqd:processor:gravatar')

run(dataSource, new TypeormDatabase({supportHotBlocks: true}), async (ctx) => {
  // Simply output all the items in the batch.
  // It is guaranteed to have all the data matching the data requests,
  // but not guaranteed to not have any other data.
  log.info(ctx.blocks, 'Got blocks')
})

Overview

Pipes SDK

Squid SDK

Alternative clients

`setFields(options)`

Data item types and field selectors

Logs

Transactions

State diffs

Traces

Block headers

A complete example

​setFields(options)

​Data item types and field selectors

​Logs

​Transactions

​State diffs

​Traces

​Block headers

​A complete example

`setFields(options)`

Data item types and field selectors

Logs

Transactions

State diffs

Traces

Block headers

A complete example