sqd-go

sqd-go is a community project developed by mev.tools. It is not maintained by the SQD core team. Report bugs and feedback in the sqd-go issue tracker.

sqd-go is a compiled Go indexer that streams EVM data from SQD Portal and writes decoded events directly into ClickHouse. You describe contracts and events in a config.yaml; sqd-go generates the tables, the decoders, and typed Go code. You only write Go yourself when you maintain derived state such as balances or positions.

Choose sqd-go when

You want decoded EVM events in ClickHouse without writing decoding or schema code
You derive state (balances, positions, counters) and want it computed in compiled Go
You run on your own infrastructure and want checkpoint-based resume and reorg recovery

For TypeScript pipelines, use the Pipes SDK. For raw HTTP streaming from any language, use the Portal API.

Architecture

sqd-go requests only the configured contracts and events from the Portal, decodes them, and inserts the results into ClickHouse MergeTree tables. With a custom processor compiled in (--state), it also derives state tables from the same block stream.

Quickstart

Install

sqd-go requires Go: the CLI is installed with go install, and code generation and stateful projects compile generated Go code. Docker is needed for the local ClickHouse stack.

curl -sSL https://raw.githubusercontent.com/subsquid-labs/sqd-go/main/install.sh | bash

The script installs the sqd-go binary into your Go bin directory (usually ~/go/bin) and warns if that directory is not on your PATH. Alternatively, build from source:

git clone https://github.com/subsquid-labs/sqd-go.git
cd sqd-go
go install .

sqd-go targets Go 1.26 (declared in its go.mod). As of August 2026, running a --state build with Go 1.27 fails while compiling the transitive dependency cockroachdb/swiss. Until that is fixed upstream, pin the toolchain with export GOTOOLCHAIN=go1.26.4.

Scaffold a project

Create a project from the built-in ERC-20 template:

sqd-go init template erc20 my-indexer

This writes five files: config.yaml, .env, compose.yml, custom_schema.go, and custom_processor.go. The generated config.yaml indexes USDC Transfer and Approval events on Ethereum mainnet:

config.yaml

name: my-indexer
ecosystem: evm
chains:
    - id: 1
      start_block: 0
      contracts:
        - name: ERC20
          address:
            - 0xA0b86991c6218b36c1d19D4a2e9Eb0cE3606eB48
          events:
            - event: Transfer(address indexed from, address indexed to, uint256 value)
            - event: Approval(address indexed owner, address indexed spender, uint256 value)

start_block: 0 means a full backfill from genesis. Set it to a recent block, or bound the range with the --start-block and --end-block flags as shown below.

Start ClickHouse

The scaffolded compose.yml runs a single ClickHouse server. sqd-go start expects ClickHouse to already be running:

cd my-indexer
docker compose up -d clickhouse

Connection settings live in the generated .env (CLICKHOUSE_HOST, CLICKHOUSE_NATIVE_PORT, CLICKHOUSE_USER, CLICKHOUSE_PASSWORD, CLICKHOUSE_DATABASE). The defaults (8123 HTTP, 9000 native, password sqd-clickhouse) work with the scaffolded Compose file; edit .env if those ports are taken on your machine.

Run the indexer

The ERC-20 template is a stateful project, so run it with --state, which compiles your processor in. Run it from the project directory (a .env file in the current working directory takes precedence over the project’s):

sqd-go start . --state --restart --start-block 20600000 --end-block 20601000

start runs code generation, connects to ClickHouse, and begins ingestion. --restart clears the project database and indexes from the configured or overridden start block. On this bounded range the run completes in seconds:

Chain 1: stats final | checkpoint: 20599999 | next: 20601001 | ... | total: 1001 blocks, 16238 events
Done.

During longer runs, sqd-go prints a stats line (throughput, events, checkpoint) and a profile line (fetch, parse, insert, custom time) every 10 seconds. For faster backfills, add --parallel-fetch to spread finalized ranges across parallel workers.

sqd-go dev manages the project’s Compose stack for the duration of a run, but does not support --state. Stateful projects must use sqd-go start <path> --state.

Query the data

The run above produces one table per configured event, a state history table and a live view from the template’s processor, and a sync_state bookkeeping table:

Table	Contents
`erc20_transfer_events`	Decoded `Transfer` events with metadata
`erc20_approval_events`	Decoded `Approval` events with metadata
`user_positions_log`	Full state history, one row per update
`user_positions_live`	Latest state per primary key (view)

Event tables contain the decoded event fields plus block_number, block_timestamp, transaction_index, and log_index. Address columns are stored as raw bytes (FixedString(20)), so use hex() to display them:

SELECT
    concat('0x', lower(hex(address))) AS address,
    total_in,
    total_out,
    transfer_count
FROM `my-indexer`.user_positions_live
ORDER BY total_in DESC
LIMIT 10

Custom state in Go

Derived state is defined by two files in the project directory. custom_schema.go declares entity shapes as Go structs; each <Entity>Schema struct becomes a ClickHouse-backed state store accessed as state.<Entity>:

custom_schema.go

// pk: Address
type UserPositionSchema struct {
    Address        common.Address // primary key
    TotalIn        uint256.Int
    TotalOut       uint256.Int
    TransferCount  uint64
    UpdatedAtBlock uint64    // set automatically by Save()
    UpdatedAt      time.Time // set automatically by Save()
}

custom_processor.go implements a Process function that is called once per block with the decoded events and updates state through Get/GetOrCreate and Save:

custom_processor.go

func Process(state *generated.State, block *generated.ParsedBlock) error {
    for event := range block.EventsIter() {
        transfer, ok := event.(*generated.ERC20Transfer)
        if !ok {
            continue
        }
        pos := state.UserPosition.GetOrCreate(transfer.To)
        pos.TotalIn.Add(&pos.TotalIn, &transfer.Value)
        pos.TransferCount++
        state.UserPosition.Save(pos, transfer.EventMeta)
    }
    return nil
}

The scaffolded template ships a complete, heavily commented version of both files. --state regenerates the project, compiles your processor into a fresh binary, and runs it. For the build mechanics of standalone projects, see the Go modules guide in the sqd-go wiki.

Networks and Portal access

Chains are selected by numeric chain id in config.yaml. Built-in Portal endpoints exist only for Ethereum mainnet (1) and Polygon (137); for any other network, set SQD_PORTAL_ENDPOINT to that network’s Portal dataset URL (from the EVM network list) with /stream appended:

SQD_PORTAL_ENDPOINT=https://portal.sqd.dev/datasets/base-mainnet/stream \
sqd-go start . --state --restart

With the current version, a chain id other than 1 or 137 falls back to the Polygon dataset unless SQD_PORTAL_ENDPOINT is set. Always set it explicitly when indexing other networks.

No API token is required for public Portal access. SQD_API_TOKEN sets a Portal API token when you have one.

CLI and configuration

Command	Purpose
`sqd-go init`	Scaffold a project interactively, from an ABI file, or from the ERC-20 template
`sqd-go codegen <path>`	Validate the configuration and generate SQL and Go code
`sqd-go start <path>`	Run codegen, connect to an existing ClickHouse, and index
`sqd-go dev <path>`	Run codegen, start the project’s Compose stack, index, and tear the stack down on exit
`sqd-go stop`	Stop the local Compose stack and remove its volumes

A run without --restart resumes from the durable checkpoint: it first deletes rows above the checkpoint from every table with a block_number column, which makes reprocessing after an interrupted run idempotent. --reindex-from <n> rewinds to a specific block instead. Runtime tuning uses SQD_* environment variables (SQD_PARALLEL_FETCHERS, SQD_PARALLEL_RPS, and others); ClickHouse credentials belong in .env. The CLI reference documents all commands, flags, and environment variables.

Memory and monitoring

Hot state lives in the Go heap with a fixed cache of 100,000 entries per entity. Evicted entries spill to a local Pebble-backed cold tier that is rebuilt from ClickHouse on each start; ClickHouse remains the single source of truth. The cold tier is on by default and can be disabled with --no-cold-cache. SQD_COLDCACHE_MB caps its block cache; opening the tier also allocates a fixed 256 MiB Bloom filter on the heap. See METRICS.md for memory tuning details. Set SQD_METRICS_CH=1 to write runtime metrics (block and event throughput, checkpoint lag, memory and garbage collection, goroutine and CPU usage) to the monitoring.indexer_metrics ClickHouse table. The sqd-go repository’s own Compose stack includes provisioned Grafana dashboards for these metrics.

Performance

The maintainers publish results for sqd-go in the open-indexer-benchmark repository, run in two configurations: default settings with plain --restart, and a tuned run (--parallel-fetch --no-replay with SQD_PARALLEL_FETCHERS=12 and SQD_PARALLEL_RPS=10):

Case (metric)	Default	Tuned
`case_1_lbtc_event_only` (time)	10m 0s (capped, 1,040,134 of 1,599,998 blocks)	3m 35s, complete
`case_2_lbtc_full` (time)	9m 45s, complete	4m 31s, complete
`erc20-transfer-events` (blocks/s)	2,860.7	7,528.1
`erc20-account-balances` (blocks/s)	3,644.4	7,185.7

The tuned configuration trades memory for throughput: peak RSS on the erc20-transfer-events case grows from 621 MB to about 2 GB. The benchmark repository lists results for other indexers under the same workloads, along with the runner implementations and conditions.

Limitations

EVM networks and ClickHouse only
No factory contract tracking: addresses are fixed in the configuration (an event can also be matched at any address), and contracts deployed dynamically by factories are not discovered
Custom derived state must be written in Go

Resources

Source code on GitHub
sqd-go wiki: CLI, config, custom schema and processor references
Open indexer benchmark

Overview

Pipes SDK

Squid SDK

Alternative clients

Choose sqd-go when

Architecture

Quickstart

Custom state in Go

Networks and Portal access

CLI and configuration

Memory and monitoring

Performance

Limitations

Resources

​Choose sqd-go when

​Architecture

​Quickstart

​Custom state in Go

​Networks and Portal access

​CLI and configuration

​Memory and monitoring

​Performance

​Limitations

​Resources

Choose sqd-go when

Architecture

Quickstart

Custom state in Go

Networks and Portal access

CLI and configuration

Memory and monitoring

Performance

Limitations

Resources