> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sqd.dev/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Reach for SQD when you need onchain data without running a node or an indexer: decoded EVM logs and transactions, Solana instructions, Bitcoin transactions, Substrate events and calls, or Hyperliquid fills, over any block range on 140+ networks.
> To query directly, POST to https://portal.sqd.dev/datasets/{dataset}/stream. The full API is described at https://docs.sqd.dev/openapi.json, and responses to the stream endpoints are JSON Lines.
> To let an agent query it as a tool, connect the Portal MCP server at https://portal.sqd.dev/mcp.
> Every page on this site is available as Markdown by appending .md to its URL.

# sqd-go

> Community-maintained Go indexer that writes decoded EVM events and state to ClickHouse.

<Info>
  `sqd-go` is a community project developed by [mev.tools](https://github.com/mev-tools). It is not maintained by the SQD core team. Report bugs and feedback in the [sqd-go issue tracker](https://github.com/subsquid-labs/sqd-go/issues).
</Info>

[`sqd-go`](https://github.com/subsquid-labs/sqd-go) is a compiled Go indexer that streams EVM data from [SQD Portal](/en/portal/overview) and writes decoded events directly into ClickHouse. You describe contracts and events in a `config.yaml`; `sqd-go` generates the tables, the decoders, and typed Go code. You only write Go yourself when you maintain derived state such as balances or positions.

## Choose sqd-go when

* You want decoded EVM events in ClickHouse without writing decoding or schema code
* You derive state (balances, positions, counters) and want it computed in compiled Go
* You run on your own infrastructure and want checkpoint-based resume and reorg recovery

For TypeScript pipelines, use the [Pipes SDK](/en/sdk/pipes-sdk/evm/quickstart). For raw HTTP streaming from any language, use the [Portal API](/en/portal/evm/quickstart).

## Architecture

```mermaid theme={"system"}
flowchart LR
    A["EVM blocks"] --> B["SQD Portal"]
    B --> C["sqd-go"]
    C --> D["Event tables"]
    C --> E["Custom processor"]
    E --> F["State tables"]
    D --> G[(ClickHouse)]
    F --> G
```

`sqd-go` requests only the configured contracts and events from the Portal, decodes them, and inserts the results into ClickHouse `MergeTree` tables. With a custom processor compiled in (`--state`), it also derives state tables from the same block stream.

## Quickstart

<Steps>
  <Step title="Install">
    `sqd-go` requires Go: the CLI is installed with `go install`, and code generation and stateful projects compile generated Go code. Docker is needed for the local ClickHouse stack.

    ```bash theme={"system"}
    curl -sSL https://raw.githubusercontent.com/subsquid-labs/sqd-go/main/install.sh | bash
    ```

    The script installs the `sqd-go` binary into your Go bin directory (usually `~/go/bin`) and warns if that directory is not on your `PATH`. Alternatively, build from source:

    ```bash theme={"system"}
    git clone https://github.com/subsquid-labs/sqd-go.git
    cd sqd-go
    go install .
    ```

    <Note>
      `sqd-go` targets Go 1.26 (declared in its `go.mod`). As of August 2026, running a `--state` build with Go 1.27 fails while compiling the transitive dependency `cockroachdb/swiss`. Until that is fixed upstream, pin the toolchain with `export GOTOOLCHAIN=go1.26.4`.
    </Note>
  </Step>

  <Step title="Scaffold a project">
    Create a project from the built-in ERC-20 template:

    ```bash theme={"system"}
    sqd-go init template erc20 my-indexer
    ```

    This writes five files: `config.yaml`, `.env`, `compose.yml`, `custom_schema.go`, and `custom_processor.go`. The generated `config.yaml` indexes USDC `Transfer` and `Approval` events on Ethereum mainnet:

    ```yaml config.yaml theme={"system"}
    name: my-indexer
    ecosystem: evm
    chains:
        - id: 1
          start_block: 0
          contracts:
            - name: ERC20
              address:
                - 0xA0b86991c6218b36c1d19D4a2e9Eb0cE3606eB48
              events:
                - event: Transfer(address indexed from, address indexed to, uint256 value)
                - event: Approval(address indexed owner, address indexed spender, uint256 value)
    ```

    `start_block: 0` means a full backfill from genesis. Set it to a recent block, or bound the range with the `--start-block` and `--end-block` flags as shown below.
  </Step>

  <Step title="Start ClickHouse">
    The scaffolded `compose.yml` runs a single ClickHouse server. `sqd-go start` expects ClickHouse to already be running:

    ```bash theme={"system"}
    cd my-indexer
    docker compose up -d clickhouse
    ```

    Connection settings live in the generated `.env` (`CLICKHOUSE_HOST`, `CLICKHOUSE_NATIVE_PORT`, `CLICKHOUSE_USER`, `CLICKHOUSE_PASSWORD`, `CLICKHOUSE_DATABASE`). The defaults (`8123` HTTP, `9000` native, password `sqd-clickhouse`) work with the scaffolded Compose file; edit `.env` if those ports are taken on your machine.
  </Step>

  <Step title="Run the indexer">
    The ERC-20 template is a stateful project, so run it with `--state`, which compiles your processor in. Run it from the project directory (a `.env` file in the current working directory takes precedence over the project's):

    ```bash theme={"system"}
    sqd-go start . --state --restart --start-block 20600000 --end-block 20601000
    ```

    `start` runs code generation, connects to ClickHouse, and begins ingestion. `--restart` clears the project database and indexes from the configured or overridden start block. On this bounded range the run completes in seconds:

    ```text theme={"system"}
    Chain 1: stats final | checkpoint: 20599999 | next: 20601001 | ... | total: 1001 blocks, 16238 events
    Done.
    ```

    During longer runs, `sqd-go` prints a `stats` line (throughput, events, checkpoint) and a `profile` line (fetch, parse, insert, custom time) every 10 seconds. For faster backfills, add `--parallel-fetch` to spread finalized ranges across parallel workers.

    <Info>
      `sqd-go dev` manages the project's Compose stack for the duration of a run, but does not support `--state`. Stateful projects must use `sqd-go start <path> --state`.
    </Info>
  </Step>

  <Step title="Query the data">
    The run above produces one table per configured event, a state history table and a live view from the template's processor, and a `sync_state` bookkeeping table:

    | Table                   | Contents                                |
    | ----------------------- | --------------------------------------- |
    | `erc20_transfer_events` | Decoded `Transfer` events with metadata |
    | `erc20_approval_events` | Decoded `Approval` events with metadata |
    | `user_positions_log`    | Full state history, one row per update  |
    | `user_positions_live`   | Latest state per primary key (view)     |

    Event tables contain the decoded event fields plus `block_number`, `block_timestamp`, `transaction_index`, and `log_index`. Address columns are stored as raw bytes (`FixedString(20)`), so use `hex()` to display them:

    ```sql theme={"system"}
    SELECT
        concat('0x', lower(hex(address))) AS address,
        total_in,
        total_out,
        transfer_count
    FROM `my-indexer`.user_positions_live
    ORDER BY total_in DESC
    LIMIT 10
    ```
  </Step>
</Steps>

## Custom state in Go

Derived state is defined by two files in the project directory. `custom_schema.go` declares entity shapes as Go structs; each `<Entity>Schema` struct becomes a ClickHouse-backed state store accessed as `state.<Entity>`:

```go custom_schema.go theme={"system"}
// pk: Address
type UserPositionSchema struct {
    Address        common.Address // primary key
    TotalIn        uint256.Int
    TotalOut       uint256.Int
    TransferCount  uint64
    UpdatedAtBlock uint64    // set automatically by Save()
    UpdatedAt      time.Time // set automatically by Save()
}
```

`custom_processor.go` implements a `Process` function that is called once per block with the decoded events and updates state through `Get`/`GetOrCreate` and `Save`:

```go custom_processor.go theme={"system"}
func Process(state *generated.State, block *generated.ParsedBlock) error {
    for event := range block.EventsIter() {
        transfer, ok := event.(*generated.ERC20Transfer)
        if !ok {
            continue
        }
        pos := state.UserPosition.GetOrCreate(transfer.To)
        pos.TotalIn.Add(&pos.TotalIn, &transfer.Value)
        pos.TransferCount++
        state.UserPosition.Save(pos, transfer.EventMeta)
    }
    return nil
}
```

The scaffolded template ships a complete, heavily commented version of both files. `--state` regenerates the project, compiles your processor into a fresh binary, and runs it. For the build mechanics of standalone projects, see the [Go modules guide](https://github.com/subsquid-labs/sqd-go/blob/main/wiki/GO_MODULES.md) in the sqd-go wiki.

## Networks and Portal access

Chains are selected by numeric chain id in `config.yaml`. Built-in Portal endpoints exist only for Ethereum mainnet (`1`) and Polygon (`137`); for any other network, set `SQD_PORTAL_ENDPOINT` to that network's Portal dataset URL (from the [EVM network list](/en/data/evm)) with `/stream` appended:

```bash theme={"system"}
SQD_PORTAL_ENDPOINT=https://portal.sqd.dev/datasets/base-mainnet/stream \
sqd-go start . --state --restart
```

<Warning>
  With the current version, a chain id other than `1` or `137` falls back to the Polygon dataset unless `SQD_PORTAL_ENDPOINT` is set. Always set it explicitly when indexing other networks.
</Warning>

No API token is required for public Portal access. `SQD_API_TOKEN` sets a Portal API token when you have one.

## CLI and configuration

| Command                 | Purpose                                                                                |
| ----------------------- | -------------------------------------------------------------------------------------- |
| `sqd-go init`           | Scaffold a project interactively, from an ABI file, or from the ERC-20 template        |
| `sqd-go codegen <path>` | Validate the configuration and generate SQL and Go code                                |
| `sqd-go start <path>`   | Run codegen, connect to an existing ClickHouse, and index                              |
| `sqd-go dev <path>`     | Run codegen, start the project's Compose stack, index, and tear the stack down on exit |
| `sqd-go stop`           | Stop the local Compose stack and remove its volumes                                    |

A run without `--restart` resumes from the durable checkpoint: it first deletes rows above the checkpoint from every table with a `block_number` column, which makes reprocessing after an interrupted run idempotent. `--reindex-from <n>` rewinds to a specific block instead. Runtime tuning uses `SQD_*` environment variables (`SQD_PARALLEL_FETCHERS`, `SQD_PARALLEL_RPS`, and others); ClickHouse credentials belong in `.env`. The [CLI reference](https://github.com/subsquid-labs/sqd-go/blob/main/wiki/CLI.md) documents all commands, flags, and environment variables.

## Memory and monitoring

Hot state lives in the Go heap with a fixed cache of 100,000 entries per entity. Evicted entries spill to a local Pebble-backed cold tier that is rebuilt from ClickHouse on each start; ClickHouse remains the single source of truth. The cold tier is on by default and can be disabled with `--no-cold-cache`. `SQD_COLDCACHE_MB` caps its block cache; opening the tier also allocates a fixed 256 MiB Bloom filter on the heap. See [METRICS.md](https://github.com/subsquid-labs/sqd-go/blob/main/wiki/METRICS.md) for memory tuning details.

Set `SQD_METRICS_CH=1` to write runtime metrics (block and event throughput, checkpoint lag, memory and garbage collection, goroutine and CPU usage) to the `monitoring.indexer_metrics` ClickHouse table. The sqd-go repository's own Compose stack includes provisioned Grafana dashboards for these metrics.

## Performance

The maintainers publish results for `sqd-go` in the [open-indexer-benchmark](https://github.com/mev-tools/open-indexer-benchmark) repository, run in two configurations: default settings with plain `--restart`, and a tuned run (`--parallel-fetch --no-replay` with `SQD_PARALLEL_FETCHERS=12` and `SQD_PARALLEL_RPS=10`):

| Case (metric)                       | Default                                        | Tuned            |
| ----------------------------------- | ---------------------------------------------- | ---------------- |
| `case_1_lbtc_event_only` (time)     | 10m 0s (capped, 1,040,134 of 1,599,998 blocks) | 3m 35s, complete |
| `case_2_lbtc_full` (time)           | 9m 45s, complete                               | 4m 31s, complete |
| `erc20-transfer-events` (blocks/s)  | 2,860.7                                        | 7,528.1          |
| `erc20-account-balances` (blocks/s) | 3,644.4                                        | 7,185.7          |

The tuned configuration trades memory for throughput: peak RSS on the `erc20-transfer-events` case grows from 621 MB to about 2 GB. The benchmark repository lists results for other indexers under the same workloads, along with the runner implementations and conditions.

## Limitations

* EVM networks and ClickHouse only
* No factory contract tracking: addresses are fixed in the configuration (an event can also be matched at any address), and contracts deployed dynamically by factories are not discovered
* Custom derived state must be written in Go

## Resources

* [Source code on GitHub](https://github.com/subsquid-labs/sqd-go)
* [sqd-go wiki](https://github.com/subsquid-labs/sqd-go/tree/main/wiki): CLI, config, custom schema and processor references
* [Open indexer benchmark](https://github.com/mev-tools/open-indexer-benchmark)


## Related topics

- [Choosing your tool](/en/sdk/options-comparison.md)
- [Changelog](/changelog.md)
- [SQD Connector for Claude](/en/ai/claude-connector.md)
- [Pricing Overview](/en/cloud/pricing/overview.md)
- [Build with SQD](/en/sdk/overview.md)