Pipes SDK 1.0 - SQD Documentation

Getting onchain data into a database still means writing plumbing: RPC pagination and rate limits, ABI decoding, reorg cleanup, resume logic after a crash. Pipes SDK removes that layer. It is a TypeScript library: you describe the events you care about, and a pipe streams them from SQD Portal into your database, decoded, typed, and kept consistent through reorgs. The 1.0 line is available on npm today, under the beta tag.

What you can do with it

Backfill years of history and keep following the chain head, with the same pipe.
Write decoded events to ClickHouse, Postgres, BigQuery, or Parquet files.
Run the same pipeline shape on EVM chains and Solana, on any network in the Portal roster.
Test the whole pipeline in CI, without network access.

This is a complete program. It prints live USDC transfers:

import { commonAbis, evmEventDecoder, evmPortalStream } from '@subsquid/pipes/evm'

const stream = evmPortalStream({
  id: 'usdc-transfers',
  portal: 'https://portal.sqd.dev/datasets/ethereum-mainnet',
  outputs: evmEventDecoder({
    range: { from: 'latest' },
    events: { transfers: commonAbis.erc20.events.Transfer },
  }),
})

for await (const { data } of stream) {
  console.log(data.transfers)
}

Swapping console.log for a database is one more component. The target keeps the write position and cleans up reorged rows on its own.

Get started

npm i @subsquid/pipes@beta

To start from a generated project, run pnpx @subsquid/pipes-cli@beta init, or follow the EVM quickstart or the Solana quickstart.

Upgrading from the 0.1.x betas or an earlier 1.0 alpha? See Upgrading from the betas.

What’s new in 1.0

1.0 rebuilds the internals around correctness: every target now recovers cleanly from crashes and reorgs. BigQuery and Parquet join ClickHouse and Postgres as built-in destinations. New tooling covers testing, local development, and tracing, and the EVM and Solana documentation has been rewritten against the new API.

The complete feature list

Area	Details
Sources	Portal streams for EVM and Solana. Historical backfills in large server-filtered batches, live tailing with `from: 'latest'`, an optional finalized-only mode, and batching controls (`maxBytes`, `maxWaitTimeMs`, `maxIdleTimeMs`, `headPollIntervalMs`).
Decoding	`evmEventDecoder` with built-in `commonAbis` and indexed-parameter filters. `solanaInstructionDecoder` with discriminator matching, inner instructions, and token balances.
Dynamic contracts	`contractFactory` tracks contracts deployed by factories: multiple factory addresses, creation-event filters, and a fork-aware SQLite store.
Composition	Named outputs, chained `.pipe()` transforms, the `OutputOf` type helper, and stateful transforms. Queries and transforms combine into modules that are reusable across pipes.
Correctness	Automatic fork handling with a finalization watermark enforced across targets. Cursors are keyed by the pipe `id`, survive restarts, and recover after crashes.
Targets	ClickHouse with managed rollbacks, Postgres via Drizzle, BigQuery over the Storage Write API, and Parquet files. Custom targets via `createTarget()`.
Testing	A scripted mock portal that plays back fork (409) and server-error responses, plus typed EVM event and block builders. Pipe logic runs in CI without network access.
Local development	The dev runner hosts several pipes in one process with per-pipe restarts and a shared metrics endpoint. `portalSqliteCache` replays Portal responses between runs.
Observability	Prometheus metrics with Pipes UI, per-batch profiling, OpenTelemetry tracing, Pino-compatible logging, and RPC latency watchers that compare Portal freshness with your RPC providers.

EVM and Solana are documented. The package also includes Hyperliquid, Tron, and Bitcoin sources, with tested examples in the repository.

Where it fits

Real-time analytics on ClickHouse

A single pipe backfills history and then follows the chain head. The ClickHouse target rolls back reorged rows on its own, and the latency watcher measures data freshness against your RPC providers. There is no rate-limit handling, log pagination, or reorg cleanup left to write.

Warehouse and lakehouse ETL

The BigQuery target writes through Storage Write API committed streams and confines reorg deletes to a bounded block range, with partitioning and clustering DDL generated from your table declarations. The Parquet target produces immutable, finalized-only files named by block range. DuckDB, Spark, Athena, and ClickHouse’s s3() read them directly, and recovery regenerates files deterministically instead of deduplicating.

Application backends on Postgres

The Drizzle target writes each batch in a transaction and advances the cursor with it, so an API reading the database never observes a half-applied block. Tables are plain Drizzle schemas, typed from the decoded events. pipes init generates the full setup.

Embedded in an existing service

A pipe is a plain async iterator. With the logger and profiler disabled it runs inside any Node.js process, with no side services. The mock portal makes pipe logic testable in CI, and the dev runner keeps a multi-pipe local setup in one terminal.

Multi-chain products

The pipeline shape is the same on every chain. Replacing evmPortalStream and evmEventDecoder with solanaPortalStream and solanaInstructionDecoder leaves the rest of the pipe unchanged, including the target. Decoders combine into reusable modules, and one Portal account covers the full network roster.

Upgrading from the betas

The renames in 1.0 are hard — the old names are removed, without deprecated aliases — so the compiler points at each call site, and stored state migrates on its own. The migration guide covers both paths: from the 0.1.x betas and from earlier 1.0 alphas.

What changed

Sources are now evmPortalStream / solanaPortalStream, the EVM decoder is evmEventDecoder, and the factory helper is contractFactory with a childAddressField option and a contractFactorySqliteStore store. The previous names (evmPortalSource, evmDecoder, factory, factorySqliteDatabase) are removed — the migration guide has the full rename tables.
Target cursors are keyed by the pipe’s source id instead of the legacy 'stream' key. Existing state rows migrate automatically on first resume; pin the old key by setting the target’s state id (settings.id for ClickHouse, settings.state.id for Drizzle).
Releases are published with npm Trusted Publishing. Every artifact carries a verifiable SLSA provenance attestation.

FAQ

Is 1.0 generally available?

Not yet. The 1.0 line is published as a beta: npm i @subsquid/pipes@beta installs it. The API is what 1.0 will ship, and the migration below is the one you need — but pin @beta until the stable release, which will be announced separately.

Does Pipes SDK replace Squid SDK?

No. Squid SDK remains the choice for self-contained indexing services in the style of TheGraph, Ponder, or Envio, such as a GraphQL API backed by its own database. Pipes SDK streams Portal data into a stack you already own. See Why Pipes SDK? for the comparison.

Which chains does it support?

EVM chains and Solana are documented. Hyperliquid, Tron, and Bitcoin sources are included with tested examples. Any dataset served by SQD Portal is reachable.

Will beta code break?

Code needs mechanical renames — 1.0 removes the old names, and the compiler flags every call site. Stored cursors and target state migrate automatically, and the CLI generates code on the new names. The migration guide is the checklist.

Where do I start?

The EVM quickstart or the Solana quickstart. Release notes are on GitHub.

We post announcements here and in our Telegram community. Questions? Get in touch.

​What you can do with it

​Get started

​What’s new in 1.0

​Where it fits

​Upgrading from the betas

​FAQ

What you can do with it

Get started

What’s new in 1.0

Where it fits

Upgrading from the betas

FAQ