> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sqd.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Pipes SDK 1.0

> Pipes SDK streams onchain data into your database. The 1.0 line is on npm, in beta.

Getting onchain data into a database still means writing plumbing: RPC pagination and rate limits, ABI decoding, reorg cleanup, resume logic after a crash. Pipes SDK removes that layer. It is a TypeScript library: you describe the events you care about, and a pipe streams them from [SQD Portal](/en/portal/overview) into your database, decoded, typed, and kept consistent through reorgs.

The 1.0 line is available on npm today, under the `beta` tag.

## What you can do with it

* Backfill years of history and keep following the chain head, with the same pipe.
* Write decoded events to ClickHouse, Postgres, BigQuery, or Parquet files.
* Run the same pipeline shape on EVM chains and Solana, on any network in the [Portal roster](/en/data/all-networks).
* Test the whole pipeline in CI, without network access.

This is a complete program. It prints live USDC transfers:

```ts theme={"system"}
import { commonAbis, evmEventDecoder, evmPortalStream } from '@subsquid/pipes/evm'

const stream = evmPortalStream({
  id: 'usdc-transfers',
  portal: 'https://portal.sqd.dev/datasets/ethereum-mainnet',
  outputs: evmEventDecoder({
    range: { from: 'latest' },
    events: { transfers: commonAbis.erc20.events.Transfer },
  }),
})

for await (const { data } of stream) {
  console.log(data.transfers)
}
```

Swapping `console.log` for a database is one more component. The target keeps the write position and cleans up reorged rows on its own.

## Get started

```bash theme={"system"}
npm i @subsquid/pipes@beta
```

To start from a generated project, run `pnpx @subsquid/pipes-cli@beta init`, or follow the [EVM quickstart](/en/sdk/pipes-sdk/evm/quickstart) or the [Solana quickstart](/en/sdk/pipes-sdk/solana/quickstart).

<Info>
  Upgrading from the 0.1.x betas or an earlier 1.0 alpha? See [Upgrading from the betas](#upgrading-from-the-betas).
</Info>

## What's new in 1.0

1.0 rebuilds the internals around correctness: every target now recovers cleanly from crashes and reorgs. BigQuery and Parquet join ClickHouse and Postgres as built-in destinations. New tooling covers testing, local development, and tracing, and the EVM and Solana documentation has been rewritten against the new API.

<Accordion title="The complete feature list">
  | Area                  | Details                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
  | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
  | **Sources**           | [Portal streams](/en/sdk/pipes-sdk/evm/reference/basic-components/source) for EVM and Solana. Historical backfills in large server-filtered batches, live tailing with `from: 'latest'`, an optional finalized-only mode, and batching controls (`maxBytes`, `maxWaitTimeMs`, `maxIdleTimeMs`, `headPollIntervalMs`).                                                                                                                                                                                                                                  |
  | **Decoding**          | [`evmEventDecoder`](/en/sdk/pipes-sdk/evm/reference/utility-components/evm-decoder) with built-in `commonAbis` and indexed-parameter filters. [`solanaInstructionDecoder`](/en/sdk/pipes-sdk/solana/reference/utility-components/instruction-decoder) with discriminator matching, inner instructions, and token balances.                                                                                                                                                                                                                             |
  | **Dynamic contracts** | [`contractFactory`](/en/sdk/pipes-sdk/evm/reference/utility-components/factory) tracks contracts deployed by factories: multiple factory addresses, creation-event filters, and a fork-aware SQLite store.                                                                                                                                                                                                                                                                                                                                             |
  | **Composition**       | Named outputs, chained [`.pipe()` transforms](/en/sdk/pipes-sdk/evm/reference/basic-components/transformer), the `OutputOf` type helper, and [stateful transforms](/en/sdk/pipes-sdk/evm/guides/advanced-topics/stateful-transforms). Queries and transforms combine into modules that are reusable across pipes.                                                                                                                                                                                                                                      |
  | **Correctness**       | Automatic [fork handling](/en/sdk/pipes-sdk/evm/guides/architecture-deep-dives/fork-handling) with a finalization watermark enforced across targets. [Cursors](/en/sdk/pipes-sdk/evm/guides/architecture-deep-dives/cursor-management) are keyed by the pipe `id`, survive restarts, and recover after crashes.                                                                                                                                                                                                                                        |
  | **Targets**           | [ClickHouse](/en/sdk/pipes-sdk/evm/reference/basic-components/target/clickhouse) with managed rollbacks, [Postgres via Drizzle](/en/sdk/pipes-sdk/evm/reference/basic-components/target/postgres-drizzle), [BigQuery](/en/sdk/pipes-sdk/evm/reference/basic-components/target/bigquery) over the Storage Write API, and [Parquet files](/en/sdk/pipes-sdk/evm/reference/basic-components/target/parquet). Custom targets via [`createTarget()`](/en/sdk/pipes-sdk/evm/reference/basic-components/target/create-target).                                |
  | **Testing**           | A [scripted mock portal](/en/sdk/pipes-sdk/evm/guides/basic-development/testing) that plays back fork (409) and server-error responses, plus typed EVM event and block builders. Pipe logic runs in CI without network access.                                                                                                                                                                                                                                                                                                                         |
  | **Local development** | The [dev runner](/en/sdk/pipes-sdk/evm/guides/basic-development/dev-runner) hosts several pipes in one process with per-pipe restarts and a shared metrics endpoint. [`portalSqliteCache`](/en/sdk/pipes-sdk/evm/reference/utility-components/sqlite-cache) replays Portal responses between runs.                                                                                                                                                                                                                                                     |
  | **Observability**     | Prometheus [metrics](/en/sdk/pipes-sdk/evm/guides/advanced-topics/metrics) with [Pipes UI](/en/sdk/pipes-sdk/evm/guides/basic-development/pipes-ui), per-batch [profiling](/en/sdk/pipes-sdk/evm/guides/advanced-topics/profiling), [OpenTelemetry tracing](/en/sdk/pipes-sdk/evm/guides/advanced-topics/tracing), Pino-compatible [logging](/en/sdk/pipes-sdk/evm/guides/advanced-topics/logging), and [RPC latency watchers](/en/sdk/pipes-sdk/evm/guides/advanced-topics/latency-monitoring) that compare Portal freshness with your RPC providers. |

  EVM and Solana are documented. The package also includes Hyperliquid, Tron, and Bitcoin sources, with tested examples in the [repository](https://github.com/subsquid-labs/pipes-sdk).
</Accordion>

## Where it fits

<AccordionGroup>
  <Accordion title="Real-time analytics on ClickHouse">
    A single pipe backfills history and then follows the chain head. The ClickHouse target rolls back reorged rows on its own, and the latency watcher measures data freshness against your RPC providers. There is no rate-limit handling, log pagination, or reorg cleanup left to write.
  </Accordion>

  <Accordion title="Warehouse and lakehouse ETL">
    The BigQuery target writes through Storage Write API committed streams and confines reorg deletes to a bounded block range, with partitioning and clustering DDL generated from your table declarations. The Parquet target produces immutable, finalized-only files named by block range. DuckDB, Spark, Athena, and ClickHouse's `s3()` read them directly, and recovery regenerates files deterministically instead of deduplicating.
  </Accordion>

  <Accordion title="Application backends on Postgres">
    The Drizzle target writes each batch in a transaction and advances the cursor with it, so an API reading the database never observes a half-applied block. Tables are plain Drizzle schemas, typed from the decoded events. `pipes init` generates the full setup.
  </Accordion>

  <Accordion title="Embedded in an existing service">
    A pipe is a plain async iterator. With the [logger and profiler disabled](/en/sdk/pipes-sdk/evm/guides/basic-development/running-bare-bones) it runs inside any Node.js process, with no side services. The mock portal makes pipe logic testable in CI, and the dev runner keeps a multi-pipe local setup in one terminal.
  </Accordion>

  <Accordion title="Multi-chain products">
    The pipeline shape is the same on every chain. Replacing `evmPortalStream` and `evmEventDecoder` with `solanaPortalStream` and `solanaInstructionDecoder` leaves the rest of the pipe unchanged, including the target. Decoders combine into reusable modules, and one Portal account covers the full [network roster](/en/data/all-networks).
  </Accordion>
</AccordionGroup>

## Upgrading from the betas

The renames in 1.0 are hard — the old names are removed, without deprecated aliases — so the compiler points at each call site, and stored state migrates on its own. The [migration guide](/en/sdk/pipes-sdk/evm/migration) covers both paths: from the 0.1.x betas and from earlier 1.0 alphas.

<Accordion title="What changed">
  * Sources are now `evmPortalStream` / `solanaPortalStream`, the EVM decoder is `evmEventDecoder`, and the factory helper is `contractFactory` with a `childAddressField` option and a `contractFactorySqliteStore` store. The previous names (`evmPortalSource`, `evmDecoder`, `factory`, `factorySqliteDatabase`) are removed — the [migration guide](/en/sdk/pipes-sdk/evm/migration) has the full rename tables.
  * Target cursors are keyed by the pipe's source `id` instead of the legacy `'stream'` key. Existing state rows migrate automatically on first resume; pin the old key by setting the target's state id (`settings.id` for ClickHouse, `settings.state.id` for Drizzle).
  * Releases are published with npm Trusted Publishing. Every artifact carries a verifiable SLSA provenance attestation.
</Accordion>

## FAQ

<AccordionGroup>
  <Accordion title="Is 1.0 generally available?">
    Not yet. The 1.0 line is published as a beta: `npm i @subsquid/pipes@beta` installs it. The API is what 1.0 will ship, and the migration below is the one you need — but pin `@beta` until the stable release, which will be announced separately.
  </Accordion>

  <Accordion title="Does Pipes SDK replace Squid SDK?">
    No. [Squid SDK](/en/sdk/squid-sdk/overview) remains the choice for self-contained indexing services in the style of TheGraph, Ponder, or Envio, such as a GraphQL API backed by its own database. Pipes SDK streams Portal data into a stack you already own. See [Why Pipes SDK?](/en/sdk/pipes-sdk/evm/why-pipes-sdk) for the comparison.
  </Accordion>

  <Accordion title="Which chains does it support?">
    EVM chains and Solana are documented. Hyperliquid, Tron, and Bitcoin sources are included with tested examples. Any dataset served by [SQD Portal](/en/portal/overview) is reachable.
  </Accordion>

  <Accordion title="Will beta code break?">
    Code needs mechanical renames — 1.0 removes the old names, and the compiler flags every call site. Stored cursors and target state migrate automatically, and the CLI generates code on the new names. The [migration guide](/en/sdk/pipes-sdk/evm/migration) is the checklist.
  </Accordion>

  <Accordion title="Where do I start?">
    The [EVM quickstart](/en/sdk/pipes-sdk/evm/quickstart) or the [Solana quickstart](/en/sdk/pipes-sdk/solana/quickstart). Release notes are on [GitHub](https://github.com/subsquid-labs/pipes-sdk/releases).
  </Accordion>
</AccordionGroup>

We post announcements here and in our [Telegram community](https://t.me/HydraDevs). Questions? [Get in touch](https://sqd.dev/contact).