Dev runner

devRunner runs multiple pipes concurrently inside a single Node.js process, with per-pipe restart retries, a shared metrics server, and per-pipe scoped loggers.

import { PipeContext, devRunner } from '@subsquid/pipes/runtime/node'

The dev runner is for local development only. All pipes share one JS thread, so a CPU-intensive pipe starves its neighbours, and an OS-level kill takes every pipe down together. For production, run each pipe as a separate process or container.

Usage

Declare each pipe as { id, params, handler } and hand the list to devRunner:

const run = devRunner<Params>(pipes, { metrics: { port: 9090 } })
await run.start()

Each handler receives a PipeContext with the pipe’s id, its params, the shared metrics server, and a logger scoped to the pipe ID. Pass those into the source so all pipes report to one metrics endpoint:

import { commonAbis, evmEventDecoder, evmPortalStream } from '@subsquid/pipes/evm'
import { PipeContext, devRunner } from '@subsquid/pipes/runtime/node'

export type Params = { dataset: string }

async function transfers({ id, params, metrics, logger }: PipeContext<Params>) {
  const stream = evmPortalStream({
    id,
    portal: `https://portal.sqd.dev/datasets/${params.dataset}`,
    metrics,
    logger,
    outputs: evmEventDecoder({
      range: { from: '0' },
      events: {
        transfers: commonAbis.erc20.events.Transfer,
      },
    }),
  })

  for await (const { ctx, data } of stream) {
    ctx.logger.debug(`fetched ${data.transfers.length} transfers`)
  }
}

async function main() {
  const run = devRunner<Params>(
    [
      { id: 'arb', params: { dataset: 'arbitrum-one' }, handler: transfers },
      { id: 'ethereum', params: { dataset: 'ethereum-mainnet' }, handler: transfers },
    ],
    { metrics: { port: 9090 } },
  )

  await run.start()
}

main()

Configuration

devRunner(pipes, config?)

Option	Default	Description
`retry`	`5`	Maximum number of restart attempts per pipe before the runner gives up and re-throws the error. Failures are retried per pipe; one pipe crashing does not restart the others.
`metrics`		`metricsServer()` options (`{ port?, enabled?, logger? }`). When provided, a single Prometheus server is shared by all pipes. When omitted, metrics are a no-op.

Each pipe declared in the runner shows up separately in Pipes UI, keyed by its id. Full runnable example: 12.runner.example.ts.

Overview

Pipes SDK

Squid SDK

Alternative clients

Usage

Configuration

​Usage

​Configuration

Usage

Configuration