How to use your dedicated Portal with a key

This guide takes a squid that reads from the public Portal and points it at a dedicated Portal that requires an API key. The data, the queries and the handler stay the same; only the data source configuration changes.

Where are you starting from?

Your squid uses .setGateway() and EvmBatchProcessor (a v2 squid). Migrate it to the public Portal first by following Migrate a squid to Portal, then come back here.
Your squid uses DataSourceBuilder from @subsquid/evm-stream with .setPortal('https://portal.sqd.dev/...'). Read on.

Prerequisites

The base URL of your dedicated Portal and an API key for it. See Portal pricing for the plans that come with a key.
Any released version of @subsquid/evm-stream (0.0.1 or later). All of them accept the http option shown below and send its headers with every Portal request. The only exception is the early 0.0.1-2-0.3887d2 prerelease, whose .setPortal() takes just a URL string or a PortalClient. If your package.json pins a prerelease, update to the latest version first:
npm i @subsquid/evm-stream@latest

Migration steps

Pass the URL and the key to .setPortal()

Switch .setPortal() from the URL string form to the options object form. The dataset URL is now built from your Portal’s base URL, and the key goes into the x-api-key header:

 const dataSource = new DataSourceBuilder()
-  .setPortal('https://portal.sqd.dev/datasets/ethereum-mainnet')
+  .setPortal({
+    url: endpointUrl + '/datasets/ethereum-mainnet',
+    http: {
+      headers: {
+        'x-api-key': portalKey,
+      },
+    },
+  })

The dataset slug (ethereum-mainnet here) is the same as on the public Portal. Portal also accepts the key as Authorization: Bearer <key>; see Presenting an API key.

Already passing http options, or a PortalClient instance?

http replaces the HTTP client settings as a whole. If you already set other options there (e.g. retryAttempts), add headers next to them:

http: {
  retryAttempts: Infinity,
  headers: { 'x-api-key': portalKey },
},

If you construct a PortalClient yourself and pass it to .setPortal(), set the same url and http options on its constructor instead.

Read the URL and the key from the environment

Do not hardcode the key. Read both values from environment variables and fail early if either is missing:

import {assertNotNull} from '@subsquid/util-internal'

const endpointUrl = assertNotNull(process.env.PORTAL_URL, 'PORTAL_URL is required')
const portalKey = assertNotNull(
  process.env.PORTAL_KEY,
  `SQD Portal endpoint ${endpointUrl} requires a PORTAL_KEY`
)

@subsquid/util-internal is already installed as a dependency of @subsquid/evm-stream. Add it to your package.json explicitly (npm i @subsquid/util-internal) if you prefer to import only direct dependencies.

Set the variables locally

Add the variables to the .env file of your squid:

.env

PORTAL_URL=https://<your-portal>
PORTAL_KEY=<your-key>

sqd process loads .env automatically. If you run the compiled squid directly, load it with node --require=dotenv/config lib/main.js. Keep .env out of version control once it holds the key.

Set the variables in SQD Cloud

Optional Only for squids deployed to SQD Cloud.Store the key as an organization secret:

sqd secrets set PORTAL_KEY <your-key> -o <organization>

Then expose both variables to the processor in squid.yaml:

squid.yaml

deploy:
  processor:
    cmd: ["sqd", "process:prod"]
    env:
      PORTAL_URL: https://<your-portal>
      PORTAL_KEY: "${{ secrets.PORTAL_KEY }}"

Smoke-test and deploy

Build and start the squid:

npm run build
npx squid-typeorm-migration apply
node --require=dotenv/config lib/main.js

A missing variable stops the squid at startup with the message from assertNotNull. A key that the Portal rejects fails the first request with a 403; the error code tells a missing key from a wrong or expired one (see Error codes).The data served by your dedicated Portal is the same as on the public one, so there is no need to re-sync. Re-deploy the squid without resetting its database.

Complete example

src/main.ts (data source part)

import {DataSourceBuilder} from '@subsquid/evm-stream'
import {assertNotNull} from '@subsquid/util-internal'
import * as usdcAbi from './abi/usdc'

const USDC_CONTRACT_ADDRESS = '0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48'

const endpointUrl = assertNotNull(process.env.PORTAL_URL, 'PORTAL_URL is required')
const portalKey = assertNotNull(
  process.env.PORTAL_KEY,
  `SQD Portal endpoint ${endpointUrl} requires a PORTAL_KEY`
)

const dataSource = new DataSourceBuilder()
  .setPortal({
    url: endpointUrl + '/datasets/ethereum-mainnet',
    http: {
      headers: {
        'x-api-key': portalKey,
      },
    },
  })
  .addLog({
    where: {
      address: [USDC_CONTRACT_ADDRESS],
      topic0: [usdcAbi.events.Transfer.topic],
    },
    range: {from: 6_082_465},
  })
  .setFields({
    log: {
      address: true,
      topics: true,
      data: true,
      transactionHash: true,
    },
  })
  .build()

The rest of the squid (the run() call and the batch handler) is unchanged. For the full file, see the squid-evm-rt-template.

Overview

Pipes SDK

Squid SDK

Alternative clients

How to use your dedicated Portal with a key

Where are you starting from?

Prerequisites

Migration steps

Complete example

​Where are you starting from?

​Prerequisites

​Migration steps

​Complete example

Where are you starting from?

Prerequisites

Migration steps

Complete example