External APIs and IPFS

Squid processors are standard Node.js processes, enabling you to fetch data from external APIs or IPFS gateways right in the batch handler. This allows you to enrich blockchain data with off-chain information.

For squids deployed to SQD Cloud, use API calls in combination with API keys set via secrets for secure access.

External API Integration

Price Data Example

You can enrich indexed transactions with historical price data using external APIs like CoinGecko:

main.ts

import axios from "axios";
import moment from "moment";
import {augmentBlock} from "@subsquid/evm-objects";

run(dataSource, new TypeormDatabase(), async (ctx) => {
  const burns: Burn[] = [];

  for (let block of ctx.blocks.map(augmentBlock)) {
    for (let txn of block.transactions) {
      burns.push(
        new Burn({
          id: txn.id,
          block: block.header.number,
          address: txn.from,
          value: txn.value,
          txHash: txn.hash,
          price: await getETHPriceByDate(block.header.timestamp),
        })
      );
    }
  }

  await ctx.store.save(burns);
});

async function getETHPriceByDate(timestamp: number): Promise<number> {
  const formatted = moment(new Date(timestamp).toISOString()).format(
    "DD-MM-YYYY"
  );
  const res = await axios.get(
    `https://api.coingecko.com/api/v3/coins/ethereum/history?date=${formatted}&localization=false`
  );
  return res.data.market_data.current_price.usd;
}

Use caching mechanisms to avoid hitting rate limits and improve performance when fetching external data.

The example above awaits an external call for every transaction, which can dominate sync time. Fetch each distinct input (e.g. one price per day) once and reuse it, and keep database writes batched as described in the batch processing guide.

IPFS Integration

For reliable indexing of content stored on IPFS (e.g., NFT metadata), we recommend using dedicated IPFS gateways like those provided by Filebase.

Public IPFS gateways may have rate limits and reliability issues. Use dedicated gateways for production squids.

IPFS Fetching Example

ipfs.ts

import Axios from "axios";
import https from "https";

// Use a dedicated gateway for production
export const BASE_URL = "https://your-dedicated-gateway.ipfs.io/ipfs/";

export const api = Axios.create({
  baseURL: BASE_URL,
  headers: {
    "Content-Type": "application/json",
  },
  withCredentials: false,
  timeout: 5000,
  httpsAgent: new https.Agent({ keepAlive: true }),
});

import { createLogger } from "@subsquid/logger";

const log = createLogger("sqd:processor:ipfs");

export const fetchMetadata = async (cid: string): Promise<any | null> => {
  try {
    const { status, data } = await api.get(`${BASE_URL}/${cid}`);
    log.info(`[IPFS] ${status} CID: ${cid}`);

    if (status < 400) {
      return data;
    }
  } catch (e) {
    log.warn(`[IPFS] ERROR CID: ${cid} - ${(e as Error).message}`);
  }
  return null;
};

Using IPFS in Processor

main.ts

run(dataSource, new TypeormDatabase(), async (ctx) => {
  for (let block of ctx.blocks) {
    for (let log of block.logs) {
      // Decode an NFT event to get the metadata URI
      const { tokenId, tokenURI } = decodeNFTEvent(log);
      const cid = extractCID(tokenURI);

      // Fetch metadata from IPFS
      const metadata = await fetchMetadata(cid);

      if (metadata) {
        // Store the entity with its off-chain metadata
        await ctx.store.save(
          new Token({
            id: tokenId,
            uri: tokenURI,
            name: metadata.name,
            image: metadata.image,
            attributes: metadata.attributes,
          })
        );
      }
    }
  }
});

For a detailed example with IPFS gateway and external API calls, see Step 3 of the BAYC tutorial.

Overview

Pipes SDK

Squid SDK

Alternative clients

External API Integration

Price Data Example

IPFS Integration

IPFS Fetching Example

Using IPFS in Processor

​External API Integration

​Price Data Example

​IPFS Integration

​IPFS Fetching Example

​Using IPFS in Processor

External API Integration

Price Data Example

IPFS Integration

IPFS Fetching Example

Using IPFS in Processor