External APIs and IPFS - SQD Documentation

Squid processors are standard Node.js processes, enabling you to fetch data from external APIs or IPFS gateways right in the batch handler. This allows you to enrich blockchain data with off-chain information.

For squids deployed to SQD Cloud, use API calls in combination with API keys set via secrets for secure access.

External API Integration

Price Data Example

You can enrich indexed transactions with historical price data using external APIs like CoinGecko:

main.ts

import axios from "axios";
import moment from "moment";

processor.run(new TypeormDatabase(), async (ctx) => {
  const prices: TokenPrice[] = [];

  for (let block of ctx.blocks) {
    // Fetch the historical price once per block and attach it to
    // whatever entities you derive from the block's data.
    const price = await getTokenPriceByDate(block.header.timestamp);

    prices.push(
      new TokenPrice({
        id: `${block.header.height}`,
        block: block.header.height,
        price,
      })
    );
  }

  await ctx.store.save(prices);
});

async function getTokenPriceByDate(timestamp: number): Promise<number> {
  const formatted = moment(new Date(timestamp).toISOString()).format(
    "DD-MM-YYYY"
  );
  const res = await axios.get(
    `https://api.coingecko.com/api/v3/coins/ethereum/history?date=${formatted}&localization=false`
  );
  return res.data.market_data.current_price.usd;
}

Use caching mechanisms to avoid hitting rate limits and improve performance when fetching external data.

The example above awaits an external call for every block, which can dominate sync time. Fetch each distinct input (e.g. one price per day) once and reuse it, and keep database writes batched as described in the batch processing guide.

IPFS Integration

For reliable indexing of content stored on IPFS (e.g., NFT metadata), we recommend using dedicated IPFS gateways like those provided by Filebase.

Public IPFS gateways may have rate limits and reliability issues. Use dedicated gateways for production squids.

IPFS Fetching Example

ipfs.ts

import Axios from "axios";
import https from "https";

// Use a dedicated gateway for production
export const BASE_URL = "https://your-dedicated-gateway.ipfs.io/ipfs/";

export const api = Axios.create({
  baseURL: BASE_URL,
  headers: {
    "Content-Type": "application/json",
  },
  withCredentials: false,
  timeout: 5000,
  httpsAgent: new https.Agent({ keepAlive: true }),
});

import { createLogger } from "@subsquid/logger";

const log = createLogger("sqd:processor:ipfs");

export const fetchMetadata = async (cid: string): Promise<any | null> => {
  try {
    const { status, data } = await api.get(`${BASE_URL}/${cid}`);
    log.info(`[IPFS] ${status} CID: ${cid}`);

    if (status < 400) {
      return data;
    }
  } catch (e) {
    log.warn(`[IPFS] ERROR CID: ${cid} - ${(e as Error).message}`);
  }
  return null;
};

Using IPFS in Processor

main.ts

processor.run(new TypeormDatabase(), async (ctx) => {
  for (let block of ctx.blocks) {
    // Pull metadata URIs out of this block's decoded items,
    // then enrich each one:
    for (let { tokenId, tokenURI, cid } of extractMetadataRefs(block)) {

      // Fetch metadata from IPFS
      const metadata = await fetchMetadata(cid);

      if (metadata) {
        // Store the entity with its off-chain metadata
        await ctx.store.save(
          new Token({
            id: tokenId,
            uri: tokenURI,
            name: metadata.name,
            image: metadata.image,
            attributes: metadata.attributes,
          })
        );
      }
    }
  }
});

​External API Integration

​Price Data Example

​IPFS Integration

​IPFS Fetching Example

​Using IPFS in Processor

External API Integration

Price Data Example

IPFS Integration

IPFS Fetching Example

Using IPFS in Processor