Private Portal setup - SQD Documentation

A private Portal stack can serve finalized SQD Network data, recent data from your own RPC endpoints, or both through one API. Start by choosing the topology. The services and storage requirements differ substantially.

Choose a topology

Topology	Historical data	Recent data	SQD lock required	Main trade-off
Archival only	SQD Network	No	Yes	Fast backfills, but data can lag the chain head
Real-time only	Your RPC through HotblocksDB	Your RPC through HotblocksDB	No	Supports custom EVM networks, but retention determines disk growth and replay depth
Archival plus real-time	SQD Network	Your RPC through HotblocksDB	Yes	Full history and current data, with more services to operate

Real-time-only storage configured from block zero grows with the chain. Estimate disk growth before using it for a large or high-throughput network.

Components and service count

Component	Responsibility	How many to run
Portal	Serves the Portal API and joins SQD Network history with real-time data	One or more stateless replicas
EVM data service	Polls and validates one RPC source, then serves full blocks to HotblocksDB	One per RPC endpoint
HotblocksDB	Stores and queries recent blocks for one or more datasets	One stateful instance per failure domain; replicas have independent disks
Hotblocks-retain	Tells HotblocksDB when blocks are available from the Network and can be removed	One per `Api`-retention HotblocksDB instance

You do not need one HotblocksDB per network. A single HotblocksDB configuration can declare several datasets and consume one or more EVM data services for each dataset. You normally run at least one EVM data service per network. Run additional services when the same network uses multiple RPC providers for redundancy.

Prerequisites

Docker and Docker Compose
A persistent volume for every HotblocksDB instance
An RPC endpoint with enough capacity for every real-time EVM dataset
debug_traceBlockByHash support when traces or state diffs are required
A load balancer and multiple replicas when one instance is not an acceptable failure domain

Accessing finalized data from the Network also requires:

at least 1,000,000 locked SQD (see Token requirements and compute units for sizing);
an Arbitrum RPC endpoint and an Ethereum L1 RPC endpoint;
a registered Portal peer ID;
Arbitrum ETH for registration and lock transactions.

The real-time ingestion example below uses the EVM data service. Other data kinds require a compatible service that can feed HotblocksDB.

Archival-only setup

Use this topology for high-throughput historical queries when a delay near the chain head is acceptable.

Clone the Portal repository

git clone https://github.com/subsquid/sqd-portal
cd sqd-portal
cp mainnet.env .env

Generate and register the Portal identity

Generate a private key and peer ID:

docker run --rm \
  -u "$(id -u):$(id -g)" \
  -v "$PWD:/cwd" \
  subsquid/keygen:latest /cwd/portal.key

Protect portal.key. Anyone with the key can run a process using that peer identity. See consequences of losing your key.

Review the network configuration

The repository’s mainnet.config.yml serves every Network dataset:

mainnet.config.yml

hostname: http://0.0.0.0:8000
sqd_network:
  datasets: https://cdn.subsquid.io/sqd-network/datasets.yml
  metadata: https://cdn.subsquid.io/sqd-network/mainnet/metadata.yml
  serve: all

Review the RPC endpoints and listen address in .env before starting the service.

Start and verify

KEY_PATH=./portal.key docker compose up -d
docker compose logs -f portal

The lock becomes active at the start of the next network epoch. After activation:

curl --fail http://127.0.0.1:8000/ready
curl --fail http://127.0.0.1:8000/datasets

Real-time-only setup for a custom EVM network

HotblocksDB already serves the Portal streaming API. If clients only need real-time stream and head endpoints, they can connect to HotblocksDB directly without running the Portal service.

Configure retention

Create a HotblocksDB dataset configuration:

hotblocks.yaml

krown-network:
  kind: evm
  retention_strategy:
    Head: 2000
  data_sources:
    - http://krown-data-service:3000

Choose one retention strategy:

Head: 2000 keeps a sliding window of the latest 2,000 blocks.
FromBlock: { number: 0 } keeps everything from block zero.
FromBlock: { number: 123456 } keeps everything from a chosen start block.

Use FromBlock only when the retained range fits on disk. Keeping the whole chain makes local reindexing fast, but storage continues to grow.

Run the services

The minimal Compose shape is:

compose.yaml

services:
  krown-data-service:
    image: subsquid/evm-data-service:latest
    command:
      - --http-rpc
      - ${KROWN_RPC_URL:?}
      - --receipts
      - --block-cache-size
      - "1000"

  hotblocks-db:
    image: subsquid/data-hotblocks:latest
    command:
      - --datasets
      - /app/hotblocks.yaml
      - --db
      - /run/db
      - --port
      - "8081"
    ports:
      - "8081:8081"
    volumes:
      - ./hotblocks.yaml:/app/hotblocks.yaml:ro
      - hotblocks-data:/run/db
    depends_on:
      - krown-data-service

volumes:
  hotblocks-data:

Set KROWN_RPC_URL, then start the stack:

docker compose up -d
docker compose logs -f krown-data-service hotblocks-db

Do not expose the EVM data service port publicly. HotblocksDB is the client-facing API in this topology.

Verify the custom dataset

curl --fail http://127.0.0.1:8081/datasets/krown-network
curl --fail http://127.0.0.1:8081/datasets/krown-network/head

Run a bounded query:

curl --fail --compressed \
  http://127.0.0.1:8081/datasets/krown-network/stream \
  -H "content-type: application/json" \
  -d '{
    "type": "evm",
    "fromBlock": 1,
    "toBlock": 10,
    "includeAllBlocks": true,
    "fields": {
      "block": {
        "number": true,
        "hash": true,
        "timestamp": true
      }
    }
  }'

Add --traces, --diffs, and --use-debug-api-for-statediffs to the EVM data service only when clients request those fields and the RPC supports the required debug methods.

Archival plus real-time setup

Use this topology when clients need fast historical backfills and current blocks through the same dataset URL. The data flow is:

RPC -> EVM data service -> HotblocksDB -> Portal -> client
SQD Network ---------------------------> Portal -> client

Configure HotblocksDB

Use Api retention for datasets whose older blocks are available from the Network:

hotblocks.yaml

ethereum-mainnet:
  kind: evm
  retention_strategy: Api
  data_sources:
    - http://ethereum-data-service:3000

base-mainnet:
  kind: evm
  retention_strategy: Api
  data_sources:
    - http://base-data-service:3000

avalanche-mainnet:
  kind: evm
  retention_strategy: Api
  data_sources:
    - http://avalanche-data-service:3000

With Api retention, HotblocksDB waits for a retention update before it starts ingesting the dataset.

Configure Hotblocks-retain

One helper can track several datasets:

retained-datasets.yaml

datasets:
  - ethereum-mainnet
  - base-mainnet
  - avalanche-mainnet

Run it against the HotblocksDB instance:

docker run --rm \
  -v "$PWD/retained-datasets.yaml:/cfg/datasets.yaml:ro" \
  subsquid/data-hotblocks-retain:latest \
  --hotblocks-url http://hotblocks-db:8081 \
  --status-url https://metadata.sqd-datasets.io/scheduler/mainnet/status.json \
  --datasets-url https://cdn.subsquid.io/sqd-network/datasets.yml \
  --datasets-config /cfg/datasets.yaml

Configure Portal routing

Declare the HotblocksDB URL inside every dataset’s real_time section:

mainnet.config.yml

hostname: http://0.0.0.0:8000
sqd_network:
  datasets: https://cdn.subsquid.io/sqd-network/datasets.yml
  metadata: https://cdn.subsquid.io/sqd-network/mainnet/metadata.yml
  serve: manual
datasets:
  ethereum-mainnet:
    real_time:
      kind: evm
      url: http://hotblocks-db:8081
  base-mainnet:
    real_time:
      kind: evm
      url: http://hotblocks-db:8081
  avalanche-mainnet:
    real_time:
      kind: evm
      url: http://hotblocks-db:8081

hotblocksDB is not a valid top-level Portal option. Unknown fields are ignored with a warning. Set datasets.<name>.real_time.url for every real-time dataset.

Portal uses the dataset name as the HotblocksDB dataset name by default. Set real_time.dataset only when the names differ.

Verify routing

curl --fail http://127.0.0.1:8000/datasets/ethereum-mainnet
curl --fail http://127.0.0.1:8000/datasets/base-mainnet
curl --fail http://127.0.0.1:8000/datasets/avalanche-mainnet

Each response should report real_time: true. Run historical and near-head stream queries to verify that both sources are reachable.

Put a custom network behind Portal

A network does not need an SQD Network archive to appear in Portal. Declare only a real-time source:

mainnet.config.yml

hostname: http://0.0.0.0:8000
sqd_network:
  datasets: https://cdn.subsquid.io/sqd-network/datasets.yml
  metadata: https://cdn.subsquid.io/sqd-network/mainnet/metadata.yml
  serve: manual
datasets:
  krown-network:
    real_time:
      kind: evm
      url: http://hotblocks-db:8081

The corresponding krown-network dataset must exist in HotblocksDB. Choose the client endpoint based on the required API:

Client need	Endpoint
Real-time streams and head information only	Connect directly to HotblocksDB
One endpoint for the Network and custom datasets	Connect through Portal
Portal metadata, dataset listing, or timestamp lookup endpoints	Connect through Portal

Squid SDK and Pipes SDK clients can use either endpoint for supported streaming operations.

Token requirements and compute units

The minimum token locking requirement for a Portal instance is 1,000,000 SQD. This should be enough for most use cases. If you want to know the exact capabilities this gives you, read on. The rate limiting mechanism of SQD Network relies on the concept of a compute unit, or CU for short. CUs do not map directly to the amount of data fetched by a Portal instance; instead, they (roughly) represent the amount of work that the network does internally while serving the Portal instance’s requests.

Understanding compute units (CU)

Network datasets are partitioned by block number. Dataset chunks are randomly distributed among worker nodes.When a Portal instance receives a data request, the following happens:

The Portal instance forwards the request to several workers that hold the chunks of the relevant dataset
The workers execute the request separately on each chunk
Workers send the results back to the Portal instance. For lightweight queries, they send one response for each dataset chunk; however, if any response exceeds 100 Mbytes, it’s split into several parts
The Portal instance concatenates the workers’ replies and serves a continuous stream of data to the user

Each response made by any of the workers in step 3 spends exactly 1 CU.

The more SQD you lock and the greater the lockup period is, the more CUs you get. Currently, each locked SQD generates 1-3 CUs at the beginning of each epoch, depending on the lockup period. In principle, any valid amount of locked SQD generates an infinite amount of CUs. However, if the rate at which your queries consume CUs exceeds the rate at which they are produced, your app will be throttled. To avoid that, you may want to understand how many CUs your queries spend. There is currently no tool for estimating the number of CUs a query needs before running it. However, on EVM you can use the following formula to get an order of magnitude estimate:

10^-5 * range_length_blocks * transactions_per_block

Notes:

This assumes lightweight queries - that is, queries that fetch much less data than the total dataset size. For heavyweight queries multiply the estimate by a factor of 2-5.
This is a rough estimate. Multiply it by ten to get a somewhat safe figure. If you want to minimize your SQD lockup, start at that safe figure, then measure the actual amount of CUs you spend and reduce the lockup accordingly.
If your network has an Etherscan-style explorer, you can estimate the transactions_per_block by visiting its front page, reading the “Transactions” stat and dividing it by the “Last finalized block” height.

How the estimate is calculated

For a lightweight query, the amount of CUs spent is determined by how many dataset chunks the network needs to examine to process it. The ingester creates chunks of roughly the same size within each dataset. Since the amount of data per block is roughly proportional to the number of transactions in that block, we can assume that the number of chunks in any given range is proportional to the number of transactions per block.Extrapolating from the Ethereum dataset:

num_chunks_in_range = range_length * (eth_chunks / eth_height) * (chain_txs_per_block / eth_txs_per_block)

Where:

eth_chunks = 3.1e4
eth_height = 2.1e7
eth_txs_per_block = 1.2e2

Multiplying all the known values together and rounding to one decimal digit, we get the 1e-5 coefficient of the final formula.Important assumption: This assumes all EVM datasets have the same chunk size as Ethereum. In reality, chunk sizes vary between 50-1000 Mbytes. Ethereum’s chunk size is roughly 500 Mbytes, so expect the estimate to be off by a factor of 0.5-10, which is within the “order of magnitude” definition.Heavyweight queries: Scale the same way but may spend more than one CU per chunk. The heaviest possible queries (fetching the whole dataset) on Ethereum consume roughly 5 CUs per chunk.

Now, if your queries consume X CUs each and you run them once per Y epochs, you need to lock up at least this much SQD:

X / (Y * boost_factor)

Here, boost_factor is a multiplier ranging from 1 to 3 depending on the lockup length.

Redundancy and capacity

Run one EVM data service per RPC provider and list all providers under the dataset’s data_sources.
Run multiple Portal replicas behind a load balancer. Portal is stateless. Replicas can share a single wallet and a single SQD lock: register one peer ID per replica from the same wallet.
Give every HotblocksDB replica its own persistent volume. Replicas ingest independently and do not share RocksDB files.
Run Hotblocks-retain alongside every HotblocksDB replica that uses Api retention.
Pin tested container versions or digests for production instead of tracking latest automatically.
Protect Portal peer keys, RPC credentials, and internal service ports.
Monitor /ready, /metrics, dataset heads, disk usage, and RPC error rates.

Troubleshooting

What are the consequences of losing my key file / getting it stolen?

If you lose your key file, you won’t be able to run your Portal instance until you generate and register a new one. If your key file is stolen, the perpetrator can cause connectivity issues, effectively creating downtime for your Portal instance. To recover:

Unregister your Portal instance on the portals page.
Generate a new key file.
Register the new portal peer ID.

Portal ignores the real-time source

Check startup logs for:

ignoring unknown config field

Remove a top-level hotblocksDB field and set datasets.<name>.real_time.url.

HotblocksDB does not ingest an `Api` dataset

Confirm Hotblocks-retain is running, includes the exact dataset name, and can reach both HotblocksDB and the SQD Network status URLs. Api retention does not start until the first retention update arrives.

Traces or state diffs are missing

Confirm the EVM data service was started with the matching flags and the RPC supports debug_traceBlockByHash. Do not enable expensive fields that clients do not query.

Disk grows continuously

Inspect retention_strategy. FromBlock retains every block from the configured height. Use a bounded Head window for real-time-only access or Api retention when the Network supplies history.

One RPC failure stops current data

Run a data service for a second provider and add both service URLs to the HotblocksDB dataset. Test provider failure before relying on the setup in production.

The portal is slower than expected

Portal performance metrics are exposed at the /metrics endpoint. Check the throttling statistics:

curl --compressed http://127.0.0.1:8000/metrics | grep throttled

Lower values of portal_stream_throttled_ratio_sum indicate better performance. If throttling is high, consider locking more SQD tokens to increase your compute unit allocation.

​Choose a topology

​Components and service count

​Prerequisites

​Archival-only setup

​Real-time-only setup for a custom EVM network

​Configure retention

​Run the services

​Verify the custom dataset

​Archival plus real-time setup

​Configure HotblocksDB

​Configure Hotblocks-retain

​Configure Portal routing

​Verify routing

​Put a custom network behind Portal

​Token requirements and compute units

​Redundancy and capacity

​Troubleshooting

​What are the consequences of losing my key file / getting it stolen?

​Portal ignores the real-time source

​HotblocksDB does not ingest an Api dataset

​Traces or state diffs are missing

​Disk grows continuously

​One RPC failure stops current data

​The portal is slower than expected

​Source and API references

Choose a topology

Components and service count

Prerequisites

Archival-only setup

Real-time-only setup for a custom EVM network

Configure retention

Run the services

Verify the custom dataset

Archival plus real-time setup

Configure HotblocksDB

Configure Hotblocks-retain

Configure Portal routing

Verify routing

Put a custom network behind Portal

Token requirements and compute units

Redundancy and capacity

Troubleshooting

What are the consequences of losing my key file / getting it stolen?

Portal ignores the real-time source

HotblocksDB does not ingest an `Api` dataset

Traces or state diffs are missing

Disk grows continuously

One RPC failure stops current data

The portal is slower than expected

Source and API references