Private Portal setup - SQD Documentation

A private Portal stack can serve finalized SQD Network data, recent data from your own RPC endpoints, or both through one API. Start by choosing the topology. The services and storage requirements differ substantially.

Choose a topology

Topology	Historical data	Recent data	SQD lock required	Main trade-off
Archival only	SQD Network	No	Yes	Fast backfills, but data can lag the chain head
Real-time only	Your RPC through HotblocksDB	Your RPC through HotblocksDB	No	Supports custom EVM networks, but retention determines disk growth and replay depth
Archival plus real-time	SQD Network	Your RPC through HotblocksDB	Yes	Full history and current data, with more services to operate

Real-time-only storage configured from block zero grows with the chain. Estimate disk growth before using it for a large or high-throughput network.

Components and service count

Component	Responsibility	How many to run
Portal	Serves the Portal API and joins SQD Network history with real-time data	One or more stateless replicas
EVM data service	Polls and validates one RPC source, then serves full blocks to HotblocksDB	One per RPC endpoint
HotblocksDB	Stores and queries recent blocks for one or more datasets	One stateful instance per failure domain; replicas have independent disks
Hotblocks-retain	Tells HotblocksDB when blocks are available from the Network and can be removed	One per `Api`-retention HotblocksDB instance

You do not need one HotblocksDB per network. A single HotblocksDB configuration can declare several datasets and consume one or more EVM data services for each dataset. You normally run at least one EVM data service per network. Run additional services when the same network uses multiple RPC providers for redundancy.

Prerequisites

Docker and Docker Compose
A persistent volume for every HotblocksDB instance
An RPC endpoint with enough capacity for every real-time EVM dataset
debug_traceBlockByHash support when traces or state diffs are required
A load balancer and multiple replicas when one instance is not an acceptable failure domain

Accessing finalized data from the Network also requires:

at least 1,000,000 locked SQD (see Token requirements and compute units for sizing);
an Arbitrum RPC endpoint and an Ethereum L1 RPC endpoint;
a registered Portal peer ID;
Arbitrum ETH for registration and lock transactions.

The real-time ingestion example below uses the EVM data service. Other data kinds require a compatible service that can feed HotblocksDB.

Archival-only setup

Use this topology for high-throughput historical queries when a delay near the chain head is acceptable.

Clone the Portal repository

git clone https://github.com/subsquid/sqd-portal
cd sqd-portal
cp mainnet.env .env

Generate and register the Portal identity

Generate a private key and peer ID:

docker run --rm \
  -u "$(id -u):$(id -g)" \
  -v "$PWD:/cwd" \
  subsquid/keygen:latest /cwd/portal.key

Protect portal.key. Anyone with the key can run a process using that peer identity. See consequences of losing your key.

Review the network configuration

The repository’s mainnet.config.yml serves every Network dataset:

mainnet.config.yml

hostname: http://0.0.0.0:8000
sqd_network:
  datasets: https://cdn.subsquid.io/sqd-network/datasets.yml
  metadata: https://cdn.subsquid.io/sqd-network/mainnet/metadata.yml
  serve: all

Review the RPC endpoints and listen address in .env before starting the service.

Start and verify

KEY_PATH=./portal.key docker compose up -d
docker compose logs -f portal

The lock becomes active at the start of the next network epoch. After activation:

curl --fail http://127.0.0.1:8000/ready
curl --fail http://127.0.0.1:8000/datasets

Real-time-only setup for a custom EVM network

HotblocksDB already serves the Portal streaming API. If clients only need real-time stream and head endpoints, they can connect to HotblocksDB directly without running the Portal service.

Configure retention

Create a HotblocksDB dataset configuration:

hotblocks.yaml

krown-network:
  kind: evm
  retention_strategy:
    Head: 2000
  data_sources:
    - http://krown-data-service:3000

Choose one retention strategy:

Head: 2000 keeps a sliding window of the latest 2,000 blocks.
FromBlock: { number: 0 } keeps everything from block zero.
FromBlock: { number: 123456 } keeps everything from a chosen start block.

Use FromBlock only when the retained range fits on disk. Keeping the whole chain makes local reindexing fast, but storage continues to grow.

Run the services

The minimal Compose shape is:

compose.yaml

services:
  krown-data-service:
    image: subsquid/evm-data-service:latest
    command:
      - --http-rpc
      - ${KROWN_RPC_URL:?}
      - --receipts
      - --block-cache-size
      - "1000"

  hotblocks-db:
    image: subsquid/data-hotblocks:latest
    command:
      - --datasets
      - /app/hotblocks.yaml
      - --db
      - /run/db
      - --port
      - "8081"
    ports:
      - "8081:8081"
    volumes:
      - ./hotblocks.yaml:/app/hotblocks.yaml:ro
      - hotblocks-data:/run/db
    depends_on:
      - krown-data-service

volumes:
  hotblocks-data:

Set KROWN_RPC_URL, then start the stack:

docker compose up -d
docker compose logs -f krown-data-service hotblocks-db

Do not expose the EVM data service port publicly. HotblocksDB is the client-facing API in this topology.

Verify the custom dataset

curl --fail http://127.0.0.1:8081/datasets/krown-network
curl --fail http://127.0.0.1:8081/datasets/krown-network/head

Run a bounded query:

curl --fail --compressed \
  http://127.0.0.1:8081/datasets/krown-network/stream \
  -H "content-type: application/json" \
  -d '{
    "type": "evm",
    "fromBlock": 1,
    "toBlock": 10,
    "includeAllBlocks": true,
    "fields": {
      "block": {
        "number": true,
        "hash": true,
        "timestamp": true
      }
    }
  }'

Add --traces, --diffs, and --use-debug-api-for-statediffs to the EVM data service only when clients request those fields and the RPC supports the required debug methods.

Archival plus real-time setup

Use this topology when clients need fast historical backfills and current blocks through the same dataset URL. The data flow is:

RPC -> EVM data service -> HotblocksDB -> Portal -> client
SQD Network ---------------------------> Portal -> client

Configure HotblocksDB

Use Api retention for datasets whose older blocks are available from the Network:

hotblocks.yaml

ethereum-mainnet:
  kind: evm
  retention_strategy: Api
  data_sources:
    - http://ethereum-data-service:3000

base-mainnet:
  kind: evm
  retention_strategy: Api
  data_sources:
    - http://base-data-service:3000

avalanche-mainnet:
  kind: evm
  retention_strategy: Api
  data_sources:
    - http://avalanche-data-service:3000

With Api retention, HotblocksDB waits for a retention update before it starts ingesting the dataset.

Configure Hotblocks-retain

Api retention means HotblocksDB does not decide on its own when a block may be dropped. Hotblocks-retain makes that decision from outside: it reads each dataset’s last indexed archival block from the Network scheduler status and posts that height to HotblocksDB as the new retention lower bound. It moves a number, never block data. Two consequences follow:

an Api dataset does not ingest anything until the first retention update arrives;
if the lower bound overtakes what Portal believes the archive holds, the blocks in between are in neither source. See How real-time data is served for where this sits in the architecture.

List the datasets to track

One instance can track several datasets. The config is a mapping keyed by the HotblocksDB dataset name:

retained-datasets.yaml

ethereum-mainnet: {}
base-mainnet: {}
avalanche-mainnet: {}

Every property is optional for a public dataset. Set name when the HotblocksDB dataset name differs from the Network dataset it tracks:

retained-datasets.yaml

my-ethereum-copy:
  name: ethereum-mainnet

Set id when the dataset is absent from the manifest at --datasets-url. Private datasets are not published there, so a name never resolves to a dataset ID — even when the HotblocksDB name and the Network name are identical. Give the ID instead:

retained-datasets.yaml

my-private-chain:
  id: s3://my-private-chain-1

This is the same ID the dataset needs in Portal’s own config, for the same reason. An entry carrying id never consults the manifest, but --datasets-url stays a required flag; point it at the public manifest as usual.

Run the service

docker run --rm \
  -v "$PWD/retained-datasets.yaml:/cfg/datasets.yaml:ro" \
  subsquid/data-hotblocks-retain:<version> \
  --hotblocks-url http://hotblocks-db:8081 \
  --status-url https://metadata.sqd-datasets.io/scheduler/mainnet/status.json \
  --datasets-url https://cdn.subsquid.io/sqd-network/datasets.yml \
  --datasets-config /cfg/datasets.yaml \
  --retain-delay-secs 600

This image publishes no latest tag. Pin an explicit version.

Flag	Required	Default	Purpose
`--hotblocks-url`	Yes	—	HotblocksDB instance that receives the lower bound
`--status-url`	Yes	—	Network scheduler status document
`--datasets-url`	Yes	—	Network datasets manifest, used to resolve names to dataset IDs
`--datasets-config`	Yes	—	Path to the mapping above
`--datasets-update-interval-secs`	No	`3600`	How often the datasets manifest is refreshed
`--retain-delay-secs`	No	`0`	Delay after an assignment’s effective time before its height is applied

On the testnet, use the tethys scheduler status and datasets-testnet.yml. As a Compose service alongside the rest of the stack:

compose.yaml

  hotblocks-retain:
    image: subsquid/data-hotblocks-retain:<version>
    command:
      - --hotblocks-url
      - http://hotblocks-db:8081
      - --status-url
      - https://metadata.sqd-datasets.io/scheduler/mainnet/status.json
      - --datasets-url
      - https://cdn.subsquid.io/sqd-network/datasets.yml
      - --datasets-config
      - /cfg/datasets.yaml
      - --retain-delay-secs
      - "600"
    volumes:
      - ./retained-datasets.yaml:/cfg/datasets.yaml:ro
    depends_on:
      - hotblocks-db
    restart: unless-stopped

Run one Hotblocks-retain per HotblocksDB instance using Api retention. Replicas ingest independently onto their own disks, so each needs its own bound pushed to it.

Choose a retain delay

--retain-delay-secs defaults to 0, which leaves no margin. Set it explicitly. Hotblocks-retain reads the archive head directly from the scheduler status. A Portal reaches the same head indirectly, through the assignment it refreshes on its own interval, so every Portal’s view of the archive trails Hotblocks-retain’s. The delay is the margin that covers that skew. Below it, the retention bound can pass a Portal’s archival ceiling and open a gap between the two sources while both services are healthy. A second reason not to run a tight margin: the scheduler height means a chunk has been indexed for assignment, not that every assigned worker has finished downloading it. 600 seconds is a reasonable starting point. It clears a Portal’s assignment refresh several times over while adding little to the retained window. Raise it if you see gaps; the cost of a generous delay is disk, and the ceiling below bounds that anyway.

Bound the window from above

The lower bound only advances when the archive advances. If archival ingestion stalls, the bound stops moving and the window grows for as long as the stall lasts. Cap it:

hotblocks.yaml

ethereum-mainnet:
  kind: evm
  retention_strategy:
    Api:
      max_blocks: 7200
  data_sources:
    - http://ethereum-data-service:3000

The retained window is then max(lower bound, head - max_blocks) .. head, so max_blocks bounds disk regardless of what the archive does. It is a soft cap: trimming drops whole chunks, so part of the first chunk can survive it. Size it from the dataset’s block rate — roughly a day of blocks is a common starting point — then check it against how often the archive publishes new chunks for that dataset. A window shorter than the interval between chunk publications trims below the archival head between them and opens a gap on a healthy system. Chains with slow or large chunks need a window wider than a day.

Verify

Hotblocks-retain applies a bound by posting to HotblocksDB, which turns an Api dataset’s runtime strategy from None into FromBlock. Read it back:

curl --fail --silent http://127.0.0.1:8081/datasets/ethereum-mainnet/retention

Before the first update, the dataset’s status carries no data either:

curl --fail --silent http://127.0.0.1:8081/datasets/ethereum-mainnet/status

{ "kind": "evm", "retentionStrategy": "None", "data": null }

Once a bound has been applied, data is populated and firstBlock advances on later polls:

curl --fail --silent http://127.0.0.1:8081/datasets/ethereum-mainnet/status \
  | jq '{retentionStrategy, firstBlock: .data.firstBlock, lastBlock: .data.lastBlock}'

Then confirm the two sources still meet. Portal names the source that answered in the x-sqd-data-source response header: network for archival workers, real_time for HotblocksDB. A request starting just above the archival head should return blocks under real_time — substitute your own fromBlock:

curl --fail --silent --dump-header - --output /dev/null \
  http://127.0.0.1:8000/datasets/ethereum-mainnet/stream \
  -H "content-type: application/json" \
  -d '{
    "type": "evm",
    "fromBlock": 25790715,
    "includeAllBlocks": true,
    "fields": { "block": { "number": true } }
  }'

An empty response means the request fell into a gap between the two sources.

Configure Portal routing

Declare the HotblocksDB URL inside every dataset’s real_time section:

mainnet.config.yml

hostname: http://0.0.0.0:8000
sqd_network:
  datasets: https://cdn.subsquid.io/sqd-network/datasets.yml
  metadata: https://cdn.subsquid.io/sqd-network/mainnet/metadata.yml
  serve: manual
datasets:
  ethereum-mainnet:
    real_time:
      kind: evm
      url: http://hotblocks-db:8081
  base-mainnet:
    real_time:
      kind: evm
      url: http://hotblocks-db:8081
  avalanche-mainnet:
    real_time:
      kind: evm
      url: http://hotblocks-db:8081

hotblocksDB is not a valid top-level Portal option. Unknown fields are ignored with a warning. Set datasets.<name>.real_time.url for every real-time dataset.

Portal uses the dataset name as the HotblocksDB dataset name by default. Set real_time.dataset only when the names differ. Portal resolves each dataset key against the Network manifest at sqd_network.datasets. A private dataset is not published in that manifest, so declare its ID:

mainnet.config.yml

datasets:
  my-private-chain:
    sqd_network:
      dataset_id: s3://my-private-chain-1
    real_time:
      kind: evm
      url: http://hotblocks-db:8081

Use dataset_name instead when the dataset is in the manifest under a name other than the config key.

Verify routing

curl --fail http://127.0.0.1:8000/datasets/ethereum-mainnet
curl --fail http://127.0.0.1:8000/datasets/base-mainnet
curl --fail http://127.0.0.1:8000/datasets/avalanche-mainnet

Each response should report real_time: true. Run historical and near-head stream queries to verify that both sources are reachable.

Put a custom network behind Portal

A network does not need an SQD Network archive to appear in Portal. Declare only a real-time source:

mainnet.config.yml

hostname: http://0.0.0.0:8000
sqd_network:
  datasets: https://cdn.subsquid.io/sqd-network/datasets.yml
  metadata: https://cdn.subsquid.io/sqd-network/mainnet/metadata.yml
  serve: manual
datasets:
  krown-network:
    real_time:
      kind: evm
      url: http://hotblocks-db:8081

The corresponding krown-network dataset must exist in HotblocksDB. Choose the client endpoint based on the required API:

Client need	Endpoint
Real-time streams and head information only	Connect directly to HotblocksDB
One endpoint for the Network and custom datasets	Connect through Portal
Portal metadata, dataset listing, or timestamp lookup endpoints	Connect through Portal

Squid SDK and Pipes SDK clients can use either endpoint for supported streaming operations.

Token requirements and compute units

The minimum token locking requirement for a Portal instance is 1,000,000 SQD. This should be enough for most use cases. If you want to know the exact capabilities this gives you, read on. The rate limiting mechanism of SQD Network relies on the concept of a compute unit, or CU for short. CUs do not map directly to the amount of data fetched by a Portal instance; instead, they (roughly) represent the amount of work that the network does internally while serving the Portal instance’s requests.

Understanding compute units (CU)

Network datasets are partitioned by block number. Dataset chunks are randomly distributed among worker nodes.When a Portal instance receives a data request, the following happens:

The Portal instance forwards the request to several workers that hold the chunks of the relevant dataset
The workers execute the request separately on each chunk
Workers send the results back to the Portal instance. For lightweight queries, they send one response for each dataset chunk; however, if any response exceeds 100 Mbytes, it’s split into several parts
The Portal instance concatenates the workers’ replies and serves a continuous stream of data to the user

Each response made by any of the workers in step 3 spends exactly 1 CU.

The more SQD you lock and the greater the lockup period is, the more CUs you get. Currently, each locked SQD generates 1-3 CUs at the beginning of each epoch, depending on the lockup period. In principle, any valid amount of locked SQD generates an infinite amount of CUs. However, if the rate at which your queries consume CUs exceeds the rate at which they are produced, your app will be throttled. To avoid that, you may want to understand how many CUs your queries spend. There is currently no tool for estimating the number of CUs a query needs before running it. However, on EVM you can use the following formula to get an order of magnitude estimate:

10^-5 * range_length_blocks * transactions_per_block

Notes:

This assumes lightweight queries - that is, queries that fetch much less data than the total dataset size. For heavyweight queries multiply the estimate by a factor of 2-5.
This is a rough estimate. Multiply it by ten to get a somewhat safe figure. If you want to minimize your SQD lockup, start at that safe figure, then measure the actual amount of CUs you spend and reduce the lockup accordingly.
If your network has an Etherscan-style explorer, you can estimate the transactions_per_block by visiting its front page, reading the “Transactions” stat and dividing it by the “Last finalized block” height.

How the estimate is calculated

For a lightweight query, the amount of CUs spent is determined by how many dataset chunks the network needs to examine to process it. The ingester creates chunks of roughly the same size within each dataset. Since the amount of data per block is roughly proportional to the number of transactions in that block, we can assume that the number of chunks in any given range is proportional to the number of transactions per block.Extrapolating from the Ethereum dataset:

num_chunks_in_range = range_length * (eth_chunks / eth_height) * (chain_txs_per_block / eth_txs_per_block)

Where:

eth_chunks = 3.1e4
eth_height = 2.1e7
eth_txs_per_block = 1.2e2

Multiplying all the known values together and rounding to one decimal digit, we get the 1e-5 coefficient of the final formula.Important assumption: This assumes all EVM datasets have the same chunk size as Ethereum. In reality, chunk sizes vary between 50-1000 Mbytes. Ethereum’s chunk size is roughly 500 Mbytes, so expect the estimate to be off by a factor of 0.5-10, which is within the “order of magnitude” definition.Heavyweight queries: Scale the same way but may spend more than one CU per chunk. The heaviest possible queries (fetching the whole dataset) on Ethereum consume roughly 5 CUs per chunk.

Now, if your queries consume X CUs each and you run them once per Y epochs, you need to lock up at least this much SQD:

X / (Y * boost_factor)

Here, boost_factor is a multiplier ranging from 1 to 3 depending on the lockup length.

Redundancy and capacity

Run one EVM data service per RPC provider and list all providers under the dataset’s data_sources.
Run multiple Portal replicas behind a load balancer. Portal is stateless. Replicas can share a single wallet and a single SQD lock: register one peer ID per replica from the same wallet.
Give every HotblocksDB replica its own persistent volume. Replicas ingest independently and do not share RocksDB files.
Run Hotblocks-retain alongside every HotblocksDB replica that uses Api retention.
Pin tested container versions or digests for production instead of tracking latest automatically.
Protect Portal peer keys, RPC credentials, and internal service ports.
Monitor /ready, /metrics, dataset heads, disk usage, and RPC error rates.

Troubleshooting

What are the consequences of losing my key file / getting it stolen?

If you lose your key file, you won’t be able to run your Portal instance until you generate and register a new one. If your key file is stolen, the perpetrator can cause connectivity issues, effectively creating downtime for your Portal instance. To recover:

Unregister your Portal instance on the portals page.
Generate a new key file.
Register the new portal peer ID.

Portal ignores the real-time source

Check startup logs for:

ignoring unknown config field

Remove a top-level hotblocksDB field and set datasets.<name>.real_time.url.

HotblocksDB does not ingest an `Api` dataset

Api retention does not start until the first retention update arrives. Check /datasets/<name>/status on HotblocksDB: "retentionStrategy": "None" with "data": null means no update has landed yet. Confirm Hotblocks-retain is running, that the dataset key in its config exactly matches the HotblocksDB dataset name, and that it can reach the HotblocksDB, status, and datasets URLs. It logs and retries on every failed fetch, so a repeating failed to fetch status or failed to refresh datasets manifest in its logs is the answer. dataset not found in manifest, skipping means the name does not resolve to a Network dataset. For a private dataset this is expected — it is not published in the manifest, so set id for that entry. Otherwise set name.

Clients get no data near the chain head

The requested start is above the archival head but below HotblocksDB’s first retained block, so neither source covers it. Portal reports this as no data rather than skipping ahead to the first retained block. See Reliability and monitoring. Compare firstBlock from /datasets/<name>/status against the archival head Portal is working from. If the former is higher, the retention bound has overtaken Portal: raise --retain-delay-secs, and check that max_blocks is wider than the dataset’s chunk publication interval.

Traces or state diffs are missing

Confirm the EVM data service was started with the matching flags and the RPC supports debug_traceBlockByHash. Do not enable expensive fields that clients do not query.

Disk grows continuously

Inspect retention_strategy. FromBlock retains every block from the configured height. Use a bounded Head window for real-time-only access or Api retention when the Network supplies history.

One RPC failure stops current data

Run a data service for a second provider and add both service URLs to the HotblocksDB dataset. Test provider failure before relying on the setup in production.

The portal is slower than expected

Portal performance metrics are exposed at the /metrics endpoint. Check the throttling statistics:

curl --compressed http://127.0.0.1:8000/metrics | grep throttled

Lower values of portal_stream_throttled_ratio_sum indicate better performance. If throttling is high, consider locking more SQD tokens to increase your compute unit allocation.

​Choose a topology

​Components and service count

​Prerequisites

​Archival-only setup

​Real-time-only setup for a custom EVM network

​Configure retention

​Run the services

​Verify the custom dataset

​Archival plus real-time setup

​Configure HotblocksDB

​Configure Hotblocks-retain

​List the datasets to track

​Run the service

​Choose a retain delay

​Bound the window from above

​Verify

​Configure Portal routing

​Verify routing

​Put a custom network behind Portal

​Token requirements and compute units

​Redundancy and capacity

​Troubleshooting

​What are the consequences of losing my key file / getting it stolen?

​Portal ignores the real-time source

​HotblocksDB does not ingest an Api dataset

​Clients get no data near the chain head

​Traces or state diffs are missing

​Disk grows continuously

​One RPC failure stops current data

​The portal is slower than expected

​Source and API references

Choose a topology

Components and service count

Prerequisites

Archival-only setup

Real-time-only setup for a custom EVM network

Configure retention

Run the services

Verify the custom dataset

Archival plus real-time setup

Configure HotblocksDB

Configure Hotblocks-retain

List the datasets to track

Run the service

Choose a retain delay

Bound the window from above

Verify

Configure Portal routing

Verify routing

Put a custom network behind Portal

Token requirements and compute units

Redundancy and capacity

Troubleshooting

What are the consequences of losing my key file / getting it stolen?

Portal ignores the real-time source

HotblocksDB does not ingest an `Api` dataset

Clients get no data near the chain head

Traces or state diffs are missing

Disk grows continuously

One RPC failure stops current data

The portal is slower than expected

Source and API references