Run a Worker - SQD Documentation

We strongly advise against using Hetzner for running SQD Network worker nodes due to reliability concerns.

Running a worker node allows you to contribute storage and compute resources to SQD Network and earn SQD token rewards.

Prerequisites

Before you begin, ensure you have the following:

Hardware Requirements

CPU: 4 vCPU
RAM: 16GB
Storage: 1TB SSD
Network: Stable 24/7 Internet connection (minimum 1Gbit)

Network Configuration

Public IP address
Two ports open for incoming traffic:
- UDP port for P2P communication (default: 12345)
- TCP port for Prometheus metrics (default: 9090)

Software Requirements

Docker
Docker Compose

Financial Requirements

100,000 SQD tokens (in your wallet or vesting contract)
Arbitrum ETH for gas fees

The SQD tokens should be available in your Primary Wallet, either directly or through a vesting contract. With sufficient funds, one Primary Wallet can register multiple workers.

Use a browser-compatible wallet like MetaMask for your Primary Wallet.

Setup Options

You can run a worker using either:

Pre-built Docker image (recommended)
Build from source code

Docker is required for both options as the configuration script uses it.

Setup Instructions

Choose data and key storage locations

Choose your storage locations:

Data directory - Select a location that can accommodate at least 1TB
Key file location - Must be outside the data directory

Protect your key file! Ensure it won’t be deleted accidentally and cannot be accessed by unauthorized parties. See consequences of losing your key.

Do not create the data folder manually, the setup script will create it.

Download the setup script

Create a new directory for installation files and download the setup script:

curl -fO https://cdn.subsquid.io/worker/setup_worker.sh

Make the script executable:

chmod +x ./setup_worker.sh

Setup script is ready to run.

Run the setup script

Execute the setup script with your chosen paths:

./setup_worker.sh <DATA_DIR_PATH> <KEY_PATH>

The script will:

Prompt you for a UDP port for P2P communication
Ask if you want to set your public IP address statically

Most users should not set the IP address manually, automatic IP discovery is more robust.

What the script creates:

Data directory at <DATA_DIR_PATH>
Key file at <KEY_PATH>
.env file with default configuration
docker-compose.yaml file for running the worker

View all files the script downloads on GitHub.

Copy your peer ID

The script’s output will end with a line like this:

Your peer ID is: 12D3KooWPfotj6qQapKeWg4RZysUjEeswLEpiSjgv16LgNTyYFTW. Now you can register it on chain.

Copy your peer ID, you’ll need it for on-chain worker registration.

Setup complete! You’re ready to register your worker.

Optional: Customize configuration

If needed, edit the generated .env file to:

Set a custom Prometheus port
Use your own RPC provider
Adjust other configuration parameters

Most users can skip this step and use the default configuration.

Worker Registration

By registering a worker, you lock your tokens in the smart contract. Once you want to withdraw them, you’ll have to wait for approximately 14 days (100,000 Ethereum blocks) plus the time until the end of the current epoch (up to 20 minutes).

Before running your worker node, you must register it on-chain.

Connect your wallet

Go to network.subsquid.io
Connect your Primary Wallet (the one holding your 100,000 SQD tokens)

Your wallet address should appear in the interface.

Access worker registration

Navigate to the “Workers” tab and click the “Add Worker” button.The worker registration form will appear.

Fill the registration form

Complete the registration form:

Funding source: Choose either:
- “Wallet” - Use SQD tokens directly from your wallet
- “Vesting contract” - Use SQD tokens from a vesting contract
Peer ID: Enter the peer ID you copied during setup
Submit the form by confirming the transaction in your wallet

Double-check your peer ID before submitting, incorrect IDs will prevent your worker from functioning properly.

Wait for activation

Go to the “Workers” tab
Monitor your worker’s status

Your worker will change to “Offline” or “Active” status at the beginning of the next epoch (within approximately 20 minutes).

Worker registered! Proceed to running your worker once the status updates.

Running Your Worker

Ensure you’re in the folder where you executed setup_worker.sh during the setup process.

Docker (Recommended)
Build from Source

Run with Docker Compose

Start your worker in detached mode:

docker compose up -d

Monitor Logs

Check the worker container logs:

docker compose logs

After a few minutes, the worker should begin outputting information about downloaded data chunks.

Use docker compose logs -f to follow logs in real-time.

Install Prerequisites

Install required dependencies (Rust, Git, Protobuf, etc.):

apt install curl git build-essential pkg-config protobuf-compiler libssl-dev libsqlite3-dev
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Load Configuration

Load the configuration files generated during setup:

source .env

Clone the Worker Repository

Clone the worker repository (this may take a few minutes):

git clone --depth 1 https://github.com/subsquid/worker-rs.git

Build the Worker

Navigate to the repository and build:

cd worker-rs
cargo build --release

Run the Worker

Start the worker:

cargo run --release

Consider daemonizing with systemd or openrc to ensure the worker restarts automatically after server reboots.

After some time, the worker will start downloading data and serving queries.

Updating Your Worker

Periodically, you’ll need to update your worker to the latest version. Follow these steps:

Always back up your key file before updating.

Docker Update
Source Build Update

Navigate to your installation files folder
Fetch the updated docker-compose.yaml:

curl -fO https://cdn.subsquid.io/worker/docker-compose.yaml

Apply the update:

docker compose up -d

Update complete! Your worker is now running the latest version.

Note the new worker version from the announcement
Navigate to your installation files folder
Enter the worker-rs subfolder
Fetch new commits:

git fetch --unshallow

Check out the new version:

git checkout <new_version_tag>

Rebuild the worker:

cargo build --release

Restart the worker process

Update complete! Your worker is now running the latest version.

You don’t need to erase the data folder or re-register your worker during updates.

Some releases may require deviations from this procedure. Always read release notes carefully.

Understanding Worker Jailing

Jailing is a scheduler-side mechanism that ensures data chunk availability across the network. The scheduler identifies potentially unreliable workers and guarantees that each chunk exists on multiple reliable nodes.

Jailed workers continue serving queries from downloaded chunks and earning rewards. Jailing is temporary, workers are automatically unjailed within approximately 20 minutes once they’re reliable again.

Impact on Rewards

While occasional jailing is normal, frequent jailing may reduce your rewards. Monitor your worker’s status and address any underlying issues.

Common Jailing Reasons

Check your worker logs to identify which issue applies: Stale Worker - Worker didn’t download assigned chunks in the last 900 seconds

Most common jailing reason
Check worker logs for the root cause
Often related to download timeouts

Inactive Worker - Worker didn’t send pings for over 600 seconds

May occur after extended downtime
Scheduler considers the worker offline and stops assigning chunks

Unreachable Worker (Currently disabled)

Was used to detect P2P network connectivity issues
Implementation challenges led to this check being disabled

If your worker stays online consistently and maintains good connectivity, it’s unlikely to be jailed.

Troubleshooting

Where do I find my peer ID?

Your peer ID can be found in several places:

Setup script output - Printed when you run setup_worker.sh (see Setup Instructions)
Worker logs:
- Docker setup: Run docker compose logs
- Source build: Check the output of the worker process

Look for a log line like this:

2024-05-28T07:43:55.591440Z  INFO subsquid_worker::transport::p2p: Local peer ID: <PEER_ID>

I see "Failed to download chunk ... operation timed out" in the worker logs

This error indicates connection issues. Adjust timeout settings based on your connection quality:If you encounter this frequently:

Set S3_TIMEOUT to 180 in your .env file
If the issue persists, also set:
- CONCURRENT_DOWNLOADS to 1
- S3_READ_TIMEOUT to 30

These adjustments reduce concurrent downloads and increase timeout thresholds, helping with slower or less stable connections.

I see "Unable to get assignment: deadline has elapsed" in the worker logs

This is a known issue in version 2.0.2.Solution:Set ASSIGNMENT_FETCH_TIMEOUT_SEC=90 in your .env file and restart the worker.

I see an "Insufficient peers..." error in the worker logs

This error can be safely ignored. It doesn’t affect worker functionality or rewards.

Can I move my worker to another server?

Yes! You can migrate your worker to a new server:

Copy the key file (at <KEY_PATH>) to the new server
Set up the new data directory
Start the worker on the new server
Make sure the worker on the old server is not running anymore

You don’t need to re-register your worker when moving to a new server.

I have just started my worker but see no logs

This is normal behavior. Wait a few minutes, the logs will start showing data download activity once the worker begins syncing.

Should I build from source or run with Docker?

Docker (Recommended) - Easier setup, suitable for most usersBuild from Source - Suitable for experienced Linux/Unix users who need more control or customization

Choose Docker unless you have specific requirements that necessitate building from source.

How do I check that my worker is updated to the latest version?

Find your worker in the network app and look at the “version” column. If the version is outdated, there will be a warning sign besides it.

Which Linux distro is recommended?

We recommend Ubuntu 22.04 LTS for optimal compatibility and stability.

I see "error from daemon in stream: Error grabbing logs"

This is a Docker issue, not a worker problem.Resources for resolution:

I'm getting 'Clone succeeded, but checkout failed' when cloning the worker repo

Manually check out the correct version:

cd worker-rs
git checkout v2.0.0

What are the consequences of losing my key file / getting it stolen?

If you lose your key file:

You won’t be able to run your worker until you generate and register a new one

If your key file is stolen:

The perpetrator can cause connectivity issues for your worker

Recovery steps:

Unregister your worker on the “Workers” tab
Generate a new key file
Register your new peer ID

Protect your key file! Store it securely and restrict access to prevent these issues.

​Prerequisites

​Hardware Requirements

​Network Configuration

​Software Requirements

​Financial Requirements

​Setup Options

​Setup Instructions

​Worker Registration

​Running Your Worker

​Run with Docker Compose

​Monitor Logs

​Install Prerequisites

​Load Configuration

​Clone the Worker Repository

​Build the Worker

​Run the Worker

​Updating Your Worker

​Understanding Worker Jailing

​Impact on Rewards

​Common Jailing Reasons

​Troubleshooting

Prerequisites

Hardware Requirements

Network Configuration

Software Requirements

Financial Requirements

Setup Options

Setup Instructions

Worker Registration

Running Your Worker

Run with Docker Compose

Monitor Logs

Install Prerequisites

Load Configuration

Clone the Worker Repository

Build the Worker

Run the Worker

Updating Your Worker

Understanding Worker Jailing

Impact on Rewards

Common Jailing Reasons

Troubleshooting