Substrate SQD Network API

warning

The Substrate API of SQD Network is currently in beta. Breaking changes may be introduced in the future releases.

SQD Network API distributes the requests over a (potentially decentralized) network of workers. The main gateway URL points at a router that provides URLs of workers that do the heavy lifting. Each worker has its own range of blocks on each dataset it serves.

Suppose you want to retrieve an output of some query on a block range starting at firstBlock (can be the genesis block) and ending at the highest available block. Proceed as follows:

1. Retrieve the dataset height from the router with GET /height and make sure it's above firstBlock.

2. Save the value of firstBlock to some variable, say currentBlock.

3. Query the router for an URL of a worker that has the data for currentBlock with GET /${currentBlock}/worker.

4. Retrieve the data from the worker by posting the query (POST /), setting the "fromBlock" query field to ${currentBlock}.

5. Parse the retrieved data to get a batch of query data plus the height of the last block available from the current worker. Take the header.number field of the last element of the retrieved JSON array - it is the height you want. Even if your query returns no data, you'll still get the block data for the last block in the range, so this procedure is safe.

6. Set currentBlock to the height from the previous step plus one.

7. Repeat steps 3-6 until all the required data is retrieved.

Main URLs of EVM gateways are available on the Supported networks page.

Implementation example:

In Python

def get_text(url: str) -> str:
    res = requests.get(url)
    res.raise_for_status()
    return res.text


def dump(
    gateway_url: str,
    query: Query,
    first_block: int,
    last_block: int
) -> None:
    assert 0 <= first_block <= last_block
    query = dict(query)  # copy query to mess with it later

    dataset_height = int(get_text(f'{gateway_url}/height'))
    next_block = first_block
    last_block = min(last_block, dataset_height)

    while next_block <= last_block:
        worker_url = get_text(f'{gateway_url}/{next_block}/worker')

        query['fromBlock'] = next_block
        query['toBlock'] = last_block
        res = requests.post(worker_url, json=query)
        res.raise_for_status()
        blocks = res.json()

        last_processed_block = blocks[-1]['header']['number']
        next_block = last_processed_block + 1
        for block in blocks:
            print(json.dumps(block))

Full code here.

Router API

GET /height (get height of the dataset)

Example response: 20856599.

GET ${firstBlock}/worker (get a suitable worker URL)

The returned worker is capable of processing POST / requests in which the "fromBlock" field is equal to ${firstBlock}.

Example response: https://rb05.sqd-archive.net/worker/query/czM6Ly9wb2xrYWRvdC00.

Worker API

POST / (query Substrate data)

Query Fields

• type: "substrate"
• fromBlock: Block number to start from (inclusive).
• toBlock: (optional) Block number to end on (inclusive). If this is not given, the query will go on for a fixed amount of time or until it reaches the height of the dataset.
• includeAllBlocks: (optional) If true, SQD Network workers will include blocks that contain no data selected by data requests into their responses.
• fields: (optional) A selector of data fields to retrieve. Common for all data items.
• events: (optional) A list of event requests.
• calls: (optional) A list of call requests.
• contractsEvents: (optional) A list of Contracts.ContractEmitted event requests.
• evmLogs: (optional) A list of EVM.Log event requests.
• ethereumTransactions: (optional) A list of Ethereum.transact call requests.
• gearMessagesQueued: (optional) A list of Gear.MessageQueued event requests.
• gearUserMessagesSent: (optional) A list of Gear.UserMessageSent event requests.

The response is a JSON array of per-block data items that covers a block range starting from fromBlock. The last block of the range is determined by the worker. You can find it by looking at the header.number field of the last element in the response array.

The first and the last block in the range are returned even if all data requests return no data for the range.

In most cases the returned range will not contain all the range requested by the user (i.e. the last block of the range will not be toBlock). To continue, retrieve a new worker URL for blocks starting at the end of the current range plus one block and repeat the query with an updated value of fromBlock.

Example Request

{
  "type": "substrate",
  "fromBlock": 4669000,
  "toBlock":4669010,
  "fields": {
    "event": {
      "name": true,
      "args": true
    },
    "call": {
      "name": true,
      "args": true   
    }               
  },
  "events": [
    {        
      "name": ["Balances.Transfer"] 
    }              
  ]              
}

Run

curl https://v2.archive.subsquid.io/network/<your-network>/4669000/worker

to get an URL of a worker capable of processing this query.

Example Response

Note that the last block in the range is included despite having no matching events.

[
  {
    "header": {
      "number": 4669000,
      "hash": "0xa4667263922a1f71708993dc923b974bdece3a117538d3654f44ace403e6614f",
      "parentHash": "0x068d9e6dc7f3245df45a00a6a18ed1a07e64a53997f5f3f89e8b09c9db267b2b"
    },
    "events": []
  },
  {
    "header": {
      "number": 4669005,
      "hash": "0x7ae89bccf9d8a3fcb33b9310bff5d83aaf905099e32dd7766443c9b96143cde9",
      "parentHash": "0x36c515ee7a74db78a4ddee5734e8a79440e08bafd7e440886c7fcfd0d6389088"
    },
    "events": [
      {
        "index": 1,
        "extrinsicIndex": 1,
        "callAddress": [],
        "name": "Balances.Transfer",
        "args": [
          "0x3a7b188d341fcd76ffdc8e684ac26c1e0720e35ca01b3f7c2308c3bde14571c2",
          "0x8cdbbe675b1ea872e9f4b1d1f7258c3757b4247ef4e4d8f5d3a600d2a6dc7e59",
          "378293330000"
        ]
      }
    ]
  },
  {
    "header": {
      "number": 4669010,
      "hash": "0xbfd4448702ab2def722c6638c3d2062b7ca2cd62f71801ab467b1484bcb259a6",
      "parentHash": "0x5cb9bfcd169b9cacfa8f3b58212c151fc69e5cadf661162cd88f221888420942"
    },
    "events": []
  }
]

Data requests

Events

{
  name: string[],
  call: boolean,
  stack: boolean,
  extrinsic: boolean
}

An event will be included in the response if it matches all the requests. A request with an empty array (e.g. { name: [] }) matches no events; omit all requests/pass an empty object to match all events.

See addEvent() SDK function reference for a detailed description of the fields of this data request; also see Field selection.

Calls

{
  name: string[],
  subcalls: boolean,
  extrinsic: boolean,
  stack: boolean,
  events: boolean
}

A call will be included in the response if it matches all the requests. A request with an empty array (e.g. { name: [] }) matches no calls; omit all requests/pass an empty object to match all calls.

See addCall() SDK function reference for a detailed description of the fields of this data request; also see Field selection.

Contracts.ContractEmitted events

info

Contract addresses supplied with this data request must be hexadecimal (i.e. decoded from SS58) and lowecased. Addresses in all responses will be in the same format.

{
  contractAddress: string[],
  call: boolean,
  stack: boolean,
  extrinsic: boolean
}

A Contracts.ContractEmitted event will be included in the response if it matches all the requests. A request with an empty array (e.g. { contractAddress: [] }) matches no events; omit all requests/pass an empty object to match all events.

See addContractsContractEmitted() SDK function reference for a detailed description of the fields of this data request; also see Field selection and ink! contracts support.

EVM.Log events

info

Contract addresses supplied with this data request must be in lowercase. Addresses in all responses will be in the same format.

{
  address: string[],
  topic0: string[],
  topic1: string[],
  topic2: string[],
  topic3: string[],
  call: boolean,
  stack: boolean,
  extrinsic: boolean
}

An EVM.Log event will be included in the response if it matches all the requests. A request with an empty array (e.g. { topic0: [] }) matches no events; omit all requests/pass an empty object to match all events.

See addEvmLog() SDK function reference for a detailed description of the fields of this data request; also see Field selection and Frontier EVM support.

Ethereum.transact calls

info

Contract addresses supplied with this data request must be in lowercase. Addresses in all responses will be in the same format.

{
  to: string[],
  sighash: string[],
  extrinsic: boolean,
  stack: boolean,
  events: boolean
}

An Ethereum.transact call will be included in the response if it matches all the requests. A request with an empty array (e.g. { sighash: [] }) matches no calls; omit all requests/pass an empty object to match all calls.

See addEthereumTransaction() SDK function reference for a detailed description of the fields of this data request; also see Field selection and Frontier EVM support.

Gear.MessageQueued events

{
  programId: string[],
  call: boolean,
  stack: boolean,
  extrinsic: boolean
}

A Gear.MessageQueued event will be included in the response if it matches all the requests. A request with an empty array (e.g. { programId: [] }) matches no events; omit all requests/pass an empty object to match all events.

See addGearMessageQueued() SDK function reference for a detailed description of the fields of this data request; also see Field selection and Gear support.

Gear.UserMessageSent events

{
  programId: string[],
  call: boolean,
  stack: boolean,
  extrinsic: boolean
}

A Gear.UserMessageSent event will be included in the response if it matches all the requests. A request with an empty array (e.g. { programId: [] }) matches no events; omit all requests/pass an empty object to match all events.

See addGearUserMessageSent() SDK function reference for a detailed description of the fields of this data request; also see Field selection and Gear support.

Data fields selector

A JSON selector of fields for the returned data items. Documented in the Field selectors section.