Skip to main content

ApeWorx SQD plugin

info

The subsquid ApeWorx plugin is currently in beta. Please report any bugs or suggestions to Squid Devs or open an issue at the GitHub repo of the plugin.

ApeWorx is a modular Web3 development framework for Python programmers. Among other things, it is capable of retrieving blockchain data in bulk. The data can come from various sources, including SQD Network.

The network provides free access to blocks and event logs data. On long block ranges (>1k blocks) data retrieval is orders of magnitude (>10x) faster compared to RPC-based data sources.

Basic usage​

In an existing ApeWorx installation, run the following to install the subsquid plugin:

ape plugins install "ape-subsquid@git+https://github.com/subsquid/ape-subsquid.git@main"

To source data from SQD Network, start ape console as usual, e.g. with

ape console --network ethereum:mainnet:geth

then pass

engine_to_use='subsquid'

to your data query methods. You can speed up the following calls:

  1. Bulk block data retrieval with chain.blocks.query:

    df = chain.blocks.query(
        '*',
        start_block=18_000_000,
        stop_block=19_000_000,
        engine_to_use='subsquid'
    )

    This query retrieves data on 1M blocks in about 11 minutes.

  2. Contract events retrieval:

    contract = Contract('0xdac17f958d2ee523a2206206994597c13d831ec7', abi='usdt.json')
    df = contract.Transfer.query(
        '*',
        start_block=18_000_000,
        stop_block=18_100_000,
        engine_to_use='subsquid'
    )

    This query retrieves 1.6M events emitted over 100k block in about 17 minutes.

warning

At the moment, all SQD Network datasets are updated only once every several thousands of blocks. The current dataset height can be retrieved with get_network_height():

from ape_subsquid import get_network_height

get_network_height() # currently 19212115 while the chain is at 19213330

Queries that request blocks above the current dataset height will fail with an ape_subsquid.exceptions.DataRangeIsNotAvailable exception. That includes queries without an explicitly set stop_block. If you don't need the recent data, you can explicitly request the data up to the block height of the SQD dataset, e.g.

subsquid_height = get_network_height()
df = chain.blocks.query(
    '*',
    start_block=19_000_000,
    stop_block=subsquid_height,
    engine_to_use='subsquid'
)

To get the latest data, retrieve the tail with the default engine and append it to the dataframe:

taildf = chain.blocks.query('*', start_block=subsquid_height+1)
df = pd.concat([df, taildf], ignore_index=True, copy=False)
info

When working with block ranges much longer than 1M blocks, the plugin may occasionally fail due to HTTP 503 errors rarely returned by the network. If you encounter this issue, split your block range into sub-1M blocks intervals and retrieve the data for each interval separately, retrying when the queries throw exceptions.

Networks support​

The following networks available via plugins from ape plugins list --all are supported:

  • arbitrum
  • avalanche
  • base
  • bsc
  • ethereum
  • fantom
  • optimism
  • polygon
  • polygon-zkevm

Extra functionality​

Low-level queries​

The plugin supports the following low-level queries:

The following queries are NOT supported:

Account history retrieval​

The plugin can be used to retrieve historical data for individual accounts, although this is much slower than working with specialized APIs such as Etherscan.

accountHistory = chain.history._get_account_history('vitalik.eth')
accountHistory.query('value', engine_to_use='subsquid')