> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sqd.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# From blocks to chunks

> How finalized blockchain data reaches chunk storage and worker disks.

Workers do not ingest every supported blockchain directly. Separate ingestion
services read blocks from configured sources and build the archival data that
workers serve.

```mermaid theme={"system"}
flowchart LR
  source["Configured block source"] --> ingest["Archive ingestion"]
  ingest --> chunk["Immutable block-range chunk"]
  chunk --> object["Persistent object storage"]
  object --> scheduler["Scheduler discovers chunk"]
  scheduler --> assignment["Assignment"]
  assignment --> worker["Worker downloads chunk"]
  worker --> local["Worker local disk"]
```

## 1. Read blocks from a source

The archive service requests a block range from a configured ingestion source.
It reads line-delimited block records, advances from the last block it received,
and retries after a source error. Chain-continuity validation can check that each
block's parent hash matches the previous block's hash.

The current archive implementation supports EVM, Solana, Bitcoin, Tron, and two
Hyperliquid dataset types. Other historical datasets can use different ingestion
programs while still producing chunks for the same Network path.

## 2. Build a chunk

A **chunk** is an immutable range of finalized blocks. The archive service groups
records into tables and writes the chunk files to persistent storage. The current
worker and query implementations read these tables from Parquet files.

Chunks are the unit used for storage placement and archival queries. A query for a
large block range can therefore cover more than one chunk.

## 3. Publish storage metadata

The scheduler scans the configured dataset locations for chunks it has not seen
before. For each chunk it records the dataset, block range, size, and file
locations needed to create an assignment.

The assignment does not copy the chunk data itself. It tells each worker which
chunks it should hold and where to download their files.

## 4. Reconcile worker storage

Each worker polls the Network state for a new assignment. When the assignment
changes, the worker compares its desired chunks with its local files:

* missing assigned chunks are downloaded from persistent storage;
* chunks in the assignment become available for local queries after download;
* chunks that are no longer assigned can be removed from local storage.

Portal reads the same assignment family for routing information. Client queries
are sent to workers; clients do not download archival chunks from object storage.

<Info>
  This page describes finalized archival data. Recent, fork-sensitive blocks use
  the separate path described in [How real-time data is
  served](/en/network/introduction/how-real-time-data-is-served).
</Info>

## Implementation sources

* [Archive and query crates](https://github.com/subsquid/data)
* [Worker assignment and storage code](https://github.com/subsquid/worker-rs)
* [Scheduler chunk discovery](https://github.com/subsquid/network-scheduler)

Continue with [Scheduling and
replication](/en/network/introduction/scheduling-and-replication) to see how
workers are selected for each chunk.


## Related topics

- [Private Portal setup](/en/portal/self-hosting.md)
- [How an archival query is served](/en/network/introduction/how-a-query-is-served.md)
- [SQD Network components](/en/network/introduction/network-components.md)
- [How real-time data is served](/en/network/introduction/how-real-time-data-is-served.md)
- [How it works](/en/sdk/squid-sdk/substrate/how-it-works.md)