> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sqd.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Reliability and monitoring

> How replication, retries, status, and metrics expose Network health.

SQD Network combines multiple chunk copies with request retries. These mechanisms
reduce the effect of a single worker failure, but they do not guarantee that every
dataset is always available. Availability still depends on valid assignments,
reachable workers, complete chunk downloads, and the services around them.

## What happens when a component changes

| Event                                            | Current behavior                                                                                                                                                               |
| ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| One worker fails a query                         | Portal can retry an assigned copy on a different worker. Temporary worker penalties and backoff are kept in Portal memory.                                                     |
| No assigned worker can serve a chunk             | Portal returns a defined error. It does not skip the missing range or fabricate a result.                                                                                      |
| A worker receives a new assignment               | It downloads missing chunks, reports its storage state, and removes chunks that are no longer desired.                                                                         |
| Portal cannot refresh the assignment             | It keeps serving from the last applied assignment. The current specification notes that staleness is not yet bounded during a publisher outage.                                |
| Portal restarts                                  | It rebuilds its in-memory view from the assignment, dataset catalog, and contract state. Portal has no durable local routing state.                                            |
| HotblocksDB fails before a response starts       | Portal can replay one connection-class failure before response headers arrive. If that attempt fails, the recent-data request fails; archival worker traffic remains separate. |
| HotblocksDB fails after streaming starts         | Portal does not replay the request because that could duplicate records. The response ends and the client resumes from its last accepted block.                                |
| A gap exists between the archive and HotblocksDB | Portal returns no data when the requested start is above the archival head but below HotblocksDB's retained window. It does not skip the gap.                                  |

<Warning>
  Replication only helps when another assigned worker has completed the chunk
  download and can answer the query. A configured replication count is not proof
  that every copy is currently reachable.
</Warning>

## Worker status

The current worker exposes an HTTP service on its metrics port:

```http theme={"system"}
GET /worker/status
GET /worker/peer-id
GET /metrics
```

`/worker/status` reports how many assigned chunks are available or downloading.
`/metrics` exposes Prometheus metrics for storage, downloads, queries, assignment
state, and peer-to-peer activity.

The [worker guide](/en/network/worker) owns the setup, update, backup, and
troubleshooting procedures. This page only explains how the signals fit into the
Network.

## Network observers and collectors

The observer joins the peer-to-peer network and exposes metrics including:

* when a peer was last seen;
* peer ping count and last ping duration;
* a worker's reported storage use and missing chunk count;
* the assignment timestamp reported by the worker.

Separate collectors receive worker heartbeats and query logs over the Network
transport and store them in ClickHouse. Portal also exposes `/ready` and `/metrics`.
Its specification defines request, dependency, worker-selection, and assignment
signals, while its gap register identifies signals that are not implemented yet.

```mermaid theme={"system"}
flowchart LR
  worker["Worker"] -->|"heartbeats and metrics"| collectors["Collectors"]
  worker -->|"peer discovery and pings"| observer["Network observer"]
  portal["Portal"] -->|"request and dependency metrics"| monitoring["Monitoring"]
  hotblocks["HotblocksDB"] -->|"ingestion, query, fork, and retention metrics"| monitoring
  collectors --> monitoring
  observer --> monitoring
```

## Implementation sources

* [Worker status, storage, and metrics](https://github.com/subsquid/worker-rs)
* [Network observer](https://github.com/subsquid/network-observer)
* [Ping and log collectors](https://github.com/subsquid/network-components)
* [Portal failure model and observability](https://github.com/subsquid/sqd-portal/tree/master/spec)

Return to [Network components](/en/network/introduction/network-components) or
continue with the operational [worker guide](/en/network/worker).


## Related topics

- [Monitoring](/en/cloud/resources/monitoring.md)
- [How an archival query is served](/en/network/introduction/how-a-query-is-served.md)
- [Data freshness monitoring](/en/sdk/pipes-sdk/evm/guides/advanced-topics/latency-monitoring.md)
- [Pricing FAQ](/en/cloud/pricing/faq.md)
- [Run a Worker](/en/network/worker.md)