How replication, retries, status, and metrics expose Network health.
SQD Network combines multiple chunk copies with request retries. These mechanisms
reduce the effect of a single worker failure, but they do not guarantee that every
dataset is always available. Availability still depends on valid assignments,
reachable workers, complete chunk downloads, and the services around them.
Portal can retry an assigned copy on a different worker. Temporary worker penalties and backoff are kept in Portal memory.
No assigned worker can serve a chunk
Portal returns a defined error. It does not skip the missing range or fabricate a result.
A worker receives a new assignment
It downloads missing chunks, reports its storage state, and removes chunks that are no longer desired.
Portal cannot refresh the assignment
It keeps serving from the last applied assignment. The current specification notes that staleness is not yet bounded during a publisher outage.
Portal restarts
It rebuilds its in-memory view from the assignment, dataset catalog, and contract state. Portal has no durable local routing state.
HotblocksDB fails before a response starts
Portal can replay one connection-class failure before response headers arrive. If that attempt fails, the recent-data request fails; archival worker traffic remains separate.
HotblocksDB fails after streaming starts
Portal does not replay the request because that could duplicate records. The response ends and the client resumes from its last accepted block.
A gap exists between the archive and HotblocksDB
Portal returns no data when the requested start is above the archival head but below HotblocksDB’s retained window. It does not skip the gap.
Replication only helps when another assigned worker has completed the chunk
download and can answer the query. A configured replication count is not proof
that every copy is currently reachable.
The current worker exposes an HTTP service on its metrics port:
GET /worker/statusGET /worker/peer-idGET /metrics
/worker/status reports how many assigned chunks are available or downloading.
/metrics exposes Prometheus metrics for storage, downloads, queries, assignment
state, and peer-to-peer activity.The worker guide owns the setup, update, backup, and
troubleshooting procedures. This page only explains how the signals fit into the
Network.
The observer joins the peer-to-peer network and exposes metrics including:
when a peer was last seen;
peer ping count and last ping duration;
a worker’s reported storage use and missing chunk count;
the assignment timestamp reported by the worker.
Separate collectors receive worker heartbeats and query logs over the Network
transport and store them in ClickHouse. Portal also exposes /ready and /metrics.
Its specification defines request, dependency, worker-selection, and assignment
signals, while its gap register identifies signals that are not implemented yet.
