Scheduling and replication

The scheduler produces an assignment: a versioned description of the datasets, chunks, workers, and worker copies used by the Network. Workers use it to manage their local storage. Portal uses it to find workers for a requested block range.

​

Inputs to an assignment

The current production scheduler uses:

• the worker set and the latest reported worker status;
• the chunks discovered in the configured dataset storage locations;
• each chunk’s size, weight, and optional minimum worker version;
• configured worker capacity, storage saturation, and minimum replication.

It first calculates a target copy count for each chunk weight from the available capacity. The target is bounded by the configured minimum and the total worker count. Version and per-worker capacity constraints can reduce the copies actually placed or cause scheduling to report that capacity is insufficient. It then uses a deterministic hash-based placement algorithm with per-worker capacity limits. The same inputs produce the same assignment. Consistent hashing keeps many placements unchanged when the worker set changes, but it does not mean that every change moves only a small amount of data.

​

What can change an assignment

An assignment can change when:

• a worker joins or leaves the worker set;
• new chunks appear in a dataset;
• a worker version changes which chunks it can serve;
• storage capacity, saturation, minimum replication, or chunk weights change.

If only a small input changes and the replication counts remain the same, the hash-based placement can preserve most existing copies. If a capacity change causes a replication count to change, many chunk-to-worker mappings can change at once.

​

How nodes apply it

The published Network state identifies the current compressed assignment. Workers and Portal poll that state and fetch a new artifact when its identifier changes. A worker looks up its own peer ID, calculates the chunks it should hold, and starts the required downloads. Portal reads chunk ranges and worker peer IDs to construct its routing view. The assignment is therefore both a storage plan and a routing input, although workers and Portal read different fields from it.

​

Implementation sources

• Production scheduling code
• Current reshuffling analysis
• Assignment format and transport
• Worker assignment polling

Continue with How a query is served to see how Portal uses the routing view.