mstar.worker.chunk_loops

mstar.worker.chunk_loops#

The worker’s side of chunked prefill: which (node, walk) pairs run as a chunk Loop, and how far each step moves them. The Loop itself does the rest (Model.apply_chunking): it counts the tokens its chunks ran and ends at its prompt’s length.

Functions

mid_prefill_speculation_off(setting, chunked)

The (node, walk) pairs the serving config's speculate_mid_prefill turns off: per node, one bool for every walk it chunks, or a mapping of walk -> bool.

Classes

ChunkLoops([walks, speculated])

This worker's chunk Loops, read off its graphs at load.
class mstar.worker.chunk_loops.ChunkLoops(walks=(), speculated=None)[source]#

Bases: object

This worker’s chunk Loops, read off its graphs at load.

Parameters:
classmethod from_graphs(worker_graphs, speculate_mid_prefill=None, chunked=())[source]#

speculate_mid_prefill is the serving config’s, checked against chunked, every (node, walk) the model chunks.

Parameters:
Return type:

ChunkLoops

static walks_in(worker_graphs)[source]#

(node, walk) pairs these graphs run inside a chunk Loop.

Read off the graphs rather than off the model’s chunking policy: the policy says where a Loop may be built, and a deployment that excludes a node from chunking builds none there. The graph cannot disagree with itself, and a Loop nothing ends spins to max_iters holding the request’s state.

Parameters:

worker_graphs (list[WorkerGraph])

Return type:

frozenset[tuple[str, str]]

mstar.worker.chunk_loops.mid_prefill_speculation_off(setting, chunked)[source]#

The (node, walk) pairs the serving config’s speculate_mid_prefill turns off: per node, one bool for every walk it chunks, or a mapping of walk -> bool. On (the default), a row mid-prefill takes its next chunk in the speculative step, as a decode row takes its next token.

Parameters:
Return type:

frozenset[tuple[str, str]]