mstar.utils.host_copy

Large host copies split over threads of their own.

The API server sizes torch’s intra-op pool for its small per-request ops (entrypoint.default_preprocess_threads), and one numpy memcpy runs on one core. Most of a large copy into a fresh allocation is the destination’s first-touch page faults, which a copy split into slices takes in parallel too. numpy releases the GIL while it copies, so slices on a pool of their own run in parallel without resizing the intra-op pool.

Classes

HostCopier(workers[, min_slice_bytes])

Copies flat uint8 host arrays into fresh torch allocations, a large one in slices on workers threads.

class mstar.utils.host_copy.HostCopier(workers, min_slice_bytes=None)

Bases: object

Copies flat uint8 host arrays into fresh torch allocations, a large one in slices on workers threads. The caller views and reshapes the bytes.

The destination is torch’s, never np.empty: numpy madvises large arrays for huge pages, and under THP defrag=madvise their first-touch faults can stall in compaction.

Parameters:

• workers (int)

• min_slice_bytes (int | None)

copy(src)

A flat uint8 torch copy of the flat uint8 array src.

Parameters:

src (ndarray)

Return type:

Tensor

slices(nbytes)

Parameters:

nbytes (int)

Return type:

int