Supported Models#
mstar ships the following model families. The table below summarizes the registered
families, their registry key (the value of model: in a config YAML), and a
representative Hugging Face identifier.
Registry keys live in mstar/model/registry.py (MODEL_REGISTRY / HF_MODELS).
Registry key |
Example Hugging Face model ID |
Description |
|---|---|---|
|
|
Unified multimodal model (text + image understanding and generation). |
|
|
Zero-shot voice-cloning TTS: T3 speech-token LM (Llama-520M, or GPT-2-medium for Turbo) with CFG and exaggeration control, S3Gen flow-matching decoder, HiFT vocoder, PerTh watermark. 24 kHz. Multilingual covers 23 languages. |
|
|
Command A+ text and image chat: a 218B-parameter MoE (25B active) with sliding-window attention. Tested on 4x GB200 at TP4; see the notes below. |
|
|
Cosmos3 world model: t2i/t2v/i2v/v2v diffusion, robot-action modes, opt-in sound. |
|
|
Cosmos3 action-policy fine-tune for the DROID platform ( |
|
|
Cosmos3-Edge (4B): dense Nemotron backbone, 480p-native t2i/t2v/i2v and
robot-action modes, plus the reasoner (image/video chat through
|
|
|
Edge action-policy fine-tune for DROID ( |
|
|
Cosmos3-Super (64B) variant of the above; TP/SP for multi-GPU serving. |
|
|
4-step distilled Super task checkpoints (guidance baked in, fixed-sigma stochastic sampler); TP=2 deployments. |
|
|
Zero-shot voice-cloning TTS: Qwen2-0.5B speech-token LM with RAS sampling, DiT flow-matching decoder and causal HiFT vocoder, streamed in chunks. Clones from a preset or an uploaded clip. 24 kHz. |
|
|
TTS (82M, not autoregressive): misaki G2P + PL-BERT prosody + iSTFTNet decoder, 54 bundled voices and voice blends, sentence-chunked streaming, batched across requests. |
|
|
Tokenizer-free TTS: MiniCPM-4 base and residual LMs emit continuous latent patches through a local flow-matching DiT, decoded by AudioVAE V2 in streamed chunks. Zero-shot, voice design, and cloning from a clip. 48 kHz. |
|
|
Full-duplex speech-to-speech (NemotronLabs VoiceChat-11B): Fast-Conformer encoder + Nemotron-H hybrid Mamba-2/attention backbone + EarTTS talker + RVQ codec; user speech in, agent text and 22.05 kHz speech out. |
|
|
GLM-5.2 (753B/40B): MLA + DSA MoE causal LM, fp8 block-scale weights, TP8. |
|
|
TTS: Llama 3.2 3B LLM emitting audio tokens + SNAC 24 kHz decoder. |
|
|
Pi0.5 vision-language-action robotics model (ViT encoder + LLM + flow action expert). |
|
|
Hybrid-attention VLM (text + image in, text out): gated DeltaNet linear attention interleaved with full attention, plus a ViT tower. |
|
|
Qwen3.8-27B on the Qwen3.5 code above (same architecture); ships a TP4 config. |
|
|
Massively multilingual zero-shot TTS: masked-diffusion canvas over a Qwen3-0.6B backbone + audio codec. Clones a voice from a reference clip. |
|
|
Omni-modal (text/image/audio/video in, text/audio out): Thinker + Talker + codec. |
|
|
Streaming text-to-speech with built-in speakers: Talker + 12 Hz speech codec. |
|
|
1.7B CustomVoice: built-in speakers plus style/emotion |
|
|
1.7B VoiceDesign: the voice is described by the request’s |
|
|
1.7B Base: zero-shot voice clone from one reference clip (x-vector, optional in-context transcript). |
|
|
Fish Audio S2-Pro: dual-AR TTS (Qwen3-4B Slow AR + 4-layer Fast AR over 10 codebooks),
44.1 kHz causal DAC codec, voice cloning from one reference clip, inline |
|
|
V-JEPA 2 video encoder + masked predictor. |
|
|
V-JEPA 2-AC encoder + action-conditioned predictor. Video frames take the official repo’s central square crop, not the HF processor’s resize and crop. |
|
|
Whisper ASR: batched, CUDA-graph-captured audio encoder + paged-KV
decoder with write-once cross-attention; language detection, forced
language/task, |
|
|
Whisper large-v3 (32 decoder layers) on the same scaffold. |
|
|
Qwen3-ASR: AuT audio encoder (8 s attention windows, ragged attention)
+ dense Qwen3 decoder; up to 20 min of audio per request, 30 languages
(detected or forced), |
|
|
The 0.6B Qwen3-ASR checkpoint on the same class, sized in its config for the
short steps of |
|
|
Audio-tower + Qwen3 LLM speech-to-text. Beta / un-optimized. |
|
|
Meta Omnilingual ASR, CTC variants: wav2vec 2.0 encoder + CTC head, speech-to-text in 1,600+ languages with no language code. Not autoregressive; batched across requests. Apache-2.0. |
|
|
FastConformer encoder + Transformer decoder: speech recognition in 25 European
languages and translation to or from English ( |
|
|
English speech recognition by an LLM: Canary’s FastConformer encoder projected into
Qwen3-1.7B (LoRA merged at load). Clips up to 40 s; longer uploads are windowed. The
OpenAI |
|
|
FastConformer + token-and-duration transducer, 25 European languages, language detected from the speech. Clips up to 10 min; longer uploads are windowed. No timestamps. |
|
|
Cache-aware streaming FastConformer + RNN-T with a language prompt ( |
|
|
Wan2.2-TI2V-5B video diffusion: text-to-video and image-to-video, 5B dense DiT. |
|
|
Waypoint-1.5 interactive video world model: 1.28B DiT generating one latent frame per step, TAEHV encode/decode. Live RGB frame output; no OpenAI surface. |
|
|
LingBot-World-V2 1.3B causal-fast: camera-controlled image-to-world video, streamed
chunk by chunk (Wan2.1-layout causal DiT, rolling KV cache, 4 distilled steps).
A request carries the whole camera trajectory as JSON in |
|
|
FLUX.2 [klein] 4B: step-distilled (4 steps, no CFG) text-to-image and multi-reference image editing; Qwen3-4B hidden-state text encoder + FLUX.2 VAE. Apache-2.0. |
|
|
FLUX.2 [klein] 9B (Qwen3-8B encoder, 4096-wide DiT), same class. Released under the FLUX Non-Commercial License; check it before deploying. |
|
|
LTX-2.5 (22B) text-to-audio+video: the distilled 8-step recipe, no guidance; native joint audio-video DiT and Gemma-4 text encoder, emitting an H.264 video and 48 kHz stereo audio. Gated; LTX-2.x Community License. |
|
|
Z-Image-Turbo: 8-step distilled single-stream flow DiT (6B) with a Qwen3-4B caption encoder and the FLUX.1 VAE; text-to-image, no CFG. Apache-2.0. Same DiT scaffold as klein. |
Notes#
Models marked (Beta) are functionally supported but not yet performance-optimized; treat their throughput/latency as provisional.
The IDs above are representative. You may use local paths or compatible variants.
Some families accept multimodal input (image/audio/video); see the model’s
process_promptfor the inputs it expects.Models on the fused MoE kernel (Qwen3-Omni, Command A+) take their tiles from the
moe_gemmtable for the GPU (mstar/kernels/tables/moe_gemm/) when it has the expert shape, and default tiles otherwise. The server logs the sweep command for each shape it serves untuned; see “Tuned kernel configs” in Adding a New Model.To add a new family, see Adding a New Model.
OmniVoice notes#
Zero-shot only: there are no built-in speakers. Pass
reference_audiowith its transcript inref_textto clone a voice, or describe one invoice.ref_textis required alongsidereference_audio.languagetakes either the name (Vietnamese) or the id (vi): a name is resolved to the id the model was trained on before the prompt is built, and an unrecognised value warns and falls back to language-agnostic mode.The
omnivoicepackage is installed separately from git rather than by an extra — see Installation.The backbone is not autoregressive: it fills a fixed canvas of eight codebook rows over a few unmasking steps, so there is no KV cache and no per-token sampling loop. Serve it with
mstar serve omnivoice --gpus 0.The backbone’s layer stack replays CUDA graphs per (requests, packed-token bucket up to 8192 tokens) with attention through a ragged attention resource; larger steps run the same layer loop eagerly. The layer loop is mstar’s own over the reference’s fused weights: q/k norms read the fused QKV output in place and each residual add folds into the next RMSNorm (flashinfer
fused_add_rmsnorm), so logits match the reference to half-precision rounding rather than bit for bit.model_kwargs: {backbone_graphs: false}runs the same layer loop eagerly, andbackbone_token_bucketssets the packed-token buckets (256 to 4096 by default).configs/omnivoice_split.yamlruns code2wav (the codec decode) on a second worker sharing GPU 0, so the unmask loop no longer stops for each finished request’s decode; the silence trim and the rest of the output shaping run on the API side in either config. It setsmps: true(see Serving) so the two workers’ kernels overlap. On an H200 it raised concurrency-32 throughput from ~153 to 176-184 audio-s/s and cut TTFA p95 from ~9 s to ~3 s, measured while code2wav still did the shaping on its worker; concurrency 1 was ~10% slower.
Higgs Audio notes#
The audio tower runs outside
torch.compile(a clip’s chunk count changes its shape, and every new shape recompiled on the serving path). Requests whose 4 s chunks share a mel width are encoded together, and full-width chunks replay a CUDA graph per chunk count; a clip under one chunk is padded to the full width and masked (HiggsAudioTower’svalid_frames) so it replays a graph too.model_kwargs: {encoder_graphs: false}runs the tower eagerly.One prefill step per request: the ChatML text around the audio and the audio embeddings go into the KV cache as one span.
Decode keeps each request’s last token in the sampler’s slot master and sends the client’s copy inline (
MSTAR_DEVICE_LOOPBACK, on by default), as Qwen3.5 does: a decode step stores and routes no tensors.
Orpheus notes#
Orpheus only ever emits
<custom_token_*>ids (SNAC codes and the speech/turn markers), so its LM head computes the logits of those rows alone (ColumnParallelLinear(output_start=...), about 18% of the head) and reads-inffor the Llama text rows, at every TP size. Sampling therefore renormalizes over the custom tokens: the reference samples the full vocabulary, where a text token’s probability during speech is negligible but not zero, so sampled output is not identical to the reference’s.
Omnilingual ASR notes#
Serve with
mstar serve omnilingual_asr_ctc_300m(or_1b,_3b,_7b) and transcribe throughPOST /v1/audio/transcriptions. No extra is needed;pip install -e '.[audio]'adds non-WAV containers (mp3, flac, …).The model is a port of Meta’s Omnilingual ASR CTC checkpoints (code and weights Apache-2.0) that loads Meta’s own
.ptfiles without fairseq2.weights_versionin the YAMLmodel_kwargspicks the release:v2(default; lower CER per Meta; its vocabulary adds capitals and punctuation, though on our FLEURS clips it still wrote lowercase, unpunctuated text) is only on Meta’s CDN (dl.fbaipublicfiles.com/mms/), which publishes no checksums, so the files are checked against sha256 values pinned inmstar/model/omnilingual_asr/config.py(computed from our downloads);v1comes from thefacebook/omniASR-CTC-*Hub repos at pinned revisions.checkpoint_path/tokenizer_pathtake local files instead.Preprocessing follows the reference exactly: decode at the file’s rate, resample each channel to 16 kHz (torchaudio), average the channels, normalize the utterance to zero mean and unit variance.
/v1/audio/transcriptionstranscribes a clip of up to 20 s whole and cuts a longer upload into the fewest windows of at most 16 s, of about equal length, each ending at the quietest 0.1 s within 5 s of its boundary, transcribed in parallel. The model hears up to 40 s (the reference’s limit, and the cap forwindow_secondsinextra_body, which sets both lengths for one request), but on FLEURS concatenations 40 s windows give more than twice the character error rate of this policy./generaterefuses a clip over 40 s with a 400 rather than truncating it.The CTC head covers every language at once:
language,promptandtemperaturehave no effect (the reference ignores a language code for CTC models too), and decoding is a per-frame argmax. There are no timestamps:srt/vttget one cue per window andverbose_jsonone segment per window, timed to the windows. Nothing detects the language either:verbose_json’slanguageis the request’slanguageas given, or null./v1/realtimeis not supported.dtypedefaults tobfloat16, the reference’s inference dtype;float32also turns off TF32 for the node’s matmuls.max_batch_size(16) caps the requests in one step; the step sorts them by length and splits them so that each forward’s padded audio (rows times the longest clip) stays withinmax_batch_seconds(320). In bf16 the encoder packs a step’s clips into one unpadded forward (a captured graph up to 16384 frames) when their total audio fitsmax_batch_seconds; a step over it encodes padded, per split. In bf16 a request batched with others can differ from the same request run alone where two characters nearly tie, as the reference’s own batchedtranscribedoes;max_batch_size: 1ordtype: float32give batch-invariant transcripts.Parity scripts against Meta’s pipeline live in
test/omnilingual_asr/(record_oracle.pyruns the reference in its own venv, as it pins torch 2.8 and fairseq2).
FastConformer ASR (canary, canary_qwen, parakeet, nemotron_asr) notes#
Install with
.[asr]. Uploads that libsndfile can’t read (M4A, WebM) are decoded by torchcodec, which needs FFmpeg.One GPU per deployment: both nodes share a rank, and there is no tensor parallelism.
All four run the FastConformer encoder
nemotron_duplexstreams (mstar/model/components/fastconformer.py), over the whole clip. Canary, Parakeet and Nemotron ASR read its dimensions from the HFencoder_config; Canary-Qwen from NeMo’sperception.encoder(FastConformerConfig.from_nemo), which refuses an encoder whose parts NeMo configures as partly causal.Request fields, the same on
/v1/audio/transcriptionsand in/generate’smodel_kwargs:parakeet: detects the language from the speech.languagemay name one of the 25 it knows (bg, hr, cs, da, nl, en, et, fi, fr, de, el, hu, it, lv, lt, mt, pl, pt, ro, sk, sl, es, sv, ru, uk) and is then ignored; any other is refused.nemotron_asr:languageis its prompt: a locale from the checkpoint’sprocessor_config.json(such asen-US), a bare code, orauto(the default).canary:language(the speech’s, defaulten; one of Parakeet’s 25),target_language(a translation to or from English; the OpenAI route also takestask="translate") andpunctuation(default on).canary_qwen: English only (checked on the OpenAI route).instructionreplaces the trained one, “Transcribe the following:”;/generate’s text is unused.
A request holds up to 610 s on Parakeet and Nemotron, and 40.96 s on Canary and Canary-Qwen.
/v1/audio/transcriptionscuts longer uploads (up to 30 min) into windows at quiet moments and joins their text. None of the four emits timestamps.Parakeet and Nemotron hold no cache. Each clip’s decoder state (the LSTM’s, and the cursor) sits in a recurrent state slot from its first decode iteration until the request is torn down; the YAMLs size
resources.transducer_state.max_slots(129) formax_concurrent_requests(128) and the sink slot, and startup refuses a cap the pool can’t hold. A decode iteration is 64 greedy steps, and its tokens stream out as it ends.compile_step: falsein theirmodel_kwargsdecodes with eager kernels instead of the compiled step.Canary’s and Canary-Qwen’s YAMLs size the KV caches for
max_concurrent_requests(64), and startup refuses a cap the caches can’t hold.Canary-Qwen’s default KV cache takes about 15 GB (64 requests at 2048 tokens); on a smaller GPU, lower
max_concurrent_requestsandresources.kv_cache.max_num_pagestogether.
BAGEL notes#
Text-to-image runs only the CFG branches a denoising step reads: two with image guidance off, and the main branch alone outside
cfg_interval, about -37% latency at 1024² on an H200. A step then computes fewer rows and returns the main branch’s velocity outside the interval where it used to recombine all three, which is equal only up to rounding, so images move. Against the previous output on one GPU, seeded text-to-image at the default guidance is 34 dB PSNR at worst and 45 dB at the median (58–60 dB between two boots of one version), text-to-image without guidance 42 / 53 dB, and edits 24 / 40 dB, about what two runs of one version’s edit differ by (23 / 39 dB).A guided request’s steps inside
cfg_intervalrun as a walk of their own,image_gen_guided2(image_gen_guided3with image guidance), betweenimage_gen_headandimage_gen. A config that placesLLMper walk (configs/bagel_*_disaggregated.yaml) lists all four on the rank that generates images.On a CFG-parallel deployment (
configs/bagel_cfg_parallel*.yaml) those steps run on the guidance GPUs instead, as theimage_gen_cfg2walk (image_gen_cfg3with image guidance); the other steps run on the main GPU. A config that placesLLM_cfg_textorLLM_cfg_imgwithgraph_walksnames these walks:LLM_cfg_textandcombine_cfgboth,LLM_cfg_imgonlyimage_gen_cfg3.
Qwen3.5 / Qwen3.8 (qwen3_5_*, qwen3_8_27b)#
Text, image and video chat on the Qwen3.5 dense family (five sizes; MoE
variants are not supported yet). Served on both POST /generate and
/v1/chat/completions (image_url and video_url parts). Images and
videos may be interleaved with text anywhere in the prompt, and prefill follows
the order they were written:
mstar serve qwen3_5_4b --gpus 0
Video follows transformers’ Qwen3VLProcessor exactly: frames are decoded
with torchcodec on the host (FFmpeg’s shared libraries must be installed;
without them a video request gets a 400, as there is no fallback decoder),
sampled at the checkpoint’s video_preprocessor_config.json rate (2 fps,
4 to 768 frames for the released checkpoints), resized within its pixel
budget, and paired into temporal groups of two frames. Each group is preceded
by a <t seconds> timestamp in the prompt and placed in MRoPE like a
one-frame image. The prompt layout is the one transformers uses from 5.10
(and vLLM and SGLang): the template’s <|vision_start|>/<|vision_end|>
around a video stay, with each group’s own pair inside. Earlier transformers
replaced the outer pair too; mstar keeps the 5.10 layout whatever version is
installed. Resizing follows the installed transformers version:
transformers 5.22 turns on the per-frame pixel cap of qwen-vl-utils by
default, which lowers the token count of short clips. Notes:
The vision encoder attends within each image and each video frame group separately, and its captured CUDA graphs hold up to 16384 patches and 64 such segments per prompt. A prompt over either limit runs the encoder without the graph (still compiled, so only the launches are extra): a long video at ordinary resolution exceeds the patches, and one over ~64 s with frames resized below ~256x256 pixels exceeds the segments.
An encoder step packs the images and videos of up to 8 requests into one replay. Requests that fit alone but would overflow it together run one after another in that step. While the block loop is captured, a request too large for a replay alone (a long video) gets a step of its own. The packed GEMMs’ shapes follow the step’s total patches, so an image’s embeddings can differ at the last bit with what it is batched beside.
On two GPUs,
configs/qwen3_5_9b_dp2.yaml(a whole replica per GPU) serves video traffic faster thanconfigs/qwen3_5_9b_tp2.yaml, whose vision encoder runs on one rank while the other waits for it in the lockstep LLM: 9B on 2xH200 with TP-async on, DP2 gave 12.5-44% more output tok/s on 30 s and 60 s videos at c=16 and c=32.A long video is one wide prefill: one 1-3 minute video is ~11-12k tokens, beyond the largest captured
prefill_visionbucket (4096), so it runs eagerly.Size the KV pool for video.
configs/qwen3_5_*.yamlkeep the default 2048 pages of 128 tokens (262k tokens per deployment); 32 concurrent long videos need about twice that. Measured on Qwen3.8-27B at TP4: 32 concurrent long videos on the default pool gave 67 tok/s with most requests timing out, and 1,585 tok/s with none failing on 8192 pages. Raiseresources: kv_cache: max_num_pagesto fit the concurrency you serve.Decoding happens in the API server process, before the request is queued, and counts against
MSTAR_MAX_VIDEO_PIXELS: at the default, a 1080p video over about 259 s gets a 400.
Most layers are gated DeltaNet, so a request holds a recurrent-state slot as
well as a KV allocation. Set gdn_state.max_slots in
configs/qwen3_5_*.yaml to the concurrency you want plus one for the sink;
CUDA-graph capture and padded replays use the sink, not slots. A slot is
~20 MiB for the 0.8B and ~50 MiB for the 27B at TP4 (half that in bf16), hence
not the 256-slot default. The state defaults to the checkpoint’s
mamba_ssm_dtype (fp32 for the released checkpoints);
gdn_state.state_dtype: bfloat16 halves state traffic at some precision and
needs FlashInfer’s fused bf16 decode kernel (K = V = 128).
Text prefill and mixed steps replay captured graphs by total tokens, on the
ladder 32 ... 8704 (powers of two, plus 2176, 4352 and 8704 for a mixed
step’s decode rows and batches a few tokens over; a mixed step keeps only the
powers of two past 2176); a step pads to its bucket and a longer one runs
eagerly. model_kwargs: prefill_token_buckets: [...] sets a
deployment’s own ladder (distinct positive token counts, checked when the
worker loads): configs/qwen3_5_9b*.yaml take the powers of two to 2048
and 2176 for a mixed step, since the LLM’s 2048-token step budget
(MAX_BATCH_TOKENS) caps prefill and mixed steps and no bucket past 2176 is
reached. Each bucket adds captures and startup time.
A captured decode step builds its rotary tables inside the graph from the
planned positions, rather than staging tables built eagerly before the replay
(that build sat on the per-step critical path: +13% output tok/s at c=32 on
0.8B, H200). Greedy output can differ from earlier releases at near-tie
tokens; prefill and eager steps are unchanged (knobs: {ingraph_decode_rope: true}
does the same for captured text prefill and mixed steps).
The prefill, mixed and vision captures compile in Inductor’s deterministic mode
whatever -O says: autotuning them bought no GPU time and took about a fifth of
the cold start. The decode capture keeps the -O level’s tuned kernels. knobs:
{compile_deterministic: true} compiles it deterministically too: greedy output then
reproduces across server processes and the cold start more than halves. One-row
decode pays for that at the larger sizes (c=1 at 9B on B300: 270 against 286 tok/s;
at 27B TP4 on H200: 151 against 169) and gains at 4B on B300 (372 against 353); from
c=4 up it makes no difference.
temperature, top_p, max_tokens and seed are the standard fields.
repetition_penalty and enable_thinking (default true; the template opens
a <think> block) are read by the model but are not OpenAI fields — pass them
via extra_body.
With resources: {sampler: {enable_logprobs: true}} in the deployment’s
YAML, "return_logprobs": true in model_kwargs on /generate puts each
token’s raw log-prob in its chunk’s metadata ({"logprobs": [...]}, one per
token, as token_ids). Qwen3.5 reports them; other models refuse the request
with a 400. Add enable_processed_logprobs: true and "logprobs_mode":
"processed" for the log-prob under the distribution it was actually drawn from
(penalties and top-k/top-p applied); greedy requests want "raw", as a greedy
token scores 0 there. Top alternatives and OpenAI’s logprobs are not
supported yet. On any deployment,
"return_token_ids": true adds each text chunk’s sampled ids
({"token_ids": [...]}), stop tokens included, so a trainer need not
re-tokenize the text.
Qwen3.8-27B (qwen3_8_27b) is the Qwen3.5 architecture and runs on the same
code, served from configs/qwen3_8_27b_tp4.yaml (TP4):
mstar serve qwen3_8_27b --gpus 0,1,2,3
Its output_gate_type: swish names the gated DeltaNet norm’s activation,
which is the SiLU every Qwen3.5 checkpoint already uses. transformers ignores
the key and hard-codes SiLU. Video is still
refused and the MTP head is skipped. The chat template’s new
reasoning_effort and preserve_thinking are not forwarded: every
request gets the template’s defaults (xhigh and preserved), whatever the
client sends.
Command A+ (command_a_plus)#
Text and image chat on Command A+: 32 MoE layers (128 routed experts, top-8,
plus an always-on shared expert), with three sliding-window (4096) layers for
every global one. Served on POST /generate (there is no OpenAI adapter
yet):
mstar serve command_a_plus --gpus 0,1,2,3
curl -s http://localhost:8000/generate -F 'text=What is in this image?' -F 'files=@cat.jpg'
Images are tiled like transformers’ Cohere2VisionProcessor (up to 12
512x512 tiles plus a thumbnail, 256 tokens each) and encoded on the first LLM
rank. In messages, an image is a {"type": "image"} item; the uploaded
files fill them in order.
Install the command_a_plus extra, and point CUDA_HOME at a toolkit
matching PyTorch’s CUDA version so the fused-MoE alignment op can JIT-build.
The ~437 GB of BF16 weights are never downloaded automatically: download the
checkpoint and set model_kwargs.checkpoint_dir in
configs/command_a_plus_tp4.yaml, the default (TP4, 65536-token
max_seq_len). Its 8192-page KV pool fits on GB200; on 141 GB GPUs such as
H200, set resources.kv_cache.max_num_pages to 2048. It keeps only the
sliding layers’ window per request, and no prefix cache yet. Tokenizer and config
metadata are pinned to one checkpoint revision.
configs/command_a_plus_tp8.yaml targets 8x 80 GB GPUs and keeps NCCL’s
all-reduce until FlashInfer’s is measured there.
model_kwargs={"messages": [{"role": ..., "content": ...}, ...]} with
system / user / assistant turns goes through the chat template; a
plain text prompt is one user turn. Sampling defaults to temperature 0.9,
top-p 0.95 and repetition penalty 1.04. The model reasons before answering, so
give it a generous max_output_tokens; a prompt plus max_output_tokens
(2048 when unset) longer than max_seq_len is rejected.
On CUDA, decode runs in CUDA graphs over a fused layer: two concatenated GEMMs,
Triton kernels, and one all-reduce per layer (tp_allreduce: flashinfer at TP4).
On 4x GB200 it decodes about 290 tok/s for one request and 3700 tok/s across
64 (benchmark/command_a_plus/bench_http.py, measured before the sliding layers
kept only their window); validate_checkpoint.py
there checks parity against Hugging Face, with --images for image prompts.
Not supported yet: FP8 weights and prefill CUDA graphs.
Kokoro notes#
Install the G2P dependencies with
pip install -e '.[kokoro]'and fetch the spaCy tagger once withpython -m spacy download en_core_web_sm(misaki does this itself on first use when it has network access)..[all]includes the G2P dependencies, and the Docker images also install the tagger.pip install 'misaki[en]'additionally bundles espeak-ng (GPL), which Kokoro uses only as the fallback for out-of-dictionary English words and as the G2P for Spanish, French, Hindi, Italian and Portuguese voices; without it those words are skipped and those languages are unavailable. Japanese and Mandarin voices needmisaki[ja]/misaki[zh].Serve with
mstar serve kokoro. Request knobs:voice(a bundled voice such asaf_heart, or a blendaf_bella+af_sky/af_bella(2)+af_sky(1)),speed(0.25-4.0),lang_code(defaults to the voice prefix:aAmerican English,bBritish,eSpanish,fFrench,hHindi,iItalian,pPortuguese,jJapanese,zMandarin) andphonemes(skip G2P and synthesize a phoneme string directly).Deployment-wide options go in the YAML’s
model_kwargs(seeconfigs/kokoro.yaml):lang_codefixes the G2P language,espeak_fallback: falsedisables the espeak-ng fallback even when it is installed,chunk_target_phonemesandfirst_chunk_target_phonemesset the sentence packing,text_buckets,frame_buckets,capture_batch_sizesandmax_batch_framesshape the CUDA graphs captured at start-up (fewer buckets on a smaller GPU),compile_decoder: falseskips thetorch.compileof the vocoder (start-up in seconds instead of minutes, about half the throughput on an H100) anddecoder_dtype: bfloat16runs the vocoder trunk in bf16 (about 13% more throughput at concurrency 32 on an H100 and 20-25% at concurrency 32-64 on an H200, Whisper WER unchanged, with the harmonic source and the iSTFT kept in fp32; the parity test covers fp32 only;configs/kokoro_bf16.yaml). Each BiLSTM’s two directions run as one block-diagonal recurrence, and cuDNN times the conv algorithms of both captured halves once per bucket, at capture (NodeSubmodule.capture_cudnn_benchmark).Text is cut at sentence boundaries into chunks of at most 510 phonemes (the PL-BERT window); each chunk is emitted to the client as soon as it is synthesized, so
stream=Trueon/v1/audio/speechreturns audio sentence by sentence. The first chunk is kept short so time to first audio is one short synthesis.Output is 24 kHz mono PCM16. The model runs in fp32 by default: its vocoder is phase-sensitive, so reduced precision is opt-in. On CUDA the text half and the frame half of the forward are captured as CUDA graphs per length bucket and the frame half is compiled with dynamic shapes, so the first start-up on a GPU takes about two minutes; rows of one step are grouped by frame bucket (
frame_grouping: singlepads them into one group instead, kept for comparison).examples/livekit_kokoro.pyandexamples/pipecat_kokoro.pyplug the server into LiveKit Agents and Pipecat through their OpenAI TTS plugins (base_urlpointed at M*,response_format="pcm");GET /v1/audio/voiceslists the voices for a picker: the bundled voices whose G2P extra is installed on the server (a request for one of the others gets a 400 with thepip installhint).
Qwen3-Omni notes#
The default
configs/qwen3omni_2gpu.yamlholds 1,024 GPU pages (128k tokens) of Thinker KV plus 1,024 host pages it offloads to. On H200-class GPUs, when concurrent requests hold more than 128k tokens of Thinker KV, serveconfigs/qwen3omni_2gpu_large_kv.yaml, which keeps 2,048 pages (256k tokens) on the GPU and no host pages; it should fit beside its captures on an H200. On 80 GB GPUs such as the H100, serveconfigs/qwen3omni_2gpu_h100.yaml. Its 512-page Thinker pool (64k tokens) was sized on an H100 when the encoders shared the Thinker’s GPU and its text ladder ran to 8192: beside 1,024 pages 27 Thinker captures, all among those of 4096 tokens and more (text and vision), ran eagerly, and 640 pages still left two vision buckets out. Chunked prefill now caps a Thinker step at 1,024 tokens, so those large buckets no longer replay, but they are still captured and still take that memory; whether 1,024 pages fit with today’s ladders and encoder placement is unmeasured. The config runs the audio and vision encoders on the Talker’s GPU, so an encoder step doesn’t wait behind a Thinker step.None of the three configs sets
max_concurrent_requests. With the Thinker’s prefix cache on, a decode step that finds the GPU pool full offloads a running request to the host pages. Only when that frees nothing does it hold requests until they time out: with no host pages (large_kv), with the host pages full, or when the running requests’ pages are mostly a prefix they share (see the prefix cache under “Config files” in Serving). Where one of those can happen and concurrent requests can together hold more Thinker KV than the GPU pool, set the cap to the GPU pages divided by the pages of the longest request served, prompt and output together, at 128 tokens a page: at the full 32k context, 8 forlarge_kv, 4 for the default and 2 for the H100 config.The audio and vision encoders attend through ragged attention resources (
audio_attnwithin each 8 s window,vision_attnwithin each image or video frame group), FlashInfer by default, so noflash-attninstall is needed. A deployment can pick the kernel underresources:in its YAML, e.g.vision_attn: {flashinfer_backend: cutlass}. Each encoder’s block stack is captured as a CUDA graph per token-count bucket; a request past the largest bucket (32768 patches, about 5 minutes of audio) runs it eagerly.The audio encoder is the AuT node Qwen3-ASR runs, fed the API server’s log-mel features: a step’s clips (up to 8 requests) go through one packed forward, up to 4096 tokens. A step over 4096 tokens runs its clips one by one, each replayed if it fits a bucket. A clip’s last 1 s chunk is zero-padded to a full chunk before the convolutions, as HF’s encoder pads it whenever its batch holds a full chunk, so a clip under 1 s encoded alone differs from HF’s: in its last conv row and, through the windowed attention, in every embedding of the clip. A clip’s embeddings can differ at the last bit with what it is batched beside, since the step’s total tokens pick the captured bucket (or the eager GEMMs’ shapes), and a prefix-cached audio prefix keeps the embeddings it was first computed with, so a later request reusing it inherits that first batch’s rounding.
A prompt runs one Thinker prefill per attachment: each attachment’s row also holds the text before it, and the last one the text after it, so text, a clip and text is one step. A prompt with no attachment is one text prefill.
Streamed audio starts after 12 codec frames (0.96 s of audio), then 13, then 25-frame chunks behind 25 frames of left context. Playback stays gapless while the Talker produces frames at about 1.5x real time over the first 50 frames (about 19 frames/s per request); the Talker batches at most 32 requests, so requests past that queue and can fall below it and pause. Set
codec_chunk_schedule: []under the deployment YAML’smodel_kwargsto start with a 25-frame chunk instead: first audio comes later, and the Talker needs only about 1.0 to 1.3x real time.Numerics: the Thinker’s q/k RMSNorm and multimodal RoPE run in one kernel with the KV write, in fp32 throughout and rounded once, where the unfused ops (and Hugging Face) round to bf16 after the norm and after each step of the rotation: no less accurate, but not bit-exact. The code predictor (shared with Qwen3-TTS, see its notes) runs each projection with its RMSNorm folded in and its residual add in the same Triton kernel (
cp_linear), and its attention as another (cp_attend), neither bit-exact with the unfused ops. Its 15 depth steps feed each sampled code into the next, so small differences can change later codes at near-ties.The Thinker’s text and audio prefill replay captured graphs on the ladder
128 ... 2048tokens, and a longer step runs eagerly.model_kwargs: prefill_token_buckets: [...]sets a deployment’s own ladder. Each bucket adds captures, graph memory and startup time.The MoE layers route through the fused softmax-top-k kernel (
TopKRouter(fused_softmax_topk=True)), which gives equal probabilities to the lower expert id wheretorch.topkmay pick another.configs/qwen3omni_pd_2gpu.yamlrunsqwen3omni_pd_disaggregated.yaml’s prefill/decode split on two GPUs: the prefill Thinker shares rank 0 with the encoders, the Talker and Code2Wav, and rank 1 runs the Thinker’s decode. At concurrency 1 its text matchesconfigs/qwen3omni_2gpu.yaml’s.
CosyVoice3 notes#
Install with
pip install -e '.[cosyvoice3]'and serve withmstar serve cosyvoice3 --gpus 0. Every request clones a voice: aref_audioclip (with its transcript inref_textfor zero-shot cloning) or a preset fromvoices_dir(seeconfigs/cosyvoice3.yaml).instructionscarries a style instruction. Output is 24 kHz.Keep the reference clip short, a few seconds like CosyVoice’s own 3.5 s prompt clip: with a 23 s clip as the preset, round-trip word error rate rose from 1.4% to 13.4%, with words dropped mid-sentence.
On
/v1/audio/speech, text longer than one piece is synthesized as ordered sentence pieces, as the reference frontend does: about 80 tokens (320 characters) of English, or 80 characters of text with Chinese, each piece cloning the same voice. In one request the LM ends long text early or garbles its end. A Chinese sentence over 300 characters, which the reference keeps whole, is also cut at its commas, so that each piece stays under the LM’s 512 tokens of transcript and text.sentence_chunking: falsesends the text whole.Japanese written in kana alone, with spaces between words, is read poorly, as it is by the official model: in tests both garbled common words such as ゆっくり and ありがとうございました on every seed.
The flow’s DiT estimator runs in bfloat16 by default; the Euler state, the rest of the flow and the HiFT vocoder stay float32 (HiFT’s F0 predictor float64).
model_kwargs: {flow_estimator_dtype: float32}in the deployment YAML runs the estimator at the reference’s precision instead, at lower throughput.
Qwen3-TTS notes#
Install the model-specific dependencies with
pip install -e '.[qwen3_tts]'pluspip install --no-deps 'qwen-tts==0.1.1'(see Installation), and launch a single-GPU deployment withmstar serve <key> --gpus 0where<key>is one ofqwen3_tts(0.6B CustomVoice),qwen3_tts_1p7b,qwen3_tts_voicedesignorqwen3_tts_base. One model class serves every 12 Hz checkpoint; the variant (speakers, instruction support, reference audio) is read from the checkpoint’sconfig.json.Requests:
voiceselects a built-in speaker (CustomVoice),languagedefaults to automatic detection,instruct(OpenAI:instructions) carries a style instruction (1.7B CustomVoice) or the voice description (VoiceDesign, required). Base clones a voice from one reference clip: on/v1/audio/speechpassref_audio(data URL, base64, or an http(s) URL when the server setsMSTAR_ALLOW_REMOTE=1; paths on the server are refused) plusref_text(its transcript) orx_vector_only_mode: true; with the SDK,client.tts(text, reference_audio="ref.wav", ref_text="..."). The clip’s conditioning (x-vector and codec frames) is memoised by content, so a voice reused across requests, or by the sentence chunks of one long request, is encoded once; named, persisted voices arrive with the shared voice registry.Text layout follows the reference defaults: CustomVoice and VoiceDesign put the whole text in the prefill; Base feeds it one token per frame. Override per request with
non_streaming_mode./v1/audio/speechinputs of 600 or more characters are synthesized as ordered sentence chunks of about 400 characters (one Talker request each, two kept in flight while the current one streams); setsentence_chunking: false(ortruefor shorter texts) per request.Audio streams in a ramp of codec chunks: the first frame is decoded on its own (first audio one Talker step after prefill), the next chunks add 3, 8 and 16 frames, then 25 frames a chunk. The decoder is causal, so the codec decodes every frame once: per stream it keeps each causal conv’s last input samples, the transformer’s sliding-window keys and values (71 per layer) and the frame count, about 5 MiB in float32, in the
codec_staterecurrent pool.resources: {codec_state: {max_slots: N}}sizes it for N - 1 streams (one slot is the pool’s sink); 256 streams by default. Each chunk width is a CUDA-graph bucket captured for batch sizes 1 to 32 (densely: the codec’s cost is per row, so a padding row costs as much as a real one); the scheduler batches each width apart, so a stream’s small first chunks are not padded to a full chunk’s width or queued behind a batch of them. A voice clone’s stream leads with the reference’s last 72 frames, decoded with the first chunk to warm the state on the voice and trimmed from its audio. The SnakeBeta activations run as one fused kernel. A deployment sets the ramp and the steady chunk withmodel_kwargs: {codec_chunk_schedule: [1, 3, 8, 16], codec_chunk_frames: 25}.configs/qwen3tts_1p7b.yamlopens with 4 frames ([4, 8, 16]): on an H200 the streaming SLO (first audio within 200 ms, under 10 ms of pauses) then holds 101 Seed-TTS sentences a second against 90 with [1, 3, 8, 16], which halves idle first audio (15 vs 31 ms p50 at 1 req/s).Talker prefill, the clone prefill included, replays a packed CUDA Graph for the smallest token bucket (32 to 1024 tokens) that holds the batch; a clone’s reference frames join the codec stream in
postprocess. Prefill and decode are one combined walk (talker_mixed): up to four new prompts ride a decode step instead of a prefill-only step every decode row waits behind, replaying a packed capture sized to the decode batch. A decode-only step uses the whole-walk CUDA Graph (up to 128 rows a step), with the 15-step CodePredictor loop captured inside it; request-local EOS suppression is carried as a graph tensor input so replay does not consult capture-slot dummy request state. Residualsubtalker_*sampling is per-request through thecode_predictoraux sampler, so custom values neither block batching nor fall off the graph. On the 1.7B checkpoints the CodePredictor projects the Talker-width inputs throughsmall_to_mtp_projectionbefore its depth loop. It is the same component as Qwen3-Omni’s code predictor (ResidualCodePredictorinmstar.model.components.code_predictor_kernels): the depth sequence opens with the Talker hidden state in a one-token pass and takes one token per group after it, five launches a layer. Each projection is onecp_linearlaunch with the RMSNorm folded into its weight (a row scale from partial sums of squares its producer recorded) and the residual add or the SwiGLU in its epilogue; the q/k norm, RoPE, KV write and attention are onecp_attendlaunch, and the frame’s 16 codec embeddings are summed in one launch per group. Where the predictor projects its inputs (small_to_mtp_projection, the 1.7B), every embedding table a later depth step reads is kept projected. None of this is bit-exact with the unfused ops: the sampled residual codes can differ at near-ties. Each Talker keeps the sum it had before the two shared the component: Qwen3-TTS accumulates in the model dtype, rounding after every add (the reference sums in fp32 and rounds once), and Qwen3-Omni in fp32, rounded once.The Codec does not run beside the Talker on a side stream (
side_stream, see Serving): its windows read and write their streams’codec_staterows, and the side lane only takes a node that holds no resources, since it neither admits nor plans a step ahead. No Qwen3-TTS config sets the field.configs/qwen3tts_1p7b_split.yaml(andconfigs/qwen3tts_split.yamlfor the 0.6B) runs the Codec on a second worker that shares GPU 0 (rank_devices) and caps the Talker KV pool so both fit. On its own it lowers first-audio latency at high concurrency; withmps: truein the YAML (see Serving;qwen3tts_split.yamlsets it) the two workers’ kernels also overlap, which raised c=32 throughput by about a fifth on an H100.The Talker’s layers run
Qwen3Attention, so each layer’s q/k norm, RoPE and KV write are one kernel (qk_norm_rope_kv_write). It reads the RoPE theta off the position resource, which declares the checkpoint’s.The codec decoder runs in float32 by default: a stream’s audio is the reference decoder’s whole decode within 1 LSB of PCM16.
model_kwargs: {codec_dtype: bfloat16}selects a bf16 codec (RMSNorm and RoPE stay float32), whichconfigs/qwen3tts_1p7b.yamlserves. Against the float32 whole decode of the 1.7B’s frames it gives 35 dB SNR median (20 dB worst), 1.0 dB mel cepstral distortion, 0.9999 speaker cosine and the same WER, and the streaming SLO holds 101 Seed-TTS req/s against float32’s 90 (H200). Its dilated convs run as undilated ones over their interleaved subsequences: cuDNN takes some bfloat16 dilated convs (384 and 768 channels) through a direct kernel without tensor cores, about 100x slower on an H100. On a B300 the bf16 codec measured slower while those convs still ran dilated; measure there before switching. The decode is compiled (Inductor) twice per chunk width, batch 1 apart: its batch dimension is dynamic.Opt-in AudioSeal watermark on the codec’s output (off by default; install
pip install -e '.[audioseal]'). Turn it on in the deployment YAML:model_kwargs: audio_watermark: enabled: true # default false message: 0xBEEF # 16-bit payload: int 0..65535 or a 16-character 0/1 string (default 0) alpha: 1.0 # strength, in (0, 4] (default 1.0) generator: audioseal_wm_streaming # AudioSeal card name or checkpoint path (default)
Unknown keys or out-of-range values fail startup, and so does
enabled: truewithout the package. Each request’s PCM is marked chunk by chunk with its own streaming state, so a streamed utterance carries the same mark as a whole one. At startup the engine captures the generator per chunk length of the codec schedule (by default 1, 3, 8, 16 and 25 frames) and batch bucket, so marking a step’s chunks costs one graph replay per distinct length. The captures hold GPU memory: about 2.5 GiB for the default 30 graphs (5 lengths, 6 batch buckets) on an H200, 5.7 GiB with 25 lengths. A chunk outside them (another length, more rows than the largest bucket, a partial frame) runs eagerly, and the first one of each kind is logged. Check files withpython -m mstar.model.audio_watermark detect --message 0xBEEF out.wav(--message, the deployment’s payload: without it the verdict is the score alone, and an unmarked file re-encoded as MP3 can score above 0.5;--threshold, default 0.5;--device; needs.[audio]forsoundfile); a file at another rate than 24 kHz is resampled to 24 kHz first, since the mark is found only at the rate it was added at. Detect on whole utterances: a clip under a few seconds can miss the mark. The helper lives inmstar/model/audio_watermark.py.Weight loading checks coverage in both directions: a parameter the checkpoint does not fill, or a checkpoint tensor the port does not load, fails startup.
The 12 Hz codec does not require the system SoX executable. M* imports only the exact upstream speech-tokenizer modules, avoiding qwen-tts’s unrelated 25 Hz SoX probe during worker startup.
For throughput/latency validation, run the native serving benchmark with the Qwen3-TTS model metadata rather than the Orpheus compatibility entry:
python -m benchmark.runner \
--url localhost:8000 \
--model qwen3_tts \
--profiling-type closed_loop \
--request-type text_to_speech \
--num-requests 20 \
--inference-system ours \
--num-warmup 2 \
--max-concurrency 4 \
--dataset seed_tts \
--output-dir .bench_outs
The benchmark stops on the model’s natural codec EOS by default. Use
--ignore-eos --output-len-min N --output-len-max N only when measuring
fixed-length decode throughput rather than end-user latency.
The first process-local request can include eager FlashInfer kernel JIT, so
keep the warmup requests enabled when reporting steady-state latency.
For cross-engine comparisons (M*, vLLM-Omni, SGLang-Omni) use the shared
streaming /v1/audio/speech client, which measures time-to-first-audio, RTF
and audio-seconds per second with the same request for every engine, and score
the saved WAVs with benchmark/tts_speech_wer.py:
python -m benchmark.tts_speech_bench --engine mstar --url http://127.0.0.1:8000 \
--model qwen3_tts_1p7b --sentences sentences_200.txt --voice vivian \
--language English --concurrency 8 --repeats 3 --out results/mstar_c8.json
Fish Audio S2-Pro notes#
The
fishaudio/s2-proweights are under the Fish Audio Research License: research and non-commercial use only. M* downloads them at runtime and does not redistribute them; read the license before deploying.pip install -e '.[fish_s2_pro]'thenmstar serve fish_s2_pro --gpus 0.The voice-clone warmup at startup uses
samples/af_heart_0.wavfromhexgrad/Kokoro-82M. An offline deployment that pre-fills its HF cache needs that file as well; without it the warmup is skipped with a warning and the first clone request pays the one-time cost.Requests:
/v1/audio/speechwithinput; inline[tag]prosody control is part of the text. Clone a voice withref_audio(data URL, base64, or an http(s) URL when the server allows remote fetch) and its transcript inref_text; with the SDK,client.tts(text, reference_audio="ref.wav", ref_text="...").voiceandspeedare ignored.Sampling defaults follow the reference (temperature 0.8, top-p 0.8, top-k 30);
do_sample: falseis greedy. The order differs: M*’s sampler applies the temperature first and takes top-k and top-p over the tempered probabilities, while the reference takes top-p over the untempered ones, so at the same values M* keeps a slightly different set of tokens per step. The top-p boundary differs as well: M*’s sampler (FlashInfer) keeps the token whose probability carries the running total past top-p, while the reference drops it unless it is the most likely token.max_new_tokens(default 2048, at most 4096 frames of 2048 samples at 44.1 kHz) andignore_eospass through.The codec decodes each chunk’s new frames from its stream’s decoder state (the
codec_staterecurrent pool, 64 streams by default, as many as the AR decodes in a step, about 0.55 GiB; size it withresources: {codec_state: {max_slots: ...}}), so every frame is decoded once. It decodes with TF32 by default: TF32 roughly halves its conv stack and the audio stays within ~60 dB SNR of the exact decode, both measured on the earlier windowed codec, not yet on the frame-once codec. Withmodel_kwargs: {codec_exact_fp32: true}it runs in fp32 with TF32 off and the streamed audio matches the whole-utterance decode to within one int16 step.
VoxCPM2 notes#
Install with
pip install -e '.[voxcpm2]'and serve withmstar serve voxcpm2 --gpus 0./v1/audio/speechtakes the text alone (zero-shot), a voice description ininstructions(voice design), aref_audioclip (cloning) orref_audiowith its transcript inref_text(continuation). Output is 48 kHz.The Talker’s decode step (both LMs, the local DiT solve, the stop head) is
torch.compile’d and captured as CUDA graphs at start-up, so a first start-up with a cold Inductor cache takes 10 to 30 minutes (591 s on a B200, 1,723 s on an H100), and about two minutes once the cache is warm.model_kwargs: {compile_talker: false}in the deployment YAML captures the eager step instead: on a B200 the server is up in about 15 seconds, with about a third less throughput at concurrency 1 and a sixth less at concurrency 8.A request may hold the whole 8,192-position context, and nothing preempts a request once a KV cache is full, so each LM’s cache defaults to 2,049 pages (32 such requests and the sink page, about 9.7 GB for both) and the YAML sets
max_concurrent_requests: 32; the conductor queues the rest. Start-up refuses a cap the caches cannot hold, so raiseresources.base_kvandresources.res_kvmax_num_pageswith it. Text overmax_text_tokens(1,365 by default) gets a 400: English speech runs at up to about 2.9 patches per token, so longer text may not fit in the 4,096-patch (655 s) audio budget.One request with long text can end early: on some seeds the stop head fires mid-text, as it does in the official
voxcpmpackage (a 983-character Russian text stopped about three quarters of the way in on 5 of 10 seeds there). Split long text into several requests.
Nemotron VoiceChat (nemotron_duplex) notes#
Three nodes in three async partitions, joined by streaming edges, with the user’s audio a client stream (
audio_in) into the first:conformer_encoder(16 kHz audio → one LLM-space embedding per 80 ms frame, streamed: each chunk runs the layers over its new frames only, against the call’s cached last 70 frames, so a step costs the same however long the call is; it emits the frames no later audio can change, the last ones at the end of input. It is the shared FastConformer, Canary’s encoder, as a stream node (FastConformerStreamSubmodule), with the featurizer of the VoiceChat reference, which predates NeMo 2.5: reflected clip edges andsamples // 160 + 1mel frames. The checkpoint’s RNN-T transcription head is not loaded) →nano_llm(frame-synchronous decode loop: each step fuses one audio frame with the previous agent-text and tool-call tokens, emits one agent-text token) →eartts_talker(one text token → 31 RVQ codes, decoded by the RVQ codec in the same step to that frame’s 80 ms of PCM, which goes straight to the client: no codec step of its own, so a frame’s audio leaves with the step that made its codes. The decoder is causal, so each call keeps its stream state in a recurrent-state pool,codec_state: each ConvNeXt block’s last 6 inputs and the inverse STFT’s 12-sample overlap, ~95 KiB a slot in bf16. A step decodes only the call’s new frame against that state, and a fresh, zeroed slot is the decoder’s causal padding, so a call’s PCM equals the full decode of its frames from its first frame; at 32 sessions the decode is ~100 GMAC per 80 ms tick, inside the talker’s CUDA graph). The decode loops are stream-terminated: they end when the upstream stream closes, not on EOS, which is an ordinary per-frame token in duplex speech.nano_llmdeclares a paged KV cache over its four attention layers with the attention plan on it (resources: nano_kv/nano_attn; no positional encoding, so no position resource), a recurrent-state pool holding the 27 Mamba-2 layers’ conv window and SSM state, one slot per session (mamba_state: 32 sessions by default, ~137 MB per slot in fp32, tunable withmax_slots), the Mamba-2 resource planned on it (mamba) and the agent-text sampler (nano_sampler). Its decode step is captured as a CUDA graph for batch sizes 1 to 32.Every per-session resource is sized for 32 sessions: a slot in each of
mamba_state,encoder_state,encoder_frontandcodec_state(max_slots, one more than the sessions, for the sink) and inencoder_kv(max_slots, no sink), and its pages innano_kvandtalker_kv(max_num_pages; both sized frommax_session_s, and the nano’s frommax_prompt_tokenstoo). The shipped configs set a conservativemax_concurrent_requests: 16(below); raising it to 32 needs no other change, and to serve more, raise the cap and all six with it. The server refuses to start with a cap one of them cannot hold, since a session past any of them would wait on admit, and its frames with it, until another ends. The nano, the talker and the encoder each run at most 32 sessions a step, their widest CUDA graph; more are split over several steps rather than run eagerly.eartts_talkeradvances every live session in one backbone pass per frame through the engine’s paged KV (talker_kv/talker_attn/talker_pos): each session owns two streams, the conditional and the unconditional CFG context, planned together under one combined label, so a step is one packed batch of two tokens per session (a session’s first step prefills its 38-token speaker/text warm-up instead). The pool’s head dim is 128: FlashInfer computes 64/128/256 exactly and returns wrong values for the talker’s 72, so q/k/v are zero-padded (exact) and RoPE is applied in torch from the positions the position resource planned. Sampling noise is drawn per row from the request’s seeded generator before the step, so a session’s speech does not depend on which other sessions share it. The per-frame step is captured as a CUDA graph for 1 to 32 sessions; a session’s first step (its 38-token speaker warm-up) runs eager. The talker KV pool is 1025 pages of 128 positions (28 GiB): 32 sessions x 2 streams x 16 pages plus a sink page, enough formax_session_s(160 s) per session, with no sliding-window eviction yet; longer audio is refused (below).Measured on one H100 80GB (
benchmark/nemotron_duplex/sessions.pyon the 106-frame demo clip, three repeats): 24-25 ms per 80 ms tick with one session, 61-64 ms with 32 concurrent sessions, 76-85 ms with 40, 87-92 ms with 48 and 95-97 ms with 64 (2026-10-08). 32 is the most that keep up with real time, hence the pools’ size; every session received all its frames. Those are one-shot replays (the whole clip uploaded, so the server runs as fast as it can). Live calls paced in real time over/generate/ws(test/nemotron_duplex/live_ws_request.py, 80 ms appends, 2026-10-17, one H100): input-to-token lag p50 48 ms with one call and about 100-120 ms with 8, which keep up; from 24 calls up the server fell behind real time, so lag grew over the call. Capturing the encoder’s live step (below) took its host time at 32 calls from 23.6 to 8.4 ms and cut live lag 2-6x at 16-32 calls (paced runs interleaved against the uncaptured encoder,MSTAR_SHM_ARENA=1); 32 calls staggered over 2 s all finish, heard latency p50 2.1-2.3 s (from the user stopping to agent audio playable without gaps). With the codec in the talker’s step, on one H200: 16 live calls keep real time whether they start together or 2 s apart; 32 all finish but sit at the edge of real time (one run kept up with 32 started together, another did not; later runs kept up at 32), so the shipped cap is a conservative 16. Served audio is checked for intelligibility by transcribing it (test/nemotron_duplex/asr_check.py, Whisper large-v3-turbo): MaskGIT sampling is knife-edge, so attention-backend numerics change the waveform but not the words.Against vLLM-Omni 0.28’s native duplex serving of the same checkpoint (three eager fp32 engines, one session per GPU by its deploy config, its realtime client): on the demo clip both systems’ response latency is the model’s turn-taking decision (first agent text about 1.8 s after the user stops; first voiced audio 1.6 s here vs 2.0 s in stream time there, 2.3-2.6 s wall there), while M* serves 32 concurrent sessions with margin on one H100.
The nano text tokenizer is read from the
nano/folder ofpipecat-ai/NVIDIA-NemotronLabs-VoiceChat-11B-Spark(the base checkpoint ships only the RNN-T tokenizer); prefetch both repositories on machines without egress.Inputs are user-only 16 kHz mono audio plus an optional system prompt (the request’s text, primed before the first frame exactly as the reference does): one clip uploaded with the request, or a live call over
/generate/ws,appendmessages untilend_input(see Using a Server; a one-shot request over the socket carriesend_input: trueon its opening message, or it waits for appends). Output is clocked by the input, one agent frame per 80 ms of user audio, so a live client keeps streaming (silence included) for as long as it wants the agent to talk. The model answers in the frames after the user stops, so append a few seconds of trailing silence to a clip. The checkpoint’s demo recordings are two-channel conversations (user left, agent right); feed the left channel only.Limits: a call is capped at
max_session_s(160 s) of audio. An upload, or any one file, past it gets a 400, as does one of more than two channels, and so does a live call whose appends pass it:audio_indeclares the session’s samples as itsmax_items, which the client stream counts before the encoder sees them (raisemax_session_s, not the stream’s limit, for longer calls). A call that ends with no more than 16 ms of audio, which the encoder cannot featurize, fails with a 500 (the worker reports it without a client status). Appends must be 16 kHz audio (they are neither resampled nor rescaled one by one, which would put a seam at every append). Send appends of 80 ms or more: the encoder takes one append per step, so shorter appends need several steps per frame and fall behind once a step takes longer than an append.Encoder numerics, measured on CPU at the checkpoint’s dimensions with random weights on the duplex encoder the shared FastConformer replaced, and not since: an upload was one encoder pass, bit for bit that encoder’s offline pass in fp32 and under bf16 autocast, save the last frame when the end of input reaches the encoder after the audio (about 1e-6 in fp32). 80 ms appends put the frames within about 2e-6 of offline in fp32 and about 0.7% under bf16 autocast. The shared encoder’s stream attention adds the relative-position term before scaling and its offline attention after, so an upload is no longer expected to match the offline pass bit for bit. From those figures, live and one-shot text should agree token for token in fp32 (inferred, not measured on served text); under the served bf16 a different chunking can flip a knife-edge frame such as the agent’s turn start. Under bf16 autocast the stream node keeps the blocks’ residual in fp32, as the replaced encoder did from its second layer on; the offline FastConformer encoders keep theirs in bf16. Against an fp32 run on 13 clips (one H200, real checkpoint, 2026-10-29), live frames have a mean per-frame relative error of 1.35%, level with the replaced encoder’s 1.39%, where a bf16 residual gave 1.63%; uploads and offline passes are as close as the replaced encoder’s too.
The encoder keeps each call’s state in a slot per session of three resources: its attention keys and values in a bounded KV (
encoder_kv), which keeps a call’s last 70 frames per layer and runs the attention, each new frame over the 70 before it, with the relative-position term as a band of distances; its conv inputs (encoder_state); and its newest audio and mel frames (encoder_front), the last two recurrent-state pools. A call that appends one frame of audio (1280 samples) on the frame grid, every live step of an 80 ms client, runs its whole step as a CUDA graph (the grid is the call’s start: an opening clip or append that is not a multiple of 1280 samples keeps the call off it for good); other calls take the front end eagerly, then the layer stack as a graph when each got at most one frame (a live step, the call’s end), or eagerly (a one-shot upload, a large append: a chunk over a second is not batched with other calls, so the engine runs each call of that step on its own). Appends of other sizes are served the same, without the whole-step graph.Checks:
test/nemotron_duplex/duplex_request.pyprepares a clip from the bundledturn_taking.wavand sends it;parity_compare.pyruns the standalone reference path on the same clip and checks the served text against it token for token (bf16, the serving precision, must match save the last frame; the fp32 oracle can differ where the agent’s turn-taking decision is a knife-edge, which its printed logit margins show).live_ws_request.pystreams a clip in 80 ms appends and reports per-frame lag; with--compare-one-shotit checks that a live call sending the whole clip as one append matches the upload save the last frame, and reports where the paced call’s text first differs from it.Default deployment:
configs/nemotron_duplex.yaml(all three nodes on one GPU);configs/nemotron_duplex_disagg.yamlputs the encoder+LLM and the talker (with its codec) on two ranks. Launch with--tensor-comm-protocol SHMon a single node. Text samples attemperature: 0.7, top_p: 0.9by default; sendtemperature: 0for greedy. The talker’s mixture-of-Gaussians sampling is stochastic by design and seeded per request.Known limitation of the checkpoint: it often answers speech that is not addressed to it, such as a side conversation or background speech, where it should keep its turn (about half of such pairs on Full-Duplex-Bench v1.5). NVIDIA’s NeMo reference inference (
examples/speechlm2/offline_voicechat_infer.pyon thenemotron-labs-voicechatbranch of NVIDIA-NeMo/Speech) answers the same side speech on the same inputs, and the model card’s known limitations say the model is not suitable for noisy environments, especially where background speech may occur. Nothing in the serving path tells speakers apart. Real interruptions and backchannels are handled well.
Chatterbox notes#
pip install -e '.[chatterbox]'thenmstar serve chatterbox --gpus 0(chatterbox_turbofor the distilled Turbo checkpoint,chatterbox_multilingualfor the 23-language one). All variants are one model class; the variant follows the registry key ormodel_kwargs: variant.Multilingual: the same graph, S3Gen and voice encoder as the English model with the
t3_mtl23ls_v2T3 weights (a 2454-token grapheme vocabulary) and a language token in front of the text. Requests passlanguage_idinextra_body(ar da de el en es fi fr he hi it ja ko ms nl no pl pt ru sv sw tr zh; the deployment’smodel_kwargs: default_language,enby default, applies when a request gives none). The text is lower-cased and NFKD-normalised, Korean is decomposed into jamo and Chinese spelled as Cangjie codes from the checkpoint’s table; Japanese kana reading, Hebrew diacritics and Russian stress marks use the same optional packages as the reference (pip install -e '.[chatterbox_multilingual]'bringspykakasiand the Chinese word segmenter;dicta_onnxandrussian_text_stresserare installed separately) and are skipped with a warning when a package is missing, as the reference does.Requests:
/v1/audio/speechwithinput,voice(default= the voice shipped in the checkpoint, or a preset name resolved under the deployment’smodel_kwargs: voices_dir), and inextra_bodyref_audio(data URL or bare base64 of a reference clip, 5-30 s, cloning; an http(s) URL only when the server setsMSTAR_ALLOW_REMOTE=1; paths on the server, clips over 60 s and clips below 8 kHz are refused),exaggeration(0-2, emotion intensity, default 0.5),cfg_weight(0-1, default 0.5; 0 disables guidance and halves the T3 work),temperature(up to 5; under 1e-5 is greedy)/top_p/top_k(the whole vocab or more is no filter)/min_p/repetition_penalty(up to 2; the upper bounds are the reference demo’s),seed,n_cfm_timesteps(S3Gen Euler steps, 10; Turbo 2),max_new_tokens(up to the deployment’smodel_kwargs: max_new_tokens_limit, 1000 in the shipped configs; withmax_concurrent_requestsset, the server refuses to start unless that many requests of that length fit the KV cache, so raise it together withmax_num_pagesor a lower cap),ignore_eos(T3 decodes allmax_new_tokenspast the stop token, for fixed-length benchmarks; the speech tokens among them are vocoded, but T3 keeps emitting stop and control tokens after the end of speech and those are dropped, so the audio is shorter thanmax_new_tokens/ 25 s: 4096 tokens gave 117 s on base and 141 s on Turbo, not 164 s) andwatermark(default on). Turbo ignorescfg_weight,exaggerationandmin_plike the reference package. The native/generateroute andclient.tts(...)take the same knobs; a clip uploaded asaudioinput is the reference voice.Graph:
voice_encoder(speaker LSTM + S3 tokenizer over the reference, cached per clip hash) ->T3(paged KV, continuous batching; guidance runs the conditional and unconditional streams through one attention plan and one captured decode graph per batch size) ->s3gen(own streaming partition).Outputs are watermarked with Resemble’s PerTh network when
resemble-perthis installed;watermark: falseper request or inmodel_kwargsturns it off, and a deployment without the package logs that outputs are unmarked. The mark is embedded on the worker’s device end to end (the package’s own routine resamples on the CPU); it decodes with the package’s detector like the package’s output, from which it differs by the resampler only.Streaming (
stream: true) emits WAV chunks as the speech tokens arrive: the first after 35 tokens on Chatterbox (1.12 s of audio, enough to cover the wait for the next chunk at 4 to 8 streams) and after 15 on Multilingual and Turbo, then 50, 100 and 200 tokens (model_kwargs: stream_first_chunk_tokens/stream_chunk_tokens/stream_chunk_growth/stream_max_chunk_tokens): each chunk buys the playback time to produce a bigger one, so a stream costs three or four flow solves instead of one per 25 tokens. Each chunk re-runs the flow decoder over all tokens so far with a fixed noise field, holds back the three look-ahead tokens and crossfades the vocoder tail, so the stream is continuous but not sample-identical to the whole-utterance decode;stream_chunk_tokens: 0synthesises whole utterances (the reference path, bit-exact with the package at a fixed seed).stream_context_tokens(default 25; 0 = whole history) bounds how many settled tokens a chunk’s flow solve keeps as left context, making the per-chunk cost constant; the window stays as close to the whole-utterance decode as the full history does (log-mel correlation 0.988 vs 0.985 on CPU). Requests whose chunks are ready together share one padded flow solve (up to 8 per step).configs/chatterbox_split.yaml(andchatterbox_turbo_split.yaml,chatterbox_multilingual_split.yaml) runs S3Gen on a second worker that shares GPU 0 (rank_devices), so T3’s decode steps stop alternating with S3Gen’s flow solves on one worker loop. It only pays under CUDA MPS, which lets the two workers’ kernels overlap, so the configs setmps: true(see Serving): on an H200 at concurrency 32 Chatterbox went from 36-38 to 58-61 audio-s/s and Turbo from 69 to 94, at the cost of a later first chunk (TTFA p50 1.6-1.8 s -> 1.9-2.1 s; Turbo 0.36 s -> 1.0 s); concurrency 1 is unchanged, and WER held (1.22% -> 1.08%). Without MPS the split measured the same as one worker.S3Gen runs from CUDA graphs by default (
s3gen_graphs): the flow solve (one graph per rows x frames x steps; the estimator’s hundreds of tiny kernels per Euler step make the eager solve launch-bound, 170 ms vs 45 ms for one row on an H100), the token encoder (per rows x token bucket) and the HiFT vocoder up to its output spectrum (per exact chunk length, its excitation noise drawn outside the graph in the reference’s order; the inverse STFT runs eagerly sincetorch.istftsynchronises). Rows are padded to the powers of two up tos3gen_max_batch_size(8) and frames tos3gen_frame_bucket(64); the built-in voice’s chunk shapes are captured at startup, other shapes on first use;s3gen_graph_stages(defaultsolve,encoder,vocoder) picks the stages. The flow estimator runs ins3gen_estimator_dtypefloat16 by default with the Euler state in float32 (within 0.02-0.15 of the float32 log-mel;bfloat16is as fast and further off;float32is the reference path, bit-exact with the package at a fixed seed).t3_prefill_graphs(default on) captures T3 prefill as packed CUDA graphs by token bucket for batches of up to four requests; decode graphs are always captured.s3gen_graphs: falsegives the eager path;s3gen_compile: trueis the older alternative and turns the graphs off.Sampling follows the reference order inside the sampler resource: repetition penalty -> temperature ->
min_p->top_p; the T3 node declaresenable_min_pon itsSamplerSpec(see Adding a New Model). T3 runs in bf16 (model_kwargs: t3_dtype: float16is the alternative; float32 is refused because the paged attention has no float32 kernels, and fp32 token parity is checked by the CPU tests intest/chatterbox).Reference clips are decoded with
soundfile(WAV/FLAC/OGG/MP3 through the bundled libsndfile); other codecs fall back totorchcodec, which needs FFmpeg’s shared libraries on the node.Keep each request short, a sentence or two. Text longer than 512 tokens is rejected, but quality drops well before that, as it does in the reference package: on three-sentence inputs (600-720 characters) Turbo often drops or babbles the last sentence (median WER 0.10, reference 0.12; base 0.02), and 1,500-2,000 characters come back from Turbo as a second or so of unrelated speech, with a 200. For long text send
sentence_chunking: trueinextra_body: the server splits it into sentence groups of up to 400 characters (at most 32) and synthesises them in order, or split it client-side.Benchmarks and parity scripts live in
benchmark/chatterbox/(bench_all.shdrives M*, Chatterbox-TTS-Server and chatterbox-vllm on one GPU;reference_greedy.py+serve_parity.pycompare a served greedy synthesis with the reference package;wer_eval.pyis the Whisper intelligibility guard;node_breakdown.pyturns a served run’s--log-statsprofiles into a per-node time table;profile_s3gen.pymicro-benchmarks the flow estimator, solve, encoder, vocoder and watermark by precision, eager vs. CUDA graph;gpu_util.pysummarisesnvidia-smi dmonsamples taken during runs).
Cosmos3 environment requirements#
flashinferis required: it is the paged KV/attention backend used by the prefill, the captured CUDA graphs, and multi-request batches.The default denoise attention backend is
dense_gen(Cosmos3Config.attention_backend), which runs bs=1 eager generation attention as one FlashAttention-3 varlen kernel from thefa3-fwdwheel (AITER’s varlen kernel on ROCm). That wheel is ABI-tied to the installed torch/CUDA build (Hopper builds exist for at least torch 2.9 + cu12.8 and torch 2.11 + cu13.0); install the one matching your environment. It ships kernels for sm_80 and sm_90a only. Where it is not importable or has no kernel for the GPU (Blackwell), the same dense pass runs on FlashInfer’s ragged prefill, whoseautokernel on a B300 is CUTLASS’s FMHA: 1.5 ms per layer for the 832x480x121 guidance pair against 6.3 ms for the paged prefill.resources: attn_gen: flashinfer_backend:picks that kernel (cudnn,cutlass,fa2); the server checks it at startup with a tiny plan and run, and a backend this GPU or FlashInfer build cannot run (a typo,fa3on Blackwell,cutlasson Hopper) stops it with aValueErrornaming the backend. Without a usable FlashInfer either, the engine logs a warning and switches to the paged backend, but that is FlashInfer too (the default backend on every platform for now), so it cannot run there either: the dense pass needs fa3-fwd (AITER on ROCm) or FlashInfer.model_kwargs.attention_backend: flashinferin the config YAML selects the paged backend explicitly.Video-input requests (video-to-video, action inverse-dynamics) decode the conditioning clip with
torchcodec; environments without it reject those requests at preprocessing (other modes are unaffected).Generated video containers are written with
torchcodec’sVideoEncoderwhen available (torchcodec >= 0.9), otherwise withtorchvision’swrite_video, which needs the PyAV (av) package.Sound-enabled video responses mux the AAC track with the
ffmpegandffprobebinaries, which must be onPATH(system packages, not pip-installable).The Wan-VAE decode dtype is gated on the cuDNN build: bf16 needs cuDNN >= 9.16 (fast Hopper bf16 conv3d); older cuDNN serves the decode in fp32/TF32 automatically.
Two GPUs (H200 numbers, measured before the Cosmos3-Edge Blackwell changes, so the one-GPU baselines are older):
configs/cosmos3_nano_sp2.yamlhalves one request’s denoise (Nano 832x480x189x35: 1.47 -> 0.78 s per step), whileconfigs/cosmos3_nano_dp2.yaml(a whole replica per GPU) serves more under concurrency: 2.13 videos/min at 2 concurrent vs 1.63 for sp2 and 1.07 on one GPU. For Edge (832x480x121, 20 steps):configs/cosmos3_edge_sp2.yamltakes one request from 6.3 to 4.4 s (generation only: the reasoner has no SP path), andconfigs/cosmos3_edge_dp2.yamlserves 15.8 videos/min at 4 concurrent vs 7.9 on one GPU.
Cosmos3 image-to-video conditioning#
An image-to-video request’s conditioning image is scaled to cover the output
size with its aspect ratio kept, center-cropped and rounded to 8 bits, as in
NVIDIA’s cosmos-framework and diffusers 0.40 and later. Setting
conditioning_resize: stretch under the config YAML’s model_kwargs
resizes it to the output size instead (antialiased bilinear, aspect ratio not
kept), as diffusers 0.39 did. Action requests ignore the setting.
Cosmos3-Edge reasoner and action loop#
cosmos3_edge serves the understanding tower as a vision-language model on the
same transformer instance and KV pool as the generator. /v1/chat/completions
takes image and video content parts (URLs or data URIs) and streams tokens; the
chat template opens a <think> block by default. extra_body knobs:
enable_thinking (or chat_template_kwargs.enable_thinking), top_k,
repetition_penalty, and for video attachments video_fps / video_num_frames
(frames are sampled at 2 fps by default, each frame a timestamped span).
curl -sN http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
"model": "cosmos3_edge", "stream": true, "max_tokens": 256,
"messages": [{"role": "user", "content": [
{"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}},
{"type": "text", "text": "The task is to put the flower into the red bottle. Plan the next steps."}]}],
"enable_thinking": false}'
The decode step is captured into CUDA graphs per batch bucket and, by default,
compiled first (compile_reasoner_decode: true; COSMOS3_REASONER_COMPILE=0
turns it off): the eager step is over a thousand tiny kernels, and the fused
step runs at the weight-streaming floor (about twice the uncompiled rate at
batch size 1 on an H100). Concurrent chat requests share decode steps
(continuous batching over the captured decode graphs, padded to the next batch
bucket). Requests never see each other’s data, but the batch bucket changes the
bf16 arithmetic of the step, so a long greedy answer can part from its solo run
where two candidate tokens tie: in every measured divergence the two tokens’
logits were equal or one bf16 ulp apart, the first differing token came after
tens to hundreds of identical ones, and identical prompts in one batch agreed
with each other. Short answers (128 tokens) came out identical in 8 of 8 runs;
256-token answers with thinking on in 2 of 8. Generation requests batch
into one denoise pass too, which is the same maths but not the same bf16
arithmetic — under classifier-free guidance the branch rounding is amplified,
so an image or clip produced alongside other requests differs from its solo
result at the kernel-drift level (~30 dB PSNR at guidance 6). A generation
step whose requests total more than max_batched_vision_tokens (a
model_kwargs key, off by default since it is a per-GPU number; 16384 on an
H100 is, at 832x480, more than 42 latent frames across the step’s requests,
e.g. two clips of 85 or more frames each; a windowed request counts its current
window) runs every request in it alone, since batching such steps cost more
than it saved on an H100, so those outputs match their solo results. Serve with one request at a time
when outputs must be bitwise repeatable.
The denoise step is compiled (compile_denoise) in Inductor’s deterministic
mode by default (denoise_compile_mode: deterministic, the engine’s compile
mode of that name): kernel configs are picked by rule, so a seed gives the same
video from every server process. default is torch.compile’s own defaults,
and level takes the mstar serve -O level’s options (-O2:
max-autotune).
Video clips decode through the shared Wan VAE decoder (vae_decode_fast):
one decoder call per latent, channels-last, in the decode dtype (bf16 with
cuDNN >= 9.16, otherwise fp32; see above), on the decoder-only copy of the VAE
the channels-last decoders on a GPU share. The first two calls (the clip-start latent and the one after it)
run eager; every later call is compiled in Inductor’s deterministic mode. An
image (one latent) keeps the compiled stock vae.decode on that copy, which
is faster for a single frame. False decodes everything with the compiled stock
vae.decode on the shared instance.
Two or more requests at the same denoise step run one cross-request forward
(max_gen_batch_size, default 8) for video, image and sound; the action step
runs eager. With compile_denoise_batched (default on) its layer stack is
compiled with the same mode, once, while the server starts, with the packed
length dynamic: no batch size or mix of resolutions compiles mid-traffic. The
per-request patchify, rotary tables and unpatchify around it run eager. At
832x480x121 one request’s guidance pair is 24k tokens, which already fills the
GPU’s GEMM and attention throughput, so a batch of N costs about N single
steps: batching keeps requests in step rather than adding throughput.
The action policy (cosmos3_edge_droid, or cosmos3_edge with an action
domain_name) predicts a chunk of robot actions from the current observation:
output_modalities=action with model_kwargs
{"action_mode": "policy", "domain_name": "droid_lerobot", "raw_action_dim": 8,
"action_chunk_size": 32}; the reply’s action payload is float32
[chunk, action_dim_padded] and the first raw_action_dim columns are the
embodiment’s actions. A policy request may also send the robot’s current state
as robot_state, one row of raw_action_dim finite numbers (a NaN or inf is
a 400), as pi0.5 takes it. The state is one more action token, kept clean at
video frame 0 in front of the chunk, so num_frames is action_chunk_size + 1
and the reply carries only the predicted rows. The released DROID policies were
trained this way. A request that sends a robot_state is a policy request, and
its chunk, fps and domain default to the checkpoint’s checkpoint.json (Edge
DROID: 32 rows, 15 fps, droid_lerobot; configs/cosmos3_droid.yaml sets the
same for Nano, whose file is empty), so NVIDIA’s RoboLab request is the image, the
prompt, the state and size="736x544": the wrist frame above the two exterior
frames reflection-padded to 736x544, the 7 joint positions and the gripper as
DROID records them, and the task with view, duration and resolution text (a JSON
caption for Edge). The DROID configs map the gripper to the checkpoint’s
1 - gripper in every action a request sends and back in every action it
gets (inverted_action_dims),
and cosmos3_edge_droid guides only inside the cookbook’s interval
[960, 1001] (guidance_interval_action; outside it the conditional branch
runs alone, batched requests too). For a control loop use /generate/ws (one
connection, pipelined observations): examples/cosmos3_action_ws_client.py
builds that request, runs the loop and reports chunks/s, actions/s and latency
percentiles.
Cosmos3 streaming rollout (windowed video)#
Long clips can be generated window by window instead of in one denoise loop,
with each finished window streamed to the client while the next one is being
denoised. The deployment opts in with enable_windowed_video: true in the
config YAML (configs/cosmos3_edge.yaml and configs/cosmos3_nano_ar.yaml
do), which adds the video_gen_ar walk and a vae_decoder_ar node in its
own window_decoder partition; a request opts in per call:
Knob |
Default |
Meaning |
|---|---|---|
|
— ( |
|
|
61 on Edge, 29 elsewhere; |
Pixel frames per window (quantized to latent frames). |
|
32 on Edge, 8 elsewhere |
|
|
61 |
|
|
|
Emit each window as its own video chunk as it is decoded instead of one
assembled clip. |
|
— |
Names a world-state session: the DiT node keeps the rollout’s last window
and the decoder its context. The most recent |
|
|
Continue the named session: window 0 is conditioned on the stored last
frames (pinned clean, like a chained overlap) and only the |
|
600 |
Seconds an idle session is kept after its request finishes, capped at the
deployment’s |
|
|
Drop the named session once this request is done (with or without
|
curl -sN http://localhost:8000/generate \
-F 'text=a drone flies over a coastal town at dawn' \
-F 'output_modalities=video' \
-F 'model_kwargs={"num_frames":241,"window_mode":"chained","stream_video":true}'
Warning
window_mode: kv is experimental, and the server logs a warning at
startup when windowed video is enabled. The checkpoints were not trained
block-causally: committed K/V conditions the next window far more weakly
than frames pinned in the sequence, so an image-conditioned kv rollout
jumps away from the scene at the first window boundary and, unless the
prompt describes the image, can turn to texture noise within a few
windows. chained with Edge’s 61/32 defaults holds the scene over 241
frames when the prompt describes the image; its first chunk arrives later,
since window 0 is 61 frames. A prompt that
contradicts the image still takes over within a few windows in every
windowed mode; non-windowed clips are not affected.
The schedule is padded up to whole windows and the video trimmed back to
num_frames; a seeded request is deterministic end to end (later windows draw
their noise from the same generator). Windowed requests batch with each other
and with plain requests at the same walk, under the same 16384-vision-token
cap on the step (each windowed request counts its current window).
gen_capture_video lists (height,
width, frames) tiers whose denoise steps replay a per-step CUDA graph (one graph per
latent shape, the clean/noisy frame layout carried as a mask input; plain t2v/i2v and
chained windows, never kv windows). It is empty by default: at 832x480 the
graph, which captures the paged attention, measured 3-6% slower than the eager dense
FA3 step for both the 121-frame clip and the 29-frame window, so it only pays for
small, launch-bound tiers.
Wan2.2 (wan22)#
Text-to-video and image-to-video on Wan2.2-TI2V-5B — the dense 5B variant
(Wan-AI/Wan2.2-TI2V-5B-Diffusers): a native video DiT, a UMT5-XXL prompt
encoder and the Wan2.2-VAE, all four nodes stateless. The A14B (MoE dual-DiT)
variants are not supported; wan22 rejects any other variant explicitly.
Install and serve on one GPU:
pip install -e ".[wan22]"
mstar serve wan22 # configs/wan22.yaml
# or: mstar-serve --config configs/wan22.yaml --port 8000
Two routes are served. POST /generate is the native one (multipart form, like
every other model); the mp4 comes back base64-encoded in outputs.video[0].data:
curl -s http://localhost:8000/generate \
-F 'text=a fluffy cat walking across a sunlit floor' \
-F 'output_modalities=video' -F 'streaming=false' \
-F 'model_kwargs={"height":480,"width":832,"num_frames":33,"num_inference_steps":50,"guidance_scale":5.0}'
POST /v1/videos/generations is the OpenAI-shaped surface (JSON body, mp4 in
data[0].b64_json). Here the size is a single WxH string — width first,
the opposite order to the height/width kwargs above — and supplying an
image (URL or data URI) turns the request into image-to-video:
curl -sS -X POST http://localhost:8000/v1/videos/generations \
-H 'Content-Type: application/json' \
-d '{"prompt": "a fluffy cat walking across a sunlit floor",
"size": "832x480", "num_frames": 33, "seed": 42,
"num_inference_steps": 50, "guidance_scale": 5.0}' \
| python -c "import sys,json,base64; d=json.load(sys.stdin); \
open('out.mp4','wb').write(base64.b64decode(d['data'][0]['b64_json']))"
test/wan22/t2v_request.sh and i2v_request.sh wrap these two calls.
Generation knobs (per request, via model_kwargs or the request body):
Knob |
Default |
Notes |
|---|---|---|
|
704 / 1280 |
The checkpoint’s native 720P tier. Both must be multiples of 32 — see below. Rejected with a 400 otherwise. |
|
81 |
Must be 4k+1 — see below. Rejected with a 400 otherwise. Latent
frames = |
|
50 |
Clamped to |
|
5.0 |
Classifier-free guidance; run as a single batched forward. |
|
|
Empty by default, matching the reference pipeline. |
|
24 |
Playback rate only — it is the mp4 container rate, not a generation
knob. Wan2.2 always generates a fixed |
The ÷32 rule. Height and width must each be an exact multiple of 32:
a pixel dimension is downsampled 16x by the VAE and then patchified 2x by the
DiT, and only exact multiples survive both. So 720x1280 is not a valid
size for this model (720/32 = 22.5) — the 720p-class tier is **704**x1280. An
unaligned size is rejected at the request seam with a 400 naming the rule and
the nearest valid sizes, because it has no clean failure deeper in: the two
paths round the latent extent differently and the DiT dies mid-forward.
The 4k+1 frame rule. num_frames must be one more than a multiple of 4
(33, 81, 121 …): the VAE compresses time by 4 around an anchor frame, so only
4k+1 survives the round trip. Anything else is silently floored — ask for
32 frames and you would get 29 — so it too is rejected with a 400 naming the
nearest valid counts.
Size ceiling. height x width x num_frames may not exceed
max_video_pixels (default 268,435,456, about 3.6x the default clip; raise
it under the yaml’s model_kwargs), each axis must fit the DiT’s 1,024
rotary positions (num_frames up to 4,093), and height and width are at most
16,384, the largest side the mp4 encoder (libx264) opens. Each is a 400 naming
the limit.
Compiled DiT shapes. With compile_dit the DiT compiles one graph per
input shape, so only the "WxHxF" shapes in compile_dit_sizes (default
1280x704x81) use the compiled transformer; each compiles on first use, once
with CFG and once without, so at most two compiles per listed shape. Any other
size or frame count runs the same transformer eager, with no compile, which is
slower per request once warm: on an H200, about 49% more at 640x352x17 and 55%
more at 832x480x33 than the same shape compiled. List the shapes a deployment
serves under the yaml’s model_kwargs; each one listed costs its compiles on
first use. The first DiT call after startup is traced twice (a guard on the
rotary table), so a server compiles at most two graphs per listed shape plus
one.
UniPC runs inline, inside the DiT node. The scheduler is the DiT scaffold’s
(components/diffusion/unipc.py, which cosmos3 runs too), not a separate
stage: the solver state (the order-2 history buffer and the corrector’s
last_sample) is carried on the denoise loop’s own edges, so it travels with
the request rather than living in a scheduler object on one rank. Requests are
therefore independent and the loop is resumable across ranks. dit_max_batch_size
(default 1) lets requests at one latent grid and guidance scale share a forward,
each row count compiling its own graph.
What is accelerated by default. The DiT runs under torch.compile
(compile_dit); its self-attention runs on the engine’s ragged attention
resource (FlashInfer, FA3 on Hopper) as an opaque op inside the compiled graph,
6% off a 1280x704x81 denoise step on an H200 against cuDNN SDPA, measured on
an earlier version of the DiT block, before the adaLN was computed in one pass
and the q/k norm and RoPE were fused
(model_kwargs: attention_backend: sdpa restores the reference kernel; the
cross-attention over the 512 text tokens stays on SDPA, which is faster at that
shape). The untiled VAE decode runs the shared Wan VAE decoder,
channels-last and compiled (vae_decode_fast; set it to false under the
yaml’s model_kwargs for stock vae.decode). Both compile on first use:
the DiT only for the compile_dit_sizes shapes (see Compiled DiT shapes), so
the first request at a listed shape is slower. The VAE decode compiles once per resolution, and only
for the sizes in vae_decode_fast_sizes (model default 1280x704; the
shipped configs add 832x480, where it decodes 1.59x faster); any other
size decodes with stock vae.decode, frames identical to
vae_decode_fast: false. The compile uses Inductor’s deterministic mode, so
every server process decodes the same latents to the same frames; torch 2.9.1
has no such mode, and there two processes can differ by a few uint8 levels.
The VAE decode
tiles only when free VRAM at decode time is short of the untiled estimate
(the auto policy; WAN22_VAE_DECODE_TILING forces a path), which keeps the untiled conv3d from OOMing a
32 GiB card. No CUDA-graph capture, no continuous batching (one request per
step) and no component offload.
Two GPUs. configs/wan22_dp2.yaml serves a whole replica per GPU: 5.2
videos/min at 4 concurrent on H200s (measured on an earlier version), while one
request takes the 1-GPU time.
LTX-2.5 (ltx2_5)#
Text-to-audio+video on Lightricks LTX-2.5 (Lightricks/LTX-2.5-Diffusers, gated):
a 22B joint audio-video DiT, a Gemma-4 12B text encoder, and video/audio VAEs with a
48 kHz vocoder. The distilled checkpoint runs the model card’s recipes without
guidance; the full (SFT) transformer, image-to-video and the duration head are not
ported yet.
Five nodes: text_encoder (native Gemma-4 + the text connectors), dit (the
DiT scaffold’s denoise loop; video and audio latents carried in fp32; every attention
through two ragged attention resources), latent_upsampler (two-stage recipe only),
vae_decoder and audio_decoder. The walk follows the requested output
modalities, so ["video"] skips the audio decode (the model still denoises both).
mstar serve ltx2_5 # configs/ltx2_5.yaml: DiT on GPU 0, the rest on GPU 1
mstar serve ltx2_5 --config configs/ltx2_5_tp2.yaml # DiT tensor-parallel over 2 GPUs
mstar serve ltx2_5 --config configs/ltx2_5_sp2.yaml # DiT sequence-parallel over 2 GPUs (fastest)
Request knobs (model_kwargs): height / width (multiples of 32; 64 for the
two-stage recipe), num_frames (8k+1), fps, seed, and recipe:
single(default): 8 distilled sigmas at the requested size (default 544x960x121).two_stage: the card’s better-quality path — 8 sigmas at half size, the x2 latent upsampler, then 3 sigmas at full size (default 1088x1920). Frames above 720p decode in VAE tiles, as the card does for this decode.
Outputs are an H.264 video and a 48 kHz stereo audio chunk (16-bit PCM).
POST /v1/videos/generations returns one mp4 with the audio muxed in as AAC
("audio": false in the body for video alone); the mux needs the ffmpeg binary
on the server’s PATH and falls back to video only without it.
ltx2_5_sp2 shards the video tokens (Ulysses): each rank holds the whole DiT and
half the tokens, and only the video self-attention exchanges them (one all-to-all
each way); the cross-attentions run on each rank’s own tokens, and the audio queries
over the video merge the ranks’ partial attentions. ltx2_5_tp2 shards the
weights instead and all-reduces after every block’s attentions and feed-forwards.
Performance (2x H100, warm; diffusers with the same recipe and seeds):
544x960x121, one request |
7.87 s |
3.70 s |
3.15 s |
544x960x121, 4 concurrent |
7.6 videos/min |
19.6 videos/min |
22.3 videos/min |
two-stage 1088x1920x121, one request |
21.6 s |
11.0 s |
11.3 s |
two-stage, 2 concurrent |
2.8 videos/min |
5.9 videos/min |
6.1 videos/min |
The DiT is compiled and its denoise step captured per shape (capture_shapes, which
the two-stage recipe’s stages replay too); other shapes run the compiled eager path, and the first
request of a new shape pays its compile. vae_compile (on in these configs) compiles the
video VAE’s decoder at load for vae_compile_shapes, by default the same shapes (~55 dB
from the eager decode; other shapes decode eager). The SP2 config doesn’t capture 1088x1920
but lists it there, so its two-stage decode runs compiled too.
FLUX.2 [klein] (flux2_klein / flux2_klein_9b)#
Text-to-image and reference-image editing on the step-distilled FLUX.2 [klein]
checkpoints. Four stateless nodes: a native Qwen3 encoder that runs only the 27 layers
whose hidden states the DiT consumes (taps 9/18/27 concatenated), the rectified-flow
transformer as the body of a denoise_loop (one Euler step per iteration, 4 by
default, no classifier-free guidance), the FLUX.2 VAE encode of reference images and
the VAE decode. All of them are exact ports; the CPU suite pins them bit-for-bit against
the diffusers modules on tiny random configs and the GPU suite compares real-weight
trajectories and PSNR against a recorded pipeline run
(test/flux2_klein/record_oracle.py).
Serve on one GPU and generate:
mstar serve flux2_klein --gpus 0
python - <<'PY'
from mstar import MStarClient
client = MStarClient("http://localhost:8000")
png = client.generate_image("a cat holding a sign that says hello world", width=1024, height=1024, seed=0)
open("cat.png", "wb").write(png)
open("edit.png", "wb").write(client.edit_image("make it a watercolor painting", "cat.png", seed=1))
PY
The OpenAI routes are POST /v1/images/generations (size as WxH, seed,
n; num_inference_steps through extra_body) and POST /v1/images/edits
(multipart image + prompt; up to four reference images are concatenated as
conditioning tokens, in order).
Deployment knobs live under model_kwargs in configs/flux2_klein.yaml:
attention_backendsdpa(default) is the reference kernel, cuDNN on an H100.flashinferruns the DiT’s joint attention on the engine’s ragged FlashInfer resource; both are CUDA-graph replayable.compiletorch.compileof the transformer, one trace per shape.compile_eager_roundingInductor rounds intermediates where eager PyTorch does, and fuses no FMAs.
compile_exact_opstrue, or a list of op classes amongnormsandactivations: those modules keep the eager result inside the compiled forward, so inductor only fuses the chains around the GEMMs and attention. Each kept module other than a 1-Dtorch.nn.RMSNormis a graph break, so this is a latency-for-exactness trade:The norms alone make the transformer bit-exact with eager (measured on klein-4B and 9B while the q/k norms were graph breaks), so klein-4B and 9B ship
[norms]. Klein’s per-head q/k norms (1-DRMSNorm) run as an op the compiled graph keeps whole. On NVIDIA GPUs that op is a Triton replica of torch’s kernel with the same arithmetic order: bit-identical and faster than torch’s one-block-per-row launch. Kept norms of one class share a class-level forward, so the blocks share one compiled frame: with a forward per module, dynamo hit its recompile limit and the small batches ran the blocks eagerly (klein-4B 2.86 / 3.10 / 3.69 against 3.33 / 3.75 / 3.80 img/s at concurrency 1 / 4 / 16, H200). Keeping the activations too adds cost and nothing else.Z-Image’s 180 norms per step cost 45% of its B=1 latency, so it ships
falseand lands at a median 34 dB (thefalsefigures were measured before its compiled forward fused the q/k norms and rotary into one kernel).[norms], which keeps that fusion off, turns it reference-faithful (at least 55.9 dB on every prompt).Without
[norms], the plain compile — even with eager rounding — lands at a median 37.6 dB (klein-4B) and 35.6 dB (9B) PSNR from eager over the 100 protocol prompts (measured with the earlier compiled VAE decode), because a 4- or 8-step distilled sampler amplifies the last bit of inductor’s own reductions and activation decompositions.
cuda_graph,capture_sizes,capture_batch_sizesThe denoise step, Euler update included, is captured per listed
[height, width]and batch size. Other shapes run the eager batched path.capture_edit_sizesklein only: edit buckets with one reference image of the output size, the shape of an edit that keeps its reference’s size. Defaults to
capture_sizes;[]captures none. The eight edit buckets of the 1024² default cost about 8 GiB of peak VRAM on klein-4B and 6 GiB on 9B, and every extra capture size adds roughly 7 minutes of startup.async_schedulingThe engine’s speculative scheduling of the image nodes: the worker assembles a batch’s next denoise step while the current one runs and merges requests that became ready meanwhile, so requests at different steps share a forward. On by default; a step a request speculates past its schedule is vetoed before any forward, and
falserestores lockstep. Measured on one H100 back to back against lockstep: klein-4B 0.365 to 0.372 s vs 0.377 to 0.390 s at B=1 and about 3% more images per second at 4 and 16 concurrent requests, klein-9B 2%, Z-Image neutral, served images unchanged — the denoise phase is compute-bound either way.max_batch_sizeRows batched per denoise step.
vae_compiletorch.compileof the VAE decode with inductor autotuning, each nearest-x2 upsample folded into its conv and the GroupNorms (with their SiLU) on a split-reduction kernel. Before those two it measured 89 to 29 ms at 1024² on an H100 and about 55 dB PSNR from the eager decode on every prompt; neither figure is re-measured since. The compiled decode is not bit-exact, and the autotuner may pick different conv kernels in another server process, so set it tofalsefor bit-exact, repeatable output — which is how the parity suite runs. The compiled decode is warmed at load for thecapture_sizesgrids and the decode batch sizes; any other latent shape decodes eagerly rather than paying a 40 to 100 s autotune inside a request. Memory cuts the other way: the eager decode of an 8-image batch at 1024² allocates about 40 GiB of activations, for served peaks of 75 GiB on klein-4B and 77 GiB on Z-Image at 16 concurrent requests against 36 to 44 GiB compiled.max_image_areaPixels, default 2048². Bounds the output size a request may ask for; larger requests are rejected with a 400 before scheduling instead of occupying the worker for minutes (an 8192² request means 262k tokens of quadratic attention).
With the klein defaults every served image was within 53 dB of the eager path on all 100
protocol prompts, and with vae_compile: false it was bit-exact; both were measured while
the q/k norms were graph breaks, and the 53 dB before the folded, split-GroupNorm VAE decode.
For scale, SGLang 0.5.22’s images sit at a median 23.8 dB (minimum 13.1) on klein-4B and
24.0 dB (14.5) on 9B from klein’s output at the same seeds.
Requests at the same output size batch across users in every node, including the
text encoder, whose input is always 512 tokens. lora lists adapters to fold into the
transformer weights at load time ([{path: ..., scale: ...}]; diffusers/PEFT-format or
BFL-layout safetensors), so a styled deployment runs at the base model’s speed. The adapter’s
own scaling follows PEFT: an alpha key in the weights, else the lora_alpha (and
use_rslora) of an adapter_config.json / config.json next to them, else alpha = r;
scale multiplies that (a public rank-64 rsLoRA adapter lands within 53 dB of diffusers’
fused result — M* merges in fp32 with one rounding, PEFT fuses in bf16).
A DoRA adapter (PEFT’s lora_magnitude_vector or Kohya’s dora_scale) is refused,
since its magnitudes would be dropped from the merge.