Supported Models

mstar ships the following model families. The table below summarizes the registered families, their registry key (the value of model: in a config YAML), and a representative Hugging Face identifier.

Registry keys live in mstar/model/registry.py (MODEL_REGISTRY / HF_MODELS).

Registered model families

Registry key	Example Hugging Face model ID	Description
bagel	ByteDance-Seed/BAGEL-7B-MoT	Unified multimodal model (text + image understanding and generation).
cosmos3	nvidia/Cosmos3-Nano	Cosmos3 world model: t2i/t2v/i2v/v2v diffusion, robot-action modes, opt-in sound.
cosmos3_droid	nvidia/Cosmos3-Nano-Policy-DROID	Cosmos3 action-policy fine-tune for the DROID platform (domain_name droid_lerobot, 10-dim raw actions); no sound pathway. The config serves the released policy sampling defaults (4 steps, guidance 3.0).
cosmos3_super	nvidia/Cosmos3-Super	Cosmos3-Super (64B) variant of the above; TP/SP for multi-GPU serving.
orpheus	canopylabs/orpheus-3b-0.1-ft	TTS: Llama 3.2 3B LLM emitting audio tokens + SNAC 24 kHz decoder.
pi05	lerobot/pi05_base	Pi0.5 vision-language-action robotics model (ViT encoder + LLM + flow action expert).
qwen3_omni	Qwen/Qwen3-Omni-30B-A3B-Instruct	Omni-modal (text/image/audio/video in, text/audio out): Thinker + Talker + codec.
qwen3_tts	Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice	Streaming text-to-speech with built-in speakers: Talker + 12 Hz speech codec.
vjepa2	facebook/vjepa2-vitl-fpc64-256	V-JEPA 2 video encoder + masked predictor.
vjepa2_ac	vjepa2-ac-vitg	V-JEPA 2-AC encoder + action-conditioned predictor.
whisper_large (Beta)	openai/whisper-large-v3	Encoder-decoder ASR (audio in, transcript out). Beta / un-optimized.
higgs_audio (Beta)	bosonai/higgs-audio-v3-stt	Audio-tower + Qwen3 LLM speech-to-text. Beta / un-optimized.

Notes

• Models marked (Beta) are functionally supported but not yet performance-optimized; treat their throughput/latency as provisional.

• The IDs above are representative. You may use local paths or compatible variants.

• Some families accept multimodal input (image/audio/video); see the model’s process_prompt for the inputs it expects.

• To add a new family, see Adding a New Model.

Qwen3-TTS notes

• Install the model-specific dependencies with pip install -e '.[qwen3_tts]' and launch the default single-GPU deployment with mstar serve qwen3_tts --gpus 0.

• The first integration supports the CustomVoice checkpoint and text-to-audio requests. voice selects one of the checkpoint’s built-in speakers and language defaults to automatic detection.

• Codec CUDA graphs are captured through batch size 8. The upstream decoder’s batch-16 capture can exhaust an H100 after Talker weights and CodePredictor graphs are resident; larger Codec batches therefore use the scheduler’s safe ceiling.

• Talker prefill remains eager because it runs once with variable sequence lengths. Decode uses a whole-walk CUDA Graph when residual sampling matches the captured defaults; request-local EOS suppression is carried as a graph tensor input so replay does not consult capture-slot dummy request state. Requests with custom subtalker_* sampling fall back to eager Talker execution while retaining the piecewise-captured 15-step CodePredictor loop.

• The 12 Hz decoder does not require the system SoX executable. M* imports only the exact upstream decoder modules, avoiding qwen-tts’s unrelated 25 Hz SoX probe during worker startup.

For throughput/latency validation, run the native serving benchmark with the Qwen3-TTS model metadata rather than the Orpheus compatibility entry:

python -m benchmark.runner \
    --url localhost:8000 \
    --model qwen3_tts \
    --profiling-type closed_loop \
    --request-type text_to_speech \
    --num-requests 20 \
    --inference-system ours \
    --num-warmup 2 \
    --max-concurrency 4 \
    --dataset seed_tts \
    --output-dir .bench_outs

The benchmark stops on the model’s natural codec EOS by default. Use --ignore-eos --output-len-min N --output-len-max N only when measuring fixed-length decode throughput rather than end-user latency. The first process-local request can include eager FlashInfer kernel JIT, so keep the warmup requests enabled when reporting steady-state latency.

Cosmos3 environment requirements

• flashinfer is required: it is the paged KV/attention backend used by the prefill, the captured CUDA graphs, and multi-request batches.

• The default denoise attention backend is dense_gen (Cosmos3Config.attention_backend), which runs bs=1 eager generation attention as one FlashAttention-3 varlen kernel from the fa3-fwd wheel. That wheel is ABI-tied to the installed torch/CUDA build (Hopper builds exist for at least torch 2.9 + cu12.8 and torch 2.11 + cu13.0); install the one matching your environment. When it is not importable, the engine logs a warning at startup and automatically falls back to the paged flashinfer backend — serving still works, only the bs=1 dense fast path is lost. model_kwargs.attention_backend: flashinfer in the config YAML selects the paged backend explicitly.

• Video-input requests (video-to-video, action inverse-dynamics) decode the conditioning clip with torchcodec; environments without it reject those requests at preprocessing (other modes are unaffected).

• Generated video containers are written with torchcodec’s VideoEncoder when available (torchcodec >= 0.9), otherwise with torchvision’s write_video, which needs the PyAV (av) package.

• Sound-enabled video responses mux the AAC track with the ffmpeg and ffprobe binaries, which must be on PATH (system packages, not pip-installable).

• The Wan-VAE decode dtype is gated on the cuDNN build: bf16 needs cuDNN >= 9.16 (fast Hopper bf16 conv3d); older cuDNN serves the decode in fp32/TF32 automatically.