mstar.model.cosyvoice3.config

Contents

mstar.model.cosyvoice3.config#

Configuration for Fun-CosyVoice3 (FunAudioLLM/Fun-CosyVoice3-0.5B-2512).

The checkpoint ships a hyperpyyaml file (cosyvoice3.yaml) that builds the reference classes directly; there is no config.json. The dataclasses below write its values down once. Every shape they declare is checked against the checkpoint when the weights load.

Sources (read, not copied): cosyvoice3.yaml from the HF snapshot and cosyvoice/{llm/llm.py,flow/flow.py,flow/flow_matching.py,flow/DiT/*, hifigan/generator.py,hifigan/f0_predictor.py,cli/model.py,cli/frontend.py} from FunAudioLLM/CosyVoice.

The reference-audio front end is Chatterbox’s: the 25 Hz speech tokenizer v3 is Chatterbox’s S3Tokenizer with twelve blocks instead of six, the speaker encoder is the same CAMPPlus, and the prompt mel is the same 24 kHz Matcha mel with fmax left at Nyquist. Their configs are Chatterbox’s dataclasses.

Classes

CausalHiFTConfig([in_channels, ...])

CausalHiFTGenerator: Chatterbox's HiFT hyper-parameters, with causal convolutions, a causal F0 predictor (three frames of look-ahead), a pre-convolution that looks four frames ahead and fixed excitation noise.

CosyVoice3Config(lm, sampling, flow, hift, ...)

DiTConfig([dim, depth, heads, dim_head, ...])

FlowConfig([vocab_size, input_size, ...])

CausalMaskedDiffWithDiT: tokens -> mu (embedding, look-ahead convs, x2 repeat) -> mel by a classifier-free-guided Euler solve of the DiT.

LMConfig([hidden_size, num_hidden_layers, ...])

CosyVoice3LM: a Qwen2 0.5B backbone (CosyVoice-BlankEN) with its own speech-token embedding and head over speech_token_size + 200 ids.

SamplingDefaults([temperature, top_p, ...])

Repetition-aware sampling (cosyvoice.utils.common.ras_sampling): a nucleus draw (top-p over at most top-k ids); a draw that already appears >= ras_window * ras_tau times in the last ras_window tokens is masked and redrawn from the full distribution.
class mstar.model.cosyvoice3.config.CausalHiFTConfig(in_channels=80, base_channels=512, nb_harmonics=8, sampling_rate=24000, nsf_alpha=0.1, nsf_sigma=0.003, nsf_voiced_threshold=10.0, upsample_rates=(8, 5, 3), upsample_kernel_sizes=(16, 11, 7), istft_n_fft=16, istft_hop_len=4, resblock_kernel_sizes=(3, 7, 11), resblock_dilation_sizes=((1, 3, 5), (1, 3, 5), (1, 3, 5)), source_resblock_kernel_sizes=(7, 7, 11), source_resblock_dilation_sizes=((1, 3, 5), (1, 3, 5), (1, 3, 5)), lrelu_slope=0.1, audio_limit=0.99, f0_cond_channels=512, conv_pre_look_right=4, f0_look_right=3, noise_seed=1986, noise_seconds=300.0)[source]#

Bases: S3GenHiFTConfig

CausalHiFTGenerator: Chatterbox’s HiFT hyper-parameters, with causal convolutions, a causal F0 predictor (three frames of look-ahead), a pre-convolution that looks four frames ahead and fixed excitation noise.

Parameters:

  • in_channels (int)

  • base_channels (int)

  • nb_harmonics (int)

  • sampling_rate (int)

  • nsf_alpha (float)

  • nsf_sigma (float)

  • nsf_voiced_threshold (float)

  • upsample_rates (tuple[int, ...])

  • upsample_kernel_sizes (tuple[int, ...])

  • istft_n_fft (int)

  • istft_hop_len (int)

  • resblock_kernel_sizes (tuple[int, ...])

  • resblock_dilation_sizes (tuple[tuple[int, ...], ...])

  • source_resblock_kernel_sizes (tuple[int, ...])

  • source_resblock_dilation_sizes (tuple[tuple[int, ...], ...])

  • lrelu_slope (float)

  • audio_limit (float)

  • f0_cond_channels (int)

  • conv_pre_look_right (int)

  • f0_look_right (int)

  • noise_seed (int)

  • noise_seconds (float)

conv_pre_look_right: int = 4#
f0_look_right: int = 3#
noise_seconds: float = 300.0#
noise_seed: int = 1986#
source_resblock_kernel_sizes: tuple[int, ...] = (7, 7, 11)#
property stream_holdback_frames: int#

Mel frames at the end of a non-final call whose audio is not final yet: the F0 predictor’s and the pre-convolution’s look-ahead, plus the frame the inverse STFT cannot finish.

class mstar.model.cosyvoice3.config.CosyVoice3Config(lm: 'LMConfig' = <factory>, sampling: 'SamplingDefaults' = <factory>, flow: 'FlowConfig' = <factory>, hift: 'CausalHiFTConfig' = <factory>, tokenizer: 'S3TokenizerConfig' = <factory>, xvector: 'S3GenXVectorConfig' = <factory>, mel: 'S3GenMelConfig' = <factory>, llm_weights: 'str' = 'llm.pt', flow_weights: 'str' = 'flow.pt', hift_weights: 'str' = 'hift.pt', speech_tokenizer_onnx: 'str' = 'speech_tokenizer_v3.onnx', campplus_onnx: 'str' = 'campplus.onnx', text_tokenizer_dir: 'str' = 'CosyVoice-BlankEN', silent_tokens: 'tuple[int, ...]'=(1, 2, 28, 29, 55, 248, 494, 2241, 2242, 2322, 2323), max_silent_tokens: 'int' = 5, max_text_tokens: 'int' = 512, max_reference_seconds: 'float' = 30.0, max_speech_tokens: 'int' = 2048, voice_cache_size: 'int' = 64, max_new_tokens_limit: 'int | None' = None, stream_hop_tokens: 'int' = 25, stream_hop_growth: 'float' = 2.0, stream_max_hop_tokens: 'int' = 100, stream_poll_tokens: 'int' = 25, hift_context_frames: 'int' = 64, flow_estimator_dtype: 'str' = 'bfloat16', flow_max_batch_size: 'int' = 8, hift_max_batch_size: 'int' = 8)[source]#

Bases: object

Parameters:
campplus_onnx: str = 'campplus.onnx'#
flow: FlowConfig#
flow_estimator_dtype: str = 'bfloat16'#
flow_max_batch_size: int = 8#
flow_weights: str = 'flow.pt'#
hift: CausalHiFTConfig#
hift_context_frames: int = 64#
hift_max_batch_size: int = 8#
hift_weights: str = 'hift.pt'#
llm_weights: str = 'llm.pt'#
lm: LMConfig#
max_new_tokens_limit: int | None = None#
max_reference_seconds: float = 30.0#
max_silent_tokens: int = 5#
max_speech_tokens: int = 2048#
max_text_tokens: int = 512#
mel: S3GenMelConfig#
sampling: SamplingDefaults#
silent_tokens: tuple[int, ...] = (1, 2, 28, 29, 55, 248, 494, 2241, 2242, 2322, 2323)#
speech_tokenizer_onnx: str = 'speech_tokenizer_v3.onnx'#
stream_hop_growth: float = 2.0#
stream_hop_tokens: int = 25#
stream_max_hop_tokens: int = 100#
stream_poll_tokens: int = 25#
text_tokenizer_dir: str = 'CosyVoice-BlankEN'#
tokenizer: S3TokenizerConfig#
voice_cache_size: int = 64#
xvector: S3GenXVectorConfig#
class mstar.model.cosyvoice3.config.DiTConfig(dim: 'int' = 1024, depth: 'int' = 22, heads: 'int' = 16, dim_head: 'int' = 64, ff_mult: 'int' = 2, mel_dim: 'int' = 80, mu_dim: 'int' = 80, spk_dim: 'int' = 80, conv_pos_kernel: 'int' = 31, conv_pos_groups: 'int' = 16, time_freq_dim: 'int' = 256, static_chunk_size: 'int' = 50)[source]#

Bases: object

Parameters:

  • dim (int)

  • depth (int)

  • heads (int)

  • dim_head (int)

  • ff_mult (int)

  • mel_dim (int)

  • mu_dim (int)

  • spk_dim (int)

  • conv_pos_kernel (int)

  • conv_pos_groups (int)

  • time_freq_dim (int)

  • static_chunk_size (int)

conv_pos_groups: int = 16#
conv_pos_kernel: int = 31#
depth: int = 22#
dim: int = 1024#
dim_head: int = 64#
ff_mult: int = 2#
heads: int = 16#
mel_dim: int = 80#
mu_dim: int = 80#
spk_dim: int = 80#
static_chunk_size: int = 50#
time_freq_dim: int = 256#
class mstar.model.cosyvoice3.config.FlowConfig(vocab_size=6561, input_size=80, output_size=80, spk_embed_dim=192, token_mel_ratio=2, pre_lookahead_len=3, pre_lookahead_channels=1024, dit=<factory>, cfm=<factory>, noise_seed=0, noise_frames=15000)[source]#

Bases: object

CausalMaskedDiffWithDiT: tokens -> mu (embedding, look-ahead convs, x2 repeat) -> mel by a classifier-free-guided Euler solve of the DiT.

Parameters:
cfm: S3GenCFMConfig#
property chunk_tokens: int#
dit: DiTConfig#
input_size: int = 80#
noise_frames: int = 15000#
noise_seed: int = 0#
output_size: int = 80#
pre_lookahead_channels: int = 1024#
pre_lookahead_len: int = 3#
spk_embed_dim: int = 192#
token_mel_ratio: int = 2#
vocab_size: int = 6561#
class mstar.model.cosyvoice3.config.LMConfig(hidden_size=896, num_hidden_layers=24, num_attention_heads=14, num_key_value_heads=2, head_dim=64, intermediate_size=4864, rms_norm_eps=1e-06, rope_theta=1000000.0, text_vocab_size=151936, speech_token_size=6561, num_control_tokens=200)[source]#

Bases: object

CosyVoice3LM: a Qwen2 0.5B backbone (CosyVoice-BlankEN) with its own speech-token embedding and head over speech_token_size + 200 ids.

Parameters:

  • hidden_size (int)

  • num_hidden_layers (int)

  • num_attention_heads (int)

  • num_key_value_heads (int)

  • head_dim (int)

  • intermediate_size (int)

  • rms_norm_eps (float)

  • rope_theta (float)

  • text_vocab_size (int)

  • speech_token_size (int)

  • num_control_tokens (int)

property eos_token: int#
head_dim: int = 64#
hidden_size: int = 896#
intermediate_size: int = 4864#
num_attention_heads: int = 14#
num_control_tokens: int = 200#
num_hidden_layers: int = 24#
num_key_value_heads: int = 2#
rms_norm_eps: float = 1e-06#
rope_theta: float = 1000000.0#
property sos_token: int#
speech_token_size: int = 6561#
property speech_vocab_size: int#
property task_id_token: int#
text_vocab_size: int = 151936#
class mstar.model.cosyvoice3.config.SamplingDefaults(temperature=1.0, top_p=0.8, top_k=25, ras_window=10, ras_tau=0.1, min_token_text_ratio=2.0, max_token_text_ratio=20.0)[source]#

Bases: object

Repetition-aware sampling (cosyvoice.utils.common.ras_sampling): a nucleus draw (top-p over at most top-k ids); a draw that already appears >= ras_window * ras_tau times in the last ras_window tokens is masked and redrawn from the full distribution.

Parameters:
max_token_text_ratio: float = 20.0#
min_token_text_ratio: float = 2.0#
ras_tau: float = 0.1#
ras_window: int = 10#
temperature: float = 1.0#
top_k: int = 25#
top_p: float = 0.8#
Contents