mstar.model.cosyvoice3.components.text#
CosyVoice3’s text tokenizer: the Qwen2 BPE of CosyVoice-BlankEN plus the
special tokens CosyVoice3 adds (prompt delimiters, paralinguistic tags, and
English and pinyin phonemes for pronunciation fixes such as [j][ǐ]).
The ids follow from the order the tokens are added, so the list below is the
reference’s (cosyvoice/tokenizer/tokenizer.py, CosyVoice3Tokenizer);
<|endofprompt|> must come out as 151646, which the LM was trained on.
Classes
|