Does the TTS model have instruction controls on top of voice cloning?

[GalenMarek14]

Does the TTS model have instruction controls on top of voice cloning?

Or, can it change emotions based on the context of the text? For example, can we clone a voice and then instruct it to sound angry, happy, shout, whisper, etc.? Or does it automatically adjust these based on the text?

Does it support non-verbal sounds like laughter or other paralinguistic elements?

Thanks!

[YWMditto]

We’re very happy that you’re paying close attention to our models and taking the time to learn about the capabilities they currently support.

Below are our answers to your questions:

1. At the moment, we do not have a single model that supports both voice cloning and instruction-based control at the same time. These two capabilities are currently provided by two different models: MOSS-TTS and MOSS-VoiceGenerator, respectively.

2. (1) Absolutely! MOSS-TTS is a very powerful, pure pre-trained foundation model. You can think of it as analogous to a text foundation model that has not undergone SFT or RLHF. As such, it has strong potential to be extended to support the kinds of features you described. When the reference audio itself contains emotion, the generated audio will also tend to mimic that emotion.
   (2) The feature you described—“first clone a voice, then allow the cloned voice (timbre) to change emotion”—is indeed beyond the current scope of our models. We will seriously consider this use case, and if it is approved internally, we hope to release a model supporting this in the near future. We’d really appreciate it if you could keep an eye on our company and our open-source updates.
   (3) As mentioned above, since our model is a pre-trained foundation model, yes—it will to some extent automatically adjust paralinguistic attributes in the generated audio based on the text content.

3. I understand you are referring to the sound event capability. Currently, MOSS-TTS does not support this feature. We may consider adding it in the next version, but for now it is not among our most urgent priorities.

If you have any further questions, feel free to continue the discussion with us!

[GalenMarek14]

Thank you very much for the response.

By the way, can you give ballpark figures for VRAM requirements and generation times (for both the 1.7B and 8B versions)? If the 8B model requires relatively high VRAM (16 GB or more) in FP16, is it possible to use quantizations like FP8?

[xiaoyangnihao]

Looking forward to the emotion or style control with clone voice too.

[YWMditto]

1. Below are the GPU memory requirements (including both the model and the audio tokenizer) and the inference speed we measured on an NVIDIA L20 (the inference speed on an RTX 4090 is generally in a similar range):

• 1.7B — 12GB VRAM — 42 tokens/s
• 8B — 24GB VRAM — 32 tokens/s

2. We will seriously consider planning this after the Chinese New Year holiday. Inference efficiency is absolutely one of our top priorities. Thank you very much for your feedback.