Upgrade to the new Chatterbox Multilingual (v2) with 23-language support

[middi1408]

Hey there,

First off, thanks a lot for all the great work on this fork — the large text handling, audiobook generation, voice cloning, and the clean Web UI have made Chatterbox much more usable in real projects. Really appreciate the effort that went into this.

I just noticed that Resemble AI released a new version called Chatterbox Multilingual, which now supports 23 languages, emotion exaggeration, and comes with a new 0.5B Llama backbone.
It would be awesome if this fork could be updated to use the new model, ideally keeping all the current enhancements (FastAPI server, Docker setup, chunking, predefined voices, etc.).

Some highlights of the new version:

• Native multilingual TTS with 23 supported languages
• Zero-shot voice cloning across languages
• Emotion/exaggeration control
• Improved stability and pacing control
• Still MIT licensed

Possible next steps:

Swap in ChatterboxMultilingualTTS.from_pretrained() for the new model

• Add a language selection dropdown to the UI
• Expose new parameters like exaggeration and cfg_weight
• Check that chunking and audiobook generation still work properly

If you’re planning to upgrade, I’d be happy to help test it.

Thanks again for maintaining such a useful and polished project — it’s made a big difference for those of us running self-hosted TTS setups.

[AnnoyingTechnology]

Any love for this issue ?

[AnnoyingTechnology]

I managed to upgrade the rocm container, codebase and python dep to all line up and have it working for multilingual.

Doing so made me realize how absolutely incredible it is that these "AI" stacks even work at all. Considering that even a minor version change is able to break and pull its own slew of dependency breakages. The number of swiss cheese holes that need to line up perfectly is mind bogling. And even then, you are greeted with lines upon lines of deprecation warning, reminding you how fragile your house of card is.

Anyways, it was bruteforced by an LLM jackpot machine and not clean enough for a PR. But if someone's interested, I could publish a fork.

[devnen]

Thanks for bringing this up and for your work on the fork.

I have just released v1.0.0, which explicitly locks the project to the standard english-only model. This was a deliberate architectural choice to ensure maximum stability, lower VRAM usage, and to preserve the highest possible quality for English users.

Recent benchmarks and user reports suggest that the multilingual model can suffer from slight "interference" (accent bleed or silence artifacts) when generating English compared to the specialized single-language model. For that reason, I am not recommending the multilingual model as a default replacement for English-centric workflows.

That said, I absolutely plan to support the multilingual variant as an opt-in / experimental feature in a future release.

There is actually an active PR for this already: PR #80. Could you take a look at PR #80 and compare it with your implementation? If your fork addresses edge cases that PR #80 misses (like language auto-detection or improved config handling), I’d love for you to comment there or collaborate so we can merge the "best of both worlds."

Please share a link to your fork here so I (and others) can take a look.

[devnen]

I've spent some time digging through what's out there for multilingual implementations, and I want to put everything in one place so we have a solid starting point.

Official Implementation References

1. Resemble AI's Official Repository

• Source: resemble-ai/chatterbox
• Key Files:
  • multilingual_app.py - Gradio demo with language selection UI
  • example_tts.py - Basic usage examples
  • chatterbox/mtl_tts.py - Core multilingual implementation
• Usage Pattern:

from chatterbox.mtl_tts import ChatterboxMultilingualTTS

multilingual_model = ChatterboxMultilingualTTS.from_pretrained(device="cuda")
wav = multilingual_model.generate(text, language_id="fr")

2. Official Documentation

• HuggingFace Model Card: ResembleAI/chatterbox
• Resemble AI Blog Post: Introducing Chatterbox Multilingual
• Supported Languages: 23 languages (Arabic, Chinese, Danish, Dutch, English, Spanish, Finnish, French, German, Greek, Hebrew, Hindi, Italian, Japanese, Korean, Malay, Norwegian, Polish, Portuguese, Russian, Swedish, Swahili, Turkish)

Community Implementations Worth Studying

1. mschmidm's Fork (Actual Fork of This Repo)

• URL: mschmidm/Chatterbox-TTS-Server
• Status: Has "minimal support for chatterbox multilingual" (mentioned in Issue New Chatterboxmodel : 23 languages #57)
• Why it matters: Shows how to integrate multilingual into our existing codebase without breaking everything
• Note: @mschmidm if you're open to sharing implementation details or submitting a PR, that would be amazing!

2. travisvn's Independent Project (Not a Fork)

• URL: travisvn/chatterbox-tts-api
• Status: Production-ready with full 22-language support
• What they did differently:
  • USE_MULTILINGUAL_MODEL environment variable (defaults to true)
  • Language metadata stored with each uploaded voice
  • Automatic language detection from voice metadata
  • /languages endpoint to retrieve supported languages
  • React frontend with language selection dropdown + flag emojis
  • OpenAI API compatibility maintained
• Why it's useful: Great reference for UI/UX patterns, API design, and how to handle configuration without breaking existing setups

3. Pull Request #80 by @4-alok

• Status: Currently open and waiting for review
• Title: "Add Multilingual TTS Support with ChatterboxMultilingualTTS"
• Note: I need to carve out time to review this properly. If anyone has thoughts or has tested it, drop your feedback in that PR discussion!

Technical Implementation Checklist

Here's what we'll need to tackle based on what I've seen:

Core Model Integration

• Swap ChatterboxTTS.from_pretrained() → ChatterboxMultilingualTTS.from_pretrained()
• Add language_id parameter to generation calls
• Implement fallback logic (probably default to English for backward compatibility)
• Test chunking/audiobook generation with multilingual model

API Enhancements

• Add /languages endpoint (returns list of supported languages)
• Extend voice upload to include language metadata
• Modify voice library storage to preserve language info
• Make sure OpenAI compatibility still works

UI Updates

• Language selection dropdown in Web UI
• Display language badges/flags next to voices
• Update predefined voices to include language metadata
• Add language filter to voice library view

New Parameters (Optional but Worth Considering)

• exaggeration (emotion intensity: 0.25-2.0)
• cfg_weight (pacing control)
• Cross-language voice transfer support (set cfg_weight=0.0)

Configuration

• USE_MULTILINGUAL_MODEL env variable for backward compatibility
• Model path configuration for multilingual model
• Memory/performance optimization flags

Model Comparison

Feature	v1.0.0 (Standard)	Multilingual v2
Languages	English only	23 languages
Backbone	0.5B model	0.5B Llama backbone
Voice Cloning	Zero-shot	Cross-language zero-shot
Emotion Control	Basic	Exaggeration parameter
Pacing Control	-	cfg_weight parameter
Watermarking	-	Built-in
License	MIT	MIT

How You Can Help

1. Review PR Add Multilingual TTS Support with ChatterboxMultilingualTTS #80 - Any feedback on the implementation approach would be valuable
2. Test mschmidm's fork - Try it out and report back on what works and what doesn't
3. Share your use cases - What languages do you actually need? Which features matter most?
4. Performance testing - If you have multilingual audio samples to test against, please share them

Next Steps

I'm keeping this issue open as the main tracking thread for multilingual implementation. Before diving into code, I want to:

1. Compile all available resources (done above)
2. Review PR Add Multilingual TTS Support with ChatterboxMultilingualTTS #80 thoroughly
3. Figure out the best approach (environment variable toggle vs separate branch)
4. Get a development branch up for community testing

Timeline: Realistically, I'm aiming to get initial multilingual support into a dev branch within the next few weeks. Testing and feedback from you all will be critical before we merge anything to main.

---

Thanks for your patience and enthusiasm! This is going to be a big upgrade, and I want to make sure we do it right.

[drAkeSteinn]

That's great news!

To be honest, I'm a bit lost when it comes to programming, but I really enjoy experimenting with AI.

I really appreciate the work and enthusiasm you're putting into this project.