Continuing the Multi-Tokenizer Research

[chandralegend]

We will continue the Research on Multi Toeknizer, which we started at C4AI. The following will be the focus of this endeavor,

1. Improve the Multi-tokenizer Package (Better Language Detection, More Language Coverage)
2. Create Synthetic Data to cover cases such as code-switching, etc.
3. Train a base model with the synthetic train dataset for baseline
4. Train a model with the same architecture and same initial weights with multi-tokenizer (different variants)
5. Evaluate the models
6. Tralanguage-specificcific model (eg- sinhala+english, hindi+english) for both single tokenizer and multi tokenizer approaches and evaluate

[chandralegend]

We are seeking interested individuals to take part in this journey