better way to deal with multi-token words

[bnicenboim]

New paper about multi token words:
https://arxiv.org/pdf/2406.14561

[bnicenboim]

An alternative is https://arxiv.org/pdf/2410.02691

[bnicenboim]

https://arxiv.org/pdf/2406.10851