Hello,
A last enhancement issue, could you make a pass on the warnings (add some constructors, remove some async keywords for example) and rename some variables https://github.com/Lokad/Tokenizers/blob/master/src/Lokad.Tokenizers/Tokenizer/BaseTokenizer.cs#L711 to respect the dotnet conventions) please?