Minimal refactoring to user char[] instead of string for core processing

[EslaMx7]

Summary

This is a basic minimal effort to maintain unit test functionality after transitioning to char[]. I aim to highlight the specific areas in the code that are heavily reliant on string. For some of these areas, we will need to reconstruct/allocate new strings, while for others, we might have to rethink the logic to accommodate char[].

Open Questions:

1. Should we convert the string properties of all other classes to char[], similar to Node.Text?
2. What methods/tools should we use to evaluate the performance enhancements (speed, memory allocation, etc...) following this refactoring?

"If you can’t measure it, you can’t improve it." – Peter Drucker

Currently, I can track the total duration of all unit tests, though it's not a precise metric, as each tokenization test case runs for less than one second.

---

Changes

Refactor Token class to use char[] instead of string

1. Updated Token.Text from string to char[], requiring extensive changes across the codebase. Methods now handle char[] and ReadOnlySpan<char> for text manipulation.
2. Updated constructors, methods, and utilities to support the new type.
3. Added extension methods for SpanRuneEnumerator to List conversion.
4. Ensured all text processing functions in TokenizationUtils, BaseTokenizer, XLMRobertaTokenizer, and SentencePieceModel are compatible with char[].