Replace gpt3-tokenizer with js-tiktoken for Enhanced Tokenization Performance

[Anajrim01]

This PR replaces the existing gpt3-tokenizer with js-tiktoken, which has demonstrated improved performance in my benchmarks. Additionally, an exploration and comparison of BPE versus heap sorting using js-tiktoken have been conducted, with results indicating BPE (stock) provides better performance.

Benchmarks:
Screenshots of benchmark results are attached, providing a comprehensive comparison between the methods tested.

Key Changes:

• Refactor codebase to utilize js-tiktoken for tokenization.
• Update dependencies to their latest versions to ensure compatibility and stability.
• Implement and test faster tokenization methods.
• Benchmark heap sorting versus BPE sorting through js-tiktoken as described in this PR.
• Based on benchmarks, BPE sorting is retained due to its superior performance over heap sorting.

To-Do:

• Refactor code to integrate js-tiktoken.
• Update project dependencies to the latest versions.
• Implement faster tokenization using js-tiktoken.
• Conduct benchmarks comparing heap sorting with BPE sorting.
• Fine-tune and ensure optimal performance of BPE sorting (no further action required on heap sorting based on current benchmarks).

This PR should also fix #8, addressing current performance concerns and paving the way for future optimizations.

Attachments:

• Benchmark results screenshot comparing old tokenizer (gpt3-tokenizer) with js-tiktoken along with omparison of BPE versus heap sorting results, demonstrating BPE's superior performance.

• Conclusion

Please review the attached documentation and benchmark results to provide feedback or suggest further enhancements.

[vercel]

@Anajrim01 is attempting to deploy a commit to the ShipBit Team on Vercel.

A member of the Team first needs to authorize it.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
slickgpt	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Sep 14, 2024 9:24am

[Shackless]

Awesome! Back in the day when I built this, there weren't many client-side libs out there to calculate tokens. The larger ones required a full node environment but it's nice to see that there are better options now.