N-gram Creation Methods Benchmark

This is a sample program to test speed and memory usage for different approaches for n-gram creation in Python

Usage

Create virtual environment and install requirements

make venv

Running the tests

make run

This command will do the following:

1. Create a file of random length lorem ipsum sentences in one sentence per line (SPL) format
2. Tests character and word 3-grams and 5-grams and print the output

See results in results.txt

Acknowledgment

This program could not be created without the joint efforts of Iván Mittelholcz, Bálint Sass, Bence Nyéki and Balázs Indig

License

This code is licensed under the MIT license