Implement simple logs processing loop that is suitable for processing large amounts of usage stats

[scheibo]

Produce a MVP multi-(threaded|process) solution which can process logs efficiently. Minimum requirements:

• bound on number of open file descriptions
• bound of memory usage
• save progress on a days worth of logs as compressed checkpoints

Other features which would be nice to have:

• logging to diagnose errors
• sending out an email on errors so that the processing can be set up with a nightly cron job without worry about missing information
  • alternatively, posting a webhook to a discord channel might be preferable?

Speed requirements are not super important, should be able to process a day's worth of logs onn Smogon faster than smogon/usage-stats:

• https://github.com/smogon/usage-stats/blob/main/RunMonthly.sh (https://github.com/smogon/usage-stats/blob/main/RunPartial.sh)
  • https://github.com/smogon/usage-stats/blob/main/MonotypeAnalysis.sh
  • https://github.com/smogon/usage-stats/blob/main/MonthlyAnalysis.sh (https://github.com/smogon/usage-stats/blob/main/PartialAnalysis.sh)

Stretch feature: compress files after each run so that at the end of the month archiving is just a matter of the final tar cvf $OUTPUT * step