Generative AI for classification

[frayle-ons]

The package is currently capable of performing semantic search for classification, taking in a user query/input text and returning 1 or more candidate classifications ranked. Generative AI, particularly the form of Retrieval Augmented Generation (RAG) can be used to peform the final classification step given a candidate list.

The purpose of this ticket is to:

• implement Agents as a feature of the package that perform RAG on the candidate list and user input to derive a final classification
• Provide a framework similar to how the Vectorisers module works - i.e. can instantiate different kinds of AI models which all plug into the VectorStore ClassifAI framework, users can write their own custom model inserting from a base class, failure checks are controlled by the VectorStore class not the plugin class
• Performs efficiently and considers the nuances of generative AI such as how to guarantee

Key point: a generative AI component of the ClassifAI package should enhance the existing statistical classification capabilities of the package and not deviate towards other aspects of generative AI such as raw text generation for question answering or summarisation like tasks.