Offline SLM RAG Application

This application provides a chat interface to interact with your documents using a locally running Small Language Model (SLM). It works entirely offline.

Features

• Offline SLM: Uses Phi-3-mini-4k-instruct via llama-cpp-python.
• RAG (Retrieval-Augmented Generation): Uses ChromaDB and SentenceTransformers to index and retrieve relevant document chunks.
• FastAPI Backend: Handles document ingestion and chat inference.
• Streamlit Frontend: Provides a user-friendly chat interface.

Prerequisites

• Python 3.10+
• Basic build tools (for llama-cpp-python compilation)

Setup

1. Create a Virtual Environment (Recommended):

   python3 -m venv venv
   source venv/bin/activate

2. Install Dependencies:

   pip install -r requirements.txt

3. Download Model:

   python download_model.py

Running the App

You can use the helper script to start both backend and frontend:

./start.sh

Or run them manually:

Backend:

uvicorn backend.main:app --reload --port 8000

Frontend:

streamlit run frontend/app.py --server.port 8501

Usage

1. Open the Streamlit app (usually http://localhost:8501).
2. Upload a PDF or Text file in the sidebar.
3. Click "Ingest Document".
4. Ask questions in the chat interface.

Architecture

See archdocs/architecture.md for C4 model diagrams.

Configuration & Model Swapping

The application uses backend/config.py to manage settings. You can easily swap the SLM model by updating this file.

1. Edit backend/config.py:

   class Config:
       MODEL_REPO = "microsoft/Phi-3-mini-4k-instruct-gguf"
       MODEL_FILENAME = "Phi-3-mini-4k-instruct-q4.gguf"
       PROMPT_TEMPLATE = "phi3"  # Options: phi3, chatml, llama2

2. Download New Model:

   python download_model.py

3. Restart Application:

   ./stop.sh
   ./start.sh