LocalLM_RSC

This repository contains a Three-Tier LLM System designed to efficiently answer questions from PDF documents using local language models for edge devices like Raspberry PI. The system is optimized to run on a Raspberry Pi 4 with 8GB RAM and uses a tiered approach to balance accuracy and performance. This was done and built for a thesis purpose : Benchmarking and Deploying Local Language Models for Social Educational Robots using Edge Devices. this thesis and repository is part of a big project called RSC (Robot study companion), you can find all the necessary information on : rsc.ee

System Requirements

• Recommended: Raspberry Pi 4 with 8GB RAM (or any Linux/Windows/macOS system)
• Minimum: 6GB available RAM
• Storage: ~10GB free space (for models and data)
• Python: 3.8 or

Installation

clone the repository

git clone <repository-url>
cd LocalLM_RSC

Ollama is required to load the model, dowload ollama using :

Linux/macOS:

curl -fsSL https://ollama.com/install.sh | sh

Windows: Download and install from https://ollama.com/download

Start ollama

ollama serve

Keep this running in a separate terminal.

Create the data directories:

mkdir -p data/documents
mkdir -p data/vector_db

Python Packages

you need : ollama chromadb numpy pymupdf sentence-transformers

pip install ollama chromadb numpy pymupdf sentence-transformers

Download the language models using Ollama:

# Embedding model (required)
ollama pull nomic-embed-text

# Small model for Tier 2 (required)
ollama pull gemma3:1b-it-qat

# Large model for Tier 3 (required)
ollama pull granite4:tiny-h

Running the architecture

1. load the documents

Place PDF documents in the data/documents/ directory:

cp your-document.pdf data/documents/

2. Run the System

Start the main application:

python LocalLM.py

First Run Workflow

On the first run, the system will:

1. Initialize the vector database
2. Verify all models are available
3. Process all PDFs in data/documents/
4. Generate questions for the cache (if AUTO_GENERATE_QUESTIONS = True)
5. Start the interactive query interface

Interactive Commands

Once running, you can use these commands:

• Ask questions: Simply type your question
• stats - Show database statistics
• tier-stats - Show tier usage statistics
• generate - Generate more questions for cache
• help - Show help message
• quit or exit - Exit the program