This framework allows to easily ingest a different kind of data and experiment with different retrieval system, evaluate each of them and gives accuracy measurements to say which one is the best.
The retrieval system takes a plain text user query and search for relevant data with a combination of strict filtering and similarity search via a vector database filled with embeddings of the data.
For more in-depth explanation, see Modular Retrieval System
uv venv
source .venv/bin/activate
uv sync
docker compose up -dVectors are stored in Qdrant vector database.
Processed transformations and documents transformation state are stored in an elasticsearch database.
The engine is responsible for:
- Indexing source documents
- Retrieving documents similar to a query
- Evaluating its search accuracy
To create an engine, it is needed to describe how to fetch data from source, how to transform documents and how to search for them.
An engine config contains:
- List of embedding models to use
- Chunking implementations
- Derivation implementations
- Transformation combinaisons (chunking + derivation + embedding model)
- Search strategies (transformations/parrallel/sequential/reranking)
- Source repository implementation
- Ingester configuration
- Evaluation dataset
There are different pre-made engines for specific data types.
- markdown: Indexes all markdown files in a directory
Default engine is markdown.
Change DEFAULT_ENGINE in ./src/engines/engines.py
- Prepare storage
make prepare
- Ingest data (all)
make ingest
- Evaluate
make evaluate
- Migrate vector DB
make migrate-vector
Useful when new vectors are added to the vector DB
- MCP OpenAPI server
make mcp
This will start a MCP server at http://localhost:8000
WebUI for interacting with data
make streamlit
Access studio at http://127.0.0.1:6334
Download a markdown documentation from a git repository
./src/engines/markdown/download_git_folder.sh https://github.com/wevm/wagmi site/react ./data/markdownIngest the documentation
make ingest