Knowledge Base: Fix database layer blockers and enable RAG pipeline

[DeepRatAI]

Summary

The Knowledge Base / Database section is accessible in the UI but completely non-functional. A full database infrastructure was designed and coded (PostgreSQL, Redis, Qdrant, ORM models, repositories, cache layer, RAG pipeline) — but none of it was ever connected to the running application. The /api/v1/knowledge/search endpoint fakes results by calling the generic LLM and returning a single hardcoded result with score 0.95.

Root Causes

1. BLOCKER — metadata reserved attribute name (src/medex/db/models.py L367)

The Message ORM model uses metadata as a column name, which is reserved by SQLAlchemy's DeclarativeBase. This causes an immediate InvalidRequestError on import, making the entire src/medex/db/ package unimportable:

sqlalchemy.exc.InvalidRequestError: Attribute name 'metadata' is reserved when using the Declarative API.

2. BLOCKER — Alembic migration never run

The migration file exists at alembic/versions/001_initial_schema.py but was never applied. The PostgreSQL database only has the app_config table (from init-db.sql). The ORM tables (users, conversations, messages, patient_contexts, tool_executions, kb_index) do not exist.

3. BLOCKER — No DB/Redis/Qdrant initialization in startup

MedeXApplication.startup() in src/medex/main.py only initializes LLM, Tools, and Agent services. No database connection, no Redis connection, no Qdrant client is created. The ServiceContainer never receives db, cache, or rag registrations.

4. BLOCKER — RAG explicitly disabled

The agent config at main.py L232 sets enable_rag=False. Even if everything else were fixed, RAG would still be off.

5. Fake endpoint — /api/v1/knowledge/search

# run_api.py L781-807 — What it actually does:
response = await medex_app.query(query=request.query, user_type="professional")
return {"results": [{"id": "kb_1", "title": "MedeX Knowledge Response",
                      "content": response.get("response", "..."),
                      "source": "MedeX RAG", "score": 0.95, "category": "medical"}]}

Always returns exactly 1 result with hardcoded score 0.95. No actual database query occurs.

6. Stub endpoint — /api/v1/search

MedeXApplication.search() always returns {"results": [], "total": 0}.

7. Zero Qdrant collections

The Qdrant container is running but has zero collections. No embeddings have ever been indexed. The collections referenced in .env.example (medex_conditions, medex_medications) were never created.

8. Legacy KB data never indexed

medical_knowledge_base.py (916 lines) and pharmaceutical_database.py (582 lines) at the project root contain rich in-memory medical data (conditions, medications, procedures, protocols, lab values, drug monographs, interactions). This data was never embedded or indexed into Qdrant.

Infrastructure Status

Component	Status
PostgreSQL 16 (medex-postgres)	✅ Container running, only app_config table
Redis 7 (medex-redis)	✅ Container running, not connected by app
Qdrant 1.12.1 (medex-qdrant)	⚠️ Running but unhealthy, 0 collections
src/medex/db/models.py	❌ Cannot import (reserved name bug)
src/medex/db/connection.py	❌ Cannot import (depends on models.py)
src/medex/db/repositories.py	❌ Cannot import (depends on models.py)
src/medex/db/cache.py	⚪ Code complete, never instantiated
src/medex/rag/	⚪ Code complete, never used (enable_rag=False)

Technical Tasks

A. Fix Database Layer

• Rename metadata → message_metadata in src/medex/db/models.py Message class
• Verify the entire src/medex/db/ package imports successfully
• Run alembic upgrade head to create ORM tables
• Verify tables exist: users, conversations, messages, patient_contexts, tool_executions, kb_index

B. Wire DB/Redis/Qdrant into Application Startup

• Add database connection initialization to MedeXApplication.startup() using src/medex/db/connection.py
• Add Redis connection initialization using src/medex/db/cache.py
• Add Qdrant client initialization
• Register db, cache, rag services in ServiceContainer
• Populate .env with database configuration from .env.example template

C. Knowledge Base Indexing Pipeline

• Create indexing script/command that reads legacy medical_knowledge_base.py and pharmaceutical_database.py
• Generate embeddings using all-MiniLM-L6-v2 (or configured model)
• Store embeddings in Qdrant collections (medex_conditions, medex_medications)
• Add make index-kb command to Makefile

D. Implement Real Search Endpoints

• Replace fake /api/v1/knowledge/search with actual Qdrant vector search via RAG service
• Implement MedeXApplication.search() method (currently stub returning empty)
• Fix /api/v1/knowledge/stats to return real counts from DB/Qdrant
• Set enable_rag=True in agent config

E. Tests

• Test database connection lifecycle (init, health check, shutdown)
• Test knowledge indexing pipeline
• Test vector search returns relevant results
• Test end-to-end: UI search → API → Qdrant → results

Files Affected

File	Change
src/medex/db/models.py L367	Rename metadata → message_metadata
src/medex/main.py	Add DB/Redis/Qdrant init to startup(), set enable_rag=True
run_api.py L768-816	Replace fake endpoints with real search
.env	Add database configuration vars
New: scripts/index_knowledge_base.py	Indexing pipeline
Makefile	Add index-kb target
tests/	New DB + RAG integration tests