Design and Implement a Dynamic Multi-LLM Model Selection Architecture

[baranylcn]

Implement a dynamic and extensible architecture that supports multiple Large Language Model (LLM) options. The system should allow switching between different LLM providers and model versions in a configurable manner, without requiring code changes. This structure should be designed to be scalable, maintainable, and easy to extend, enabling the addition of new models or providers in the future with minimal effort.

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

✅ Successfully generated implementation plan.

💡 Use @coderabbitai <feedback> to replan

Examples

• Example 1
• Example 2

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[coderabbitai]

Coding Plan

Approach

Design and implement a dynamic, configuration-driven architecture that supports multiple LLM providers (OpenAI, Anthropic, Google Gemini) with the ability to switch between them via environment variables. The system should be extensible for adding new providers and prepare the codebase for future async API usage while maintaining compatibility with the existing Streamlit application.

Observations

The application is a FastAPI-based backend with a Streamlit frontend for resume analysis. Currently, it uses Google Gemini exclusively with a hardcoded model selection and synchronous calls. Configuration is managed via Starlette's Config with flat, module-level variables accessed through direct imports. The API layer exists but has no LLM-related endpoints yet, though dependency injection patterns are established. Prompts are provider-agnostic string templates. The codebase follows a straightforward, explicit style without heavy abstractions or frameworks.

Assumptions

Assumption 1: Initial Provider Support

Options Considered:

• Gemini only (current state)
• Gemini + OpenAI (minimal extension)
• Gemini + OpenAI + Anthropic (demonstrates full extensibility)

Chosen Option: Gemini + OpenAI + Anthropic (demonstrates full extensibility)

Rationale: Implement all three providers (Gemini, OpenAI, Anthropic) to fully demonstrate the architecture's extensibility and cover the major LLM providers developers might want to use. This shows the pattern works across different API styles and authentication methods while not being so many as to create excessive implementation burden.

Assumption 2: Synchronous vs Asynchronous Support

Options Considered:

• Sync only (meets current needs but requires refactoring for API)
• Async only (forces changes to Streamlit app unnecessarily)
• Both sync and async interfaces (more code but accommodates both use cases)

Chosen Option: Both sync and async interfaces (more code but accommodates both use cases)

Rationale: Implement both synchronous and asynchronous versions of the service layer. This allows the existing Streamlit app to continue using sync calls while preparing the codebase for FastAPI endpoints. The shared interface ensures consistency, and the separation keeps concerns clear without forcing unnecessary async complexity into the Streamlit UI code.

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: Configuration Foundation

Extend the existing Starlette Config pattern to support provider and model selection through environment variables, enabling switching between LLM providers without code changes.

Task 1: Add LLM Configuration Variables

Add environment variable definitions for provider selection, model configuration, and multiple provider credentials.

• Add LLM_PROVIDER config variable (default: "gemini") to specify active provider
• Add LLM_MODEL config variable to allow model selection per provider
• Add API key variables for OpenAI (OPENAI_API_KEY) and Anthropic (ANTHROPIC_API_KEY) alongside existing GEMINI_API_KEY
• Update .env.template with new variables and examples for each provider
• Group LLM-related configuration variables together in levelup/config.py for clarity

Task 2: Define Provider Configuration Schema

Create a structured approach to provider-specific settings within the flat configuration pattern.

• Add provider-model mapping configuration to document default models per provider (e.g., "gemini" -> "gemini-2.0-flash-lite", "openai" -> "gpt-4", "anthropic" -> "claude-3-sonnet")
• Add optional temperature and max_tokens configuration variables with sensible defaults that can be overridden per deployment
• Document configuration precedence: environment variables override defaults

🤖 Prompt for AI agents

This phase establishes the configuration foundation for multi-provider LLM
support by extending the Starlette Config pattern.

In `levelup/config.py`:
- Add `LLM_PROVIDER` config variable with default "gemini"
- Add `LLM_MODEL` config variable for model selection
- Add `OPENAI_API_KEY` and `ANTHROPIC_API_KEY` config variables
- Add provider-model mapping dictionary for default models per provider
- Add `LLM_TEMPERATURE` and `LLM_MAX_TOKENS` config variables with defaults
- Group all LLM-related variables together

In `.env.template`:
- Add example entries for all new LLM configuration variables
- Include comments showing example values for each provider

Phase 2: LLM Service Abstraction Layer

Create provider-agnostic interfaces and implement adapters for each LLM provider, allowing the application to treat all providers uniformly through a factory pattern.

Task 1: Define Service Interfaces

Create Protocol or ABC base classes that define the contract all LLM providers must implement.

• Create levelup/services/ directory for service layer abstractions
• Define BaseLLMProvider protocol in levelup/services/llm/base.py with generate_content() method signature that accepts prompt and returns response text
• Define AsyncLLMProvider protocol with async version of the same interface
• Include error types and response structure definitions that normalize across providers

Task 2: Implement Provider Adapters

Create concrete implementations for each LLM provider that conform to the base interface.

• Create levelup/services/llm/providers/ directory for provider implementations
• Implement GeminiProvider in levelup/services/llm/providers/gemini.py wrapping existing Google GenAI SDK patterns
• Implement OpenAIProvider in levelup/services/llm/providers/openai.py using OpenAI SDK
• Implement AnthropicProvider in levelup/services/llm/providers/anthropic.py using Anthropic SDK
• Each adapter should handle provider-specific authentication, request formatting, and response parsing while exposing the common interface
• Include both sync and async versions in each provider file, sharing common logic where possible

Task 3: Create Provider Factory

Build a factory that reads configuration and instantiates the appropriate provider at runtime.

• Create levelup/services/llm/factory.py with get_llm_provider() function
• Factory reads LLM_PROVIDER config variable and returns corresponding provider instance
• Inject API keys and model configuration from config module into provider constructors
• Factory should validate that required credentials exist for the selected provider
• Provide both get_llm_provider() (sync) and get_async_llm_provider() (async) factory functions
• Include clear error messages when provider is unsupported or credentials are missing

Task 4: Add Provider Health Checks

Implement basic validation to ensure providers are properly configured.

• Add validate_configuration() method to base protocol
• Each provider adapter implements configuration validation (API key present, SDK available)
• Factory calls validation during provider instantiation
• Return actionable error messages for configuration issues

🤖 Prompt for AI agents

This phase creates the core LLM service abstraction layer with provider adapters
and factory pattern.

Create `levelup/services/llm/base.py`:
- Define `BaseLLMProvider` Protocol with `generate_content(prompt: str) -> str`
method
- Define `AsyncLLMProvider` Protocol with async version
- Add `validate_configuration() -> None` method to both protocols
- Define common error types for provider failures

Create provider adapters in `levelup/services/llm/providers/`:
- `gemini.py`: Implement `GeminiProvider` wrapping Google GenAI SDK with
sync/async support
- `openai.py`: Implement `OpenAIProvider` wrapping OpenAI SDK with sync/async
support
- `anthropic.py`: Implement `AnthropicProvider` wrapping Anthropic SDK with
sync/async support
- Each adapter handles authentication, request formatting, response parsing
- Include both sync and async versions in each file

Create `levelup/services/llm/factory.py`:
- Implement `get_llm_provider()` that reads `LLM_PROVIDER` config and returns
sync provider instance
- Implement `get_async_llm_provider()` for async provider instances
- Inject API keys and model config from config module
- Validate credentials exist for selected provider
- Raise clear errors for unsupported providers or missing credentials

Phase 3: Streamlit Integration

Update the existing Streamlit application to use the new LLM service layer, maintaining backward compatibility while decoupling LLM logic from UI concerns.

Task 1: Refactor Module-Level LLM Initialization

Replace hardcoded Gemini client with factory-based provider instantiation.

• Remove direct genai.configure() and genai.GenerativeModel() calls from levelup/app.py
• Replace module-level Model initialization with factory call to get_llm_provider()
• Move LLM initialization inside functions rather than module-level if possible, or lazy-initialize on first use

Task 2: Update Resume Analysis Function

Modify analyzecv_pdf_withllm() to use the provider interface instead of direct Gemini SDK calls.

• Update function to accept provider instance or retrieve it via factory
• Replace direct Model.generate_content() call with provider's generate_content() interface method
• Ensure error handling accommodates different provider error types
• Maintain existing JSON extraction and response processing logic
• Keep existing Streamlit UI integration (st.error() calls) unchanged

Task 3: Verify Prompt Compatibility

Ensure existing prompt templates work across all providers without modification.

• Test that prompts from levelup/prompts.py work with all three providers
• Adjust prompt formatting if any provider requires specific markers (system messages, role definitions)
• Document any provider-specific prompt considerations in code comments

🤖 Prompt for AI agents

This phase integrates the new LLM service layer into the existing Streamlit
application.

In `levelup/app.py`:
- Remove direct `genai.configure()` and `genai.GenerativeModel()` calls
- Replace module-level Model with factory-based provider instantiation via
`get_llm_provider()`
- Lazy-initialize provider on first use or move initialization inside functions

Update `analyzecv_pdf_withllm()` function:
- Accept provider instance or retrieve via factory
- Replace `Model.generate_content()` with provider's `generate_content()` method
- Update error handling to accommodate different provider error types
- Preserve existing JSON extraction and response processing logic
- Keep Streamlit UI integration unchanged

Verify prompt compatibility:
- Test prompts from `levelup/prompts.py` with all providers
- Add provider-specific formatting if needed
- Document any provider-specific considerations in comments

Phase 4: Async Support and API Readiness

Prepare the FastAPI layer for future LLM endpoints by adding async service instantiation patterns and documenting usage.

Task 1: Add LLM Service Dependency

Create FastAPI dependency injection patterns for the async LLM service.

• Add get_llm_service() dependency function in levelup/api/deps.py
• Function should return async provider instance via factory
• Follow existing get_db() pattern for consistency
• Include proper lifecycle management and error handling

Task 2: Create Example API Route Pattern

Provide a reference implementation showing how to use the async LLM service in API routes.

• Create levelup/api/routes/llm.py with a sample endpoint demonstrating async LLM usage
• Document the endpoint pattern in code comments for future developers
• Endpoint should show proper error handling, timeout patterns, and response formatting
• Include this route in the API router or document it as an example template

Task 3: Document Extension Patterns

Create documentation for adding new providers and using the service layer.

• Add CONTRIBUTING.md or extend existing documentation with section on adding new LLM providers
• Document the provider interface contract and factory registration process
• Include code examples showing how to implement a new provider adapter
• Document configuration variables and environment setup for each provider
• Add inline code comments in factory and base classes guiding future extensions

🤖 Prompt for AI agents

This phase prepares FastAPI for async LLM endpoints and documents extensibility
patterns.

In `levelup/api/deps.py`:
- Add `get_llm_service()` dependency function that returns async provider via
`get_async_llm_provider()`
- Follow existing `get_db()` pattern for consistency
- Include lifecycle management and error handling

Create `levelup/api/routes/llm.py`:
- Implement sample endpoint demonstrating async LLM service usage
- Show proper error handling, timeout patterns, and response formatting
- Add comprehensive code comments for future developers
- Include route in API router or mark as example template

Create documentation:
- Add `CONTRIBUTING.md` or extend existing docs with LLM provider section
- Document provider interface contract and factory registration
- Include code examples for implementing new provider adapters
- Document configuration variables and environment setup for each provider
- Add inline comments in factory and base classes for extension guidance

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

This phase establishes the configuration foundation for multi-provider LLM
support by extending the Starlette Config pattern.

In `levelup/config.py`:
- Add `LLM_PROVIDER` config variable with default "gemini"
- Add `LLM_MODEL` config variable for model selection
- Add `OPENAI_API_KEY` and `ANTHROPIC_API_KEY` config variables
- Add provider-model mapping dictionary for default models per provider
- Add `LLM_TEMPERATURE` and `LLM_MAX_TOKENS` config variables with defaults
- Group all LLM-related variables together

In `.env.template`:
- Add example entries for all new LLM configuration variables
- Include comments showing example values for each provider
===============================================================================

Task: 2

This phase creates the core LLM service abstraction layer with provider adapters
and factory pattern.

Create `levelup/services/llm/base.py`:
- Define `BaseLLMProvider` Protocol with `generate_content(prompt: str) -> str`
method
- Define `AsyncLLMProvider` Protocol with async version
- Add `validate_configuration() -> None` method to both protocols
- Define common error types for provider failures

Create provider adapters in `levelup/services/llm/providers/`:
- `gemini.py`: Implement `GeminiProvider` wrapping Google GenAI SDK with
sync/async support
- `openai.py`: Implement `OpenAIProvider` wrapping OpenAI SDK with sync/async
support
- `anthropic.py`: Implement `AnthropicProvider` wrapping Anthropic SDK with
sync/async support
- Each adapter handles authentication, request formatting, response parsing
- Include both sync and async versions in each file

Create `levelup/services/llm/factory.py`:
- Implement `get_llm_provider()` that reads `LLM_PROVIDER` config and returns
sync provider instance
- Implement `get_async_llm_provider()` for async provider instances
- Inject API keys and model config from config module
- Validate credentials exist for selected provider
- Raise clear errors for unsupported providers or missing credentials
===============================================================================

Task: 3

This phase integrates the new LLM service layer into the existing Streamlit
application.

In `levelup/app.py`:
- Remove direct `genai.configure()` and `genai.GenerativeModel()` calls
- Replace module-level Model with factory-based provider instantiation via
`get_llm_provider()`
- Lazy-initialize provider on first use or move initialization inside functions

Update `analyzecv_pdf_withllm()` function:
- Accept provider instance or retrieve via factory
- Replace `Model.generate_content()` with provider's `generate_content()` method
- Update error handling to accommodate different provider error types
- Preserve existing JSON extraction and response processing logic
- Keep Streamlit UI integration unchanged

Verify prompt compatibility:
- Test prompts from `levelup/prompts.py` with all providers
- Add provider-specific formatting if needed
- Document any provider-specific considerations in comments
===============================================================================

Task: 4

This phase prepares FastAPI for async LLM endpoints and documents extensibility
patterns.

In `levelup/api/deps.py`:
- Add `get_llm_service()` dependency function that returns async provider via
`get_async_llm_provider()`
- Follow existing `get_db()` pattern for consistency
- Include lifecycle management and error handling

Create `levelup/api/routes/llm.py`:
- Implement sample endpoint demonstrating async LLM service usage
- Show proper error handling, timeout patterns, and response formatting
- Add comprehensive code comments for future developers
- Include route in API router or mark as example template

Create documentation:
- Add `CONTRIBUTING.md` or extend existing docs with LLM provider section
- Document provider interface contract and factory registration
- Include code examples for implementing new provider adapters
- Document configuration variables and environment setup for each provider
- Add inline comments in factory and base classes for extension guidance

💡 Iterate on the plan with: @coderabbitai <feedback>

Example Feedback
- @coderabbitai You can skip phase 3. Add a simple unit test case for phase 2.
- @coderabbitai For assumption 1 go ahead with option 3 and replan.

---

💬 Have feedback or questions? Drop into our discord!