Skip to content

Add dedicated VLM model configuration and wire it through multi-tenant VLM creation #10

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
BukeLy merged 4 commits into from
Dec 15, 2025
Merged

Add dedicated VLM model configuration and wire it through multi-tenant VLM creation #10

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b7b1960
Initial plan
Copilot Dec 15, 2025
de4c36f
拆分VLM模型配置
Copilot Dec 15, 2025
e5548d2
完善VLM配置独立化
Copilot Dec 15, 2025
bb21225
优化VLM密钥回退一致性
Copilot Dec 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion env.example
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,11 @@ LLM_API_KEY="your_llm_api_key_here"
LLM_BASE_URL="https://ark.ap-southeast.bytepluses.com/api/v3"
# 使用的模型名称
LLM_MODEL=seed-1-6-250615
# 用于多模态图片理解的 VLM 模型（必填，独立于 LLM_MODEL）
VLM_MODEL=seed-1-6-250615
Comment on lines +18 to +19
Copy link

Copilot AI Dec 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The documentation says VLM_MODEL is required (必填), but in the code at src/config.py:24, vlm_model uses Field(...) which makes it required at the pydantic validation level. However, the comment also says it should be "independent from LLM_MODEL" (独立于 LLM_MODEL), but the default value for LLM_MODEL and the example value for VLM_MODEL are both the same: seed-1-6-250615. This creates confusion about whether they should actually be different models. Consider clarifying whether VLM_MODEL can use the same model as LLM_MODEL (for models that support both text and vision), or if they must be different models.

Suggested change
# 用于多模态图片理解的 VLM 模型（必填，独立于 LLM_MODEL）
VLM_MODEL=seed-1-6-250615
# 用于多模态图片理解的 VLM 模型（必填，独立于 LLM_MODEL，需支持图片输入）
VLM_MODEL=seed-vlm-1-6-250615 # 示例：与 LLM_MODEL 不同，需为支持视觉的模型

Copilot uses AI. Check for mistakes.
# 可选：VLM 使用独立的密钥/域名（未设置时复用 LLM 配置）
# VLM_API_KEY="your_vlm_api_key_here"
# VLM_BASE_URL="https://api.example.com/v1"
# LLM 供应商标识（ark/openai/claude）
LLM_PROVIDER=ark
# VLM 图片理解 API 超时时间（秒，默认 120 秒）
Comment on lines 15 to 25
Copy link
Owner

@BukeLy BukeLy Dec 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LLM和VLM难道不可以换供应商吗?难道一定要用LLM的Base URL吗?

Copy link
Contributor Author

Copilot AI Dec 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已支持 VLM 独立的 API Key/Base URL（env 示例已添加 VLM_API_KEY/VLM_BASE_URL），不再强制复用 LLM Base URL（e5548d2）。

Expand Down Expand Up @@ -318,4 +323,3 @@ TZ=Asia/Shanghai

# --- Python 配置 ---
PYTHONUNBUFFERED=1 # 禁用输出缓冲，实时查看日志

4 changes: 3 additions & 1 deletion src/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@
重构原因: 统一配置管理，从服务商导向改为功能导向命名
"""

import os
from typing import Optional
from pydantic import Field
from pydantic_settings import BaseSettings
Expand All @@ -22,6 +21,9 @@ class LLMConfig(BaseSettings):
api_key: str = Field(..., description="LLM API Key")
base_url: str = Field(..., description="LLM API Base URL")
model: str = Field(default="seed-1-6-250615", description="LLM Model Name")
vlm_model: str = Field(..., description="VLM Model Name", alias="VLM_MODEL")
vlm_api_key: Optional[str] = Field(default=None, description="VLM API Key", alias="VLM_API_KEY")
vlm_base_url: Optional[str] = Field(default=None, description="VLM API Base URL", alias="VLM_BASE_URL")
Comment on lines +24 to +26
Copy link

Copilot AI Dec 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The VLM fields use explicit aliases (VLM_MODEL, VLM_API_KEY, VLM_BASE_URL) that don't follow the LLM_ prefix defined in the Config class. For these aliases to work correctly, the LLMConfig.Config class needs populate_by_name = True. Without this setting, pydantic-settings will not be able to load these environment variables. Other config classes in this file that use aliases (like DeepSeekOCRConfig at line 158) already include this setting.

Copilot uses AI. Check for mistakes.
vlm_timeout: int = Field(default=120, description="VLM Image Understanding Timeout (seconds)")
timeout: int = Field(default=60, description="General LLM Timeout (seconds)")

Expand Down
23 changes: 13 additions & 10 deletions src/multi_tenant.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,9 +48,12 @@ def __init__(
self._creation_locks: defaultdict = defaultdict(asyncio.Lock)

# 共享配置（从集中配置管理读取）
self.ark_api_key = config.llm.api_key
self.ark_base_url = config.llm.base_url
self.ark_model = config.llm.model
self.llm_api_key = config.llm.api_key
self.llm_base_url = config.llm.base_url
self.llm_model = config.llm.model
self.vlm_model = config.llm.vlm_model
self.vlm_api_key = config.llm.vlm_api_key or config.llm.api_key
self.vlm_base_url = config.llm.vlm_base_url or config.llm.base_url

self.sf_api_key = config.embedding.api_key
self.sf_base_url = config.embedding.base_url
Expand Down Expand Up @@ -85,9 +88,9 @@ def _create_llm_func(self, llm_config: Dict):
import asyncio

# 从配置中提取参数（支持租户覆盖）
model = llm_config.get("model", self.ark_model)
api_key = llm_config.get("api_key", self.ark_api_key)
base_url = llm_config.get("base_url", self.ark_base_url)
model = llm_config.get("model", self.llm_model)
api_key = llm_config.get("api_key", self.llm_api_key)
base_url = llm_config.get("base_url", self.llm_base_url)

# 获取 RateLimiter 参数（租户可配置）
# 注意：这里的 max_async 是 RateLimiter 的并发控制，不是 LightRAG 的
Expand Down Expand Up @@ -278,9 +281,9 @@ def _create_vision_model_func(self, llm_config: Dict):
import aiohttp

# 从配置中提取参数（支持租户覆盖）
model = llm_config.get("model", self.ark_model)
api_key = llm_config.get("api_key", self.ark_api_key)
base_url = llm_config.get("base_url", self.ark_base_url)
model = llm_config.get("vlm_model", self.vlm_model)
api_key = llm_config.get("vlm_api_key") or llm_config.get("api_key") or self.vlm_api_key
base_url = llm_config.get("vlm_base_url") or llm_config.get("base_url") or self.vlm_base_url
vlm_timeout = llm_config.get("vlm_timeout", self.vlm_timeout)

# 获取速率限制器（VLM 使用 LLM 的限制）
Expand Down Expand Up @@ -552,4 +555,4 @@ async def get_tenant_lightrag(tenant_id: str) -> LightRAG:
LightRAG: 该租户的实例
"""
manager = get_multi_tenant_manager()
return await manager.get_instance(tenant_id)
return await manager.get_instance(tenant_id)
3 changes: 3 additions & 0 deletions src/tenant_config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -301,6 +301,9 @@ def _merge_llm_config(self, tenant_config: Optional[TenantConfigModel]) -> Dict[
"""
base = {
"model": config.llm.model,
"vlm_model": config.llm.vlm_model,
"vlm_api_key": config.llm.vlm_api_key,
"vlm_base_url": config.llm.vlm_base_url,
"api_key": config.llm.api_key,
"base_url": config.llm.base_url,
"timeout": config.llm.timeout,
Expand Down