-
Notifications
You must be signed in to change notification settings - Fork 10
Open
Description
PYTHONPATH="$PYTHONPATH" TORCH_CUDA_ARCH_LIST="8.9"
ktransformers --model_path deepseek-ai/DeepSeek-V3-0324 \ 这个应该填什么
--gguf_path ./R1Q4GGUF \ 这个是gguf文件的路径,应该用不到了吧?
--port 10002
--optimize_config_path ./ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml \
--use_cuda_graph
--temperature 0.5
--cpu_infer 120
--port 10002
--max_new_tokens 8192
现在R1Q4KM正常运行, 这个V3-0324 应该怎么运行?
Or you want to start server with transformers, the model_path should include safetensors 这是KT官网的介绍
ktransformers --type transformers --model_path /mnt/data/model/Qwen2-0.5B-Instruct --port 10002 --web True 参数就不详细填了, 就指定model_path 就可以运行吗?
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels