Local LLM support

[razvanab]

Can you please add local LLM support, please?
Ollama support will be nice too.
Thank you.

[si0103]

Adding to the above: it would be great to allow for referencing folders that contain models already.
Thanks a bunch!

[KnutJaegersberg]

yeah it's not really open source if it doesn't allow local inference with some of the recent open models

[dareima]

yeah it's not really open source if it doesn't allow local inference with some of the recent open models

I think it would be unfair to say it like that. It is open source. You could add support for local models yourself.
It would certainly benefit this great project to have it by design though, which might very well be the case soon.

[dareima]

Can you please add local LLM support, please? Ollama support will be nice too. Thank you.

@razvanab did you check this?

[lombarthus]

Can you please add local LLM support, please? Ollama support will be nice too. Thank you.

@razvanab did you check this?

Yes, but then it still seems to require an OpenAI API Key.

[rushizirpe]

Try:
export OPENAI_API_KEY="DUMMY_KEY"
And then:
python app.py

[razvanab]

Yes, I did check that, and I didn't have much success.

[rushizirpe]

Here's how I am able to resolve:

pip install llama-cpp-python[server]
wget https://huggingface.co/TheBloke/zephyr-7B-beta-GGUF/resolve/main/zephyr-7b-beta.Q6_K.gguf
python3 -m llama_cpp.server --model ./zephyr-7b-beta.Q6_K.gguf --host 127.0.0.1 --port 8888
export OPENAI_API_KEY=DUMMY
python app.py

Custom Base API: http://localhost:8888/v1
Text Generation Model: openai/custom_model

[jsCarvalhoNeto]

My dear, could you detail these procedures for laymen?

[mfoldes]

My dear, could you detail these procedures for laymen?

If you're going to run a local model, you'll have to download it and run it locally. This person is giving you commands to run to do that.

Here are his instructions in a shell script

#!/bin/bash

# Install the required Python package
pip install llama-cpp-python[server]

# Download the model file
wget https://huggingface.co/TheBloke/zephyr-7B-beta-GGUF/resolve/main/zephyr-7b-beta.Q6_K.gguf

# Start the server with the specified model
python3 -m llama_cpp.server --model ./zephyr-7b-beta.Q6_K.gguf --host 127.0.0.1 --port 8888 &

# Export the API key
export OPENAI_API_KEY=DUMMY

# Run the application
python app.py

[jsCarvalhoNeto]

In the third step when running "python3 -m llama_cpp.server --model ./zephyr-7b-beta.Q6_K.gguf --host 127.0.0.1 --port 8888" I get the following message: C:\Users\santo\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\python.exe: Error while finding module specification for 'llama_cpp.server' (ModuleNotFoundError: No module named 'llama_cpp')" Could someone help?

[moebiussurfing]

try python -m llama_cpp.server --model ./zephyr-7b-beta.Q6_K.gguf --host 127.0.0.1 --port 8888 --n_ctx 16384
notice the last argument if you need more context size.

[moebiussurfing]

I followed the steps but I got an API KEY error (even using the DUMMY alias):
warning Error code: 401 - {'error': {'message': 'Incorrect API key provided: DUMMY. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}

I think this happens when submitting the generated podcast text to the TTS generator.