Add support for Ollama, Palm, Claude-2, Cohere, Replicate Llama2 CodeLlama (100+LLMs) - using LiteLLM

[ishaan-jaff]

This PR adds support for the above mentioned LLMs using LiteLLM https://github.com/BerriAI/litellm/
LiteLLM is a lightweight package to simplify LLM API calls - use any llm as a drop in replacement for gpt-3.5-turbo.

Example

from litellm import completion

## set ENV variables
os.environ["OPENAI_API_KEY"] = "openai key"
os.environ["COHERE_API_KEY"] = "cohere key"

messages = [{ "content": "Hello, how are you?","role": "user"}]

# openai call
response = completion(model="gpt-3.5-turbo", messages=messages)

# cohere call
response = completion(model="command-nightly", messages)

# anthropic call
response = completion(model="claude-instant-1", messages=messages)

[cielonet]

@krrishdholakia No prob. I'm currently out of town and will be back on Monday. I'll repost the error msg I was getting. It looked to me like the msg "expected string or buffer" was a msg generated by litellm because a value (I think it was part of the logging key) in the api call was not correctly formatted. When I ran it with raiseExceptions=False the api calls never sent that particular field and the system started working again. I did use the logging http copy/paste so if you have access to the logs/feedback people submit you should see mine from Thursday when I was working on this (e.g. focus on looking for "expected string or buffer") Anyways like I said I will be back Monday and will provide more feedback. I suggest adding a timeout to your telemetry as well if internet is not avaiable because otherwise it freezes this system and it as a pain to figure out that the telemetry was causing everything to pause until it finds an internet connection. :-/ Thanks again.

[noahnoahk]

how about the PR?

[OhNotWilliam]

I've tried those changes locally and trying to run the code with azure openai service doesn't seem to work. I'll let you know if I get it to function.

[krrishdholakia]

@OhNotWilliam we don't log any of the responses - it's all client-side (even the web url you saw was just an encoded url string). If you have the traceback, please let me know - happy to help debug.

[krrishdholakia]

We've also had people running this via the local OpenAI-proxy - https://docs.litellm.ai/docs/proxy_server

[dnhkng]

I've tried those changes locally and trying to run the code with azure openai service doesn't seem to work. I'll let you know if I get it to function.

@OhNotWilliam: Check my PR #192, which gets Azure working

[ImBIOS]

Love it, with several little fix to go.

[sammcj]

Any movement on getting this PR merged?

[dsnid3r]

Where do we stand on this? What is still outstanding/how can I help?

[dsnid3r]

@ishaan-jaff

[nobodykr]

Hi, is this still open ? Very confused

[ChieF-TroN]

Any update on when this will be implemented?

[TGM]

Ollama annouced OpenAI compability making LiteLLM irrelevant
https://ollama.com/blog/openai-compatibility

[nobodykr]

@TGM thank you for the headsup.
I think this is great. thank you .

[hemangjoshi37a]

If anyone has any documentation on clearly how to implement this what is provided in the title of this issue/PR please provide with that . thanks a lot. Happy coding .

[BreeanaBellflower]

litellm is great. I'm looking forward to seeing this get introduced to ChatDev. Not sure if you're looking to add support directly right now, but if so you may want to either add or generalize entries in ModelTypes, num_max_token_map, etc.

[BreeanaBellflower]

When testing this for Claude, line 78 below ("if not isinstance(response, Dict)") fails because the response is an instance of ModelResponse. Similar thing happening in chat_agent.py line 192.

[mororo250]

If this feature is still desired, I'd be happy to help facilitate the merge

[lexasub]

all is good

[nobodykr]

all is good

what you mean ? merged or not ?

[AZhurkin]

So need update jiter-0.6.1 openai-1.52.1 in requirements.txt, without it the chatdev is not started.

Check with llmstudio, still trying to connect to OpenAI not to local server on LLMStudio.

I set OPENAI_API_KEY, OPENAI_API_BASE and MODEL.

What else needs to be done?

[nojux-official]

Updates?