Abstract LLM provider #88
Description
Namely, want to support OpenAI and Anthropic.
In addition to those below, the other main challenges are (1) abstracting the vector database and (2) managing model context.
| Dimension | OpenAI Chat Completions | OpenAI Responses | Anthropic Messages |
|---|---|---|---|
| State Management | Stateless (you manage messages array) | Stateful (server-side via previous_response_id) | Stateless (you manage messages array) |
| Execution | Sync, Async, Streaming | Sync, Async, Streaming | Sync, Async, Streaming |
| Structured Output | Yes (response_format, function calling) | Yes (same mechanisms) | Yes (tool use with JSON schema) |
| Tool Calls | Yes (functions/tools) | Yes (built-in tools like web search, file search, computer use) | Yes (tool use) |
| Extended Reasoning | No | Yes (reasoning parameter) | Yes (extended_thinking parameter) |
| Built-in Tools | No | Yes (web search, file search, code interpreter, computer use) | No (tools must be defined) |
| Primary Use Case | General chat, most integrations | Agentic workflows, multi-turn with state | General chat, agentic workflows |