Other tool-calling (agentic) frameworks

[uogbuji]

Just a tickler of stuff to keep apprised of, including other patterns & conventions to possibly implement.

• "I built a PR Agent with all the Agentic Frameworks - Here's my experience" (compares CrewAI, Autogen, Langchain, LlamaIndex, OpenAI)

[uogbuji]

Useful antecedents in Choosing the Best Structured Output Parser Approach | 3 Ways To Generate Structured Output

OpenAI JSON mode:

from openai import OpenAI
import os

# Load OpenAI Client (Create and view your API key - https://platform.openai.com/account/api-keys)
client = OpenAI(api_key=os.environ['OPENAI_API_KEY'])
# Define Prompt
prompt = """Extract JSON about PII: { "name": "name of person", "ssn": "Social security number of the person", "email": "Email address of the person", "age": Age of the person} in """
# Sample Input Text 
input_text = """Tony Stark
                Age 32
                
                Education
                - University of Southern California, Los Angeles
                  - M.S. in Computer Science (Specialization in Data Science)
                  - 2020
          
                IronMan@Avengers.ai"""
# Generate JSON Output
json_response = client.chat.completions.create(
    messages=[
        {
            "role": "user",
            "content": "You are a helpful assistant designed to output JSON.",
        },
        {
            "role": "user",
            "content": prompt + input_text,
        }
    ],
    model="gpt-3.5-turbo",
    response_format={ "type": "json_object" }
)
# Output
print(json_response.choices[0].message.content)
"""
{
  "name": "Tony Stark",
  "ssn": "",
  "email": "IronMan@Avengers.ai",
  "age": 32
}
"""

OpenAI function-calling

from openai import OpenAI
import os

# Load OpenAI Client (Create and view your API key - https://platform.openai.com/account/api-keys)
client = OpenAI(api_key=os.environ['OPENAI_API_KEY'])
# Define Custom Function
pii_data_parsing_custom_functions = [ { 'name': 'extract_pii_data', 
    'description': 'Extract personal identifiable information from provided data', 
    'parameters': { 'type': 'object', 
                    'properties': 
                      { 'name': { 'type': 'string', 'description': 'Complete name of the person' }, 
                        'ssn': { 'type': 'string', 'description': 'Social security number of the person' }, 
                        'email': { 'type': 'string', 'description': 'Email address of the person' }, 
                        'age': { 'type': 'integer', 'description': 'Age of the person' } 
                       } } }]
# Sample Input Text 
input_text = """Tony Stark
                Age 32
                
                Education
                - University of Southern California, Los Angeles
                  - M.S. in Computer Science (Specialization in Data Science)
                  - 2020
          
                IronMan@Avengers.ai"""
# Generate JSON Output
json_response = client.chat.completions.create(
    model = 'gpt-3.5-turbo',
    messages = [{'role': 'user', 'content': input_text}],
    functions = pii_data_parsing_custom_functions,
    function_call = 'auto'
)
# Output
print(json_response.choices[0].message.function_call.arguments)
"""
{
  "name": "Tony Stark",
  "ssn": "",
  "email": "IronMan@Avengers.ai",
  "age": 32
}
"""

Using Outlines with Phi-3:

from enum import Enum
from pydantic import BaseModel, constr, EmailStr
import outlines
import torch

# Define PII model using Pydantic
class PII(BaseModel):
    name: constr(max_length=100)
    ssn: constr(pattern=r'^\d{3}-\d{2}-\d{4}$')
    email: EmailStr
    age: int

# Load the model from outlines
model = outlines.models.transformers("microsoft/Phi-3-mini-4k-instruct")
# Construct the structured sequence generator
generator = outlines.generate.json(model, PII)
# Sample Input Text 
input_text = """Tony Stark
                Age 32
                
                Education
                - University of Southern California, Los Angeles
                  - M.S. in Computer Science (Specialization in Data Science)
                  - 2020
          
                IronMan@Avengers.ai"""
# Generate JSON Output
json_response = generator(input_text)
# Output 
print(json_response
"""
{
  "name": "Tony Stark",
  "ssn": "",
  "email": "IronMan@Avengers.ai",
  "age": 32
}
"""

[uogbuji]

Had a quick look at llama-agentic-system. It's more a self-contained environment incorporating Llama 3.1, which automatically calls certain tools (and apparently will allow you to define additional tools) and runs in a sandboxed system.

BTW, this subtle tidbit tucked into their README hints at what I think is a big shift in thinking at least in one major foundation model's training: Additionally, we would like to shift safety evaluation from the model level to the overall system level. This allows the underlying model to remain broadly steerable and adaptable to use cases which need varying levels of safety protection.

I suspect that for targeted tools, this project will offer better tool-calling performance than Toolio, but I also think Toolio's more modular approach makes more sense, and I'm willing to bet that models will keep getting smart enough to reliably use it for more and more cases.

Also: The inference server currently only supports CUDA. It will not work on Apple Silicon machines.

[uogbuji]

fastmlx is a new project I hadn't heard of before, from the prolific Prince Canuma. It's an OpenAI-style HTTP server for MLX, and he's prepping a PR for tool-calling.

Th tool-calling approach is the llama-agentic-system prompt and XML-ish Llama 3.1 format. (Full details of that from Meta | A nice overview from TogetherAI).

[uogbuji]

Anthropic Claude Tool Use is somewhat OpenAI-like, except that, for example, tool results are returned to the LLM server with a user message containing content: {"type": "tool_result"…. It supports image tool response types.

[uogbuji]

Cohere's Multi-step Tool Use (Agents)

Note: Cohere open-sourced a terrarium—"simple Python sandbox for helpful LLM data agents", i.e. its environment for safely executing arbitrary Python tools.