Chain Chat - Multimodal AI Chat Application

A sophisticated multimodal chat application that supports text, images, audio, and web search across multiple AI providers including OpenAI, Anthropic, and xAI.

🚀 Features

Core Functionality

• Multi-Agent Chains: Create sequences of AI agents with different models and capabilities
• Real-time Streaming: Live streaming responses from AI models
• Session Management: Persistent chat sessions with full history
• Model Switching: Support for OpenAI, Anthropic Claude, and xAI Grok models

Multimodal Capabilities

• Image Upload & Analysis: Upload and analyze images with vision-capable models
• Voice Recording & Transcription: Record audio messages with automatic transcription
• Web Search Integration: Search the web and include results in AI context
• File Storage: Secure file storage with Convex backend

Advanced Features

• Smart Model Detection: Automatic capability detection based on selected model
• Professional UI: Modern, responsive interface with dark theme
• Error Handling: Comprehensive error handling and user feedback
• Type Safety: Full TypeScript implementation

🛠️ Tech Stack

• Frontend: Next.js 15, React 18, TypeScript, Tailwind CSS
• Backend: Convex (database & file storage)
• AI Providers: OpenAI, Anthropic, xAI
• Audio Processing: Web Audio API, OpenAI Whisper
• File Handling: React Dropzone, Image compression

📋 Prerequisites

Before setting up the application, you'll need:

1. Node.js (v18 or higher)
2. npm or yarn
3. Convex Account - Sign up at convex.dev
4. API Keys (at least one):
   • OpenAI API Key - Get from OpenAI
   • Anthropic API Key - Get from Anthropic
   • xAI API Key - Get from xAI

🚀 Setup Instructions

1. Clone and Install Dependencies

git clone <repository-url>
cd chain-chat
npm install

2. Set Up Convex

# Install Convex CLI globally
npm install -g convex

# Initialize Convex in your project
npx convex dev

# Follow the prompts to create a new Convex project
# This will create a .env.local file with CONVEX_DEPLOYMENT and NEXT_PUBLIC_CONVEX_URL

3. Configure Environment Variables

Create a .env.local file in the root directory:

# Convex (automatically generated by convex dev)
CONVEX_DEPLOYMENT=your-deployment-name
NEXT_PUBLIC_CONVEX_URL=https://your-deployment.convex.cloud

# AI Provider API Keys (add at least one)
OPENAI_API_KEY=sk-your-openai-key
ANTHROPIC_API_KEY=sk-ant-your-anthropic-key
XAI_API_KEY=xai-your-xai-key

# Optional: For production web search (currently using mock data)
# GOOGLE_SEARCH_API_KEY=your-google-api-key
# GOOGLE_SEARCH_ENGINE_ID=your-search-engine-id

4. Deploy Convex Schema

# Push the database schema to Convex
npx convex deploy

5. Start Development Server

npm run dev

The application will be available at http://localhost:3000

🧪 Testing the Integration

Visit http://localhost:3000/test to run the multimodal integration tests. This will verify:

• ✅ Image upload API functionality
• ✅ Audio transcription API (requires OpenAI API key)
• ✅ Web search API
• ✅ LLM integration setup
• ✅ Streaming agent configuration

📖 Usage Guide

Creating Agent Chains

1. Add Agents: Click the "+" button to add new agents to your chain
2. Select Models: Choose from available AI models based on your API keys
3. Add Multimodal Content:
   • 📎 Images: Click the paperclip icon to upload images
   • 🎤 Audio: Click the microphone to record voice messages
   • 🔍 Web Search: Click the search icon to search the web
4. Run Chain: Click "Run Chain" to execute all agents in sequence

Model Capabilities

Provider	Models	Text	Vision	Audio	Web Search
OpenAI	GPT-4o	✅	✅	✅	✅
OpenAI	GPT-4o Mini	✅	❌	✅	✅
OpenAI	GPT-4 Turbo	✅	❌	❌	✅
Anthropic	Claude 3.5 Sonnet	✅	✅	❌	✅
Anthropic	Claude 3.5 Haiku	✅	✅	❌	✅
xAI	Grok Beta	✅	✅	❌	✅

🔧 API Endpoints

File Upload

• POST /api/upload-file
• Handles image and audio file uploads
• Integrates with Convex file storage
• Supports validation and compression

Audio Transcription

• POST /api/transcribe-audio
• Uses OpenAI Whisper for speech-to-text
• Supports multiple audio formats
• Returns transcription with timing data

Web Search

• POST /api/web-search
• Currently returns mock search results
• Ready for integration with real search APIs
• Supports structured search result format

Streaming Agent

• POST /api/stream-agent
• Real-time streaming responses
• Supports all AI providers
• Handles multimodal context

Chain Execution

• POST /api/run-chain
• Executes multi-agent chains
• Processes multimodal attachments
• Stores results in database

🏗️ Architecture

Database Schema (Convex)

// Chat Sessions
chatSessions: {
  _id: Id<"chatSessions">
  _creationTime: number
  title: string
  userId?: string
}

// Agent Steps
agentSteps: {
  _id: Id<"agentSteps">
  sessionId: Id<"chatSessions">
  index: number
  model: string
  prompt: string
  response?: string
  isComplete: boolean
  isStreaming: boolean
  error?: string
  tokenUsage?: TokenUsage
}

// Attachments
attachments: {
  _id: Id<"attachments">
  sessionId: Id<"chatSessions">
  agentStepId?: Id<"agentSteps">
  type: "image" | "audio"
  fileName: string
  fileSize: number
  mimeType: string
  storageId: Id<"_storage">
  metadata: ImageMetadata | AudioMetadata
}

Component Structure

components/
├── agent-input.tsx          # Main agent configuration component
├── chat-area.tsx           # Chat display and history
├── ui/
│   └── AttachmentDisplay.tsx # Multimodal content display
└── modality/
    ├── ImageUpload.tsx     # Image upload and preview
    ├── VoiceRecorder.tsx   # Audio recording interface
    ├── WebSearch.tsx       # Web search interface
    └── ModalityIcons.tsx   # Modality selection icons

🔒 Security Considerations

• API keys are stored securely in environment variables
• File uploads are validated for type and size
• Convex handles secure file storage
• All API endpoints include error handling
• TypeScript ensures type safety throughout

🚀 Production Deployment

Vercel Deployment

1. Connect to Vercel:

   npm install -g vercel
   vercel

2. Configure Environment Variables in Vercel dashboard:

   • Add all environment variables from .env.local
   • Ensure Convex deployment is set to production

3. Deploy Convex to Production:

   npx convex deploy --prod

Environment Setup for Production

• Set up production Convex deployment
• Configure production API keys
• Set up real web search API (Google Custom Search, Bing, etc.)
• Configure proper CORS settings
• Set up monitoring and logging

🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests if applicable
5. Submit a pull request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🆘 Support

If you encounter any issues:

1. Check the troubleshooting section
2. Review the API documentation
3. Run the integration tests at /test
4. Check the browser console for errors
5. Verify your API keys and environment variables

🔧 Troubleshooting

Common Issues

"Cannot find module" errors:

rm -rf node_modules package-lock.json
npm install

Convex connection issues:

npx convex dev
# Follow the setup prompts again

API key errors:

• Verify API keys are correctly set in .env.local
• Check API key permissions and quotas
• Ensure keys are not expired

File upload failures:

• Check file size limits (10MB for images, 25MB for audio)
• Verify supported file formats
• Check Convex storage configuration

---

Built with ❤️ using Next.js, Convex, and modern AI APIs.

Video Implementation

This project uses HTML5 video elements for optimal performance and reliability:

• ✅ Simple & Fast: No complex video processing or API routes
• ✅ Vercel Ready: Videos deploy automatically with your app
• ✅ CDN Optimized: Served through Vercel's global CDN
• ✅ Git Friendly: Demo video included, large videos excluded
• ✅ Performance: Optimized bundle size (12.6 kB landing page)

Video Files

• Demo Video: public/videos/chained_demo.mp4 (included in Git)
• Future Videos: Add to .gitignore to exclude large files from Git

Quick Start

1. Clone the repository

   git clone <your-repo-url>
   cd chain-chat

2. Install dependencies

   npm install

3. Set up environment variables

   cp .env.example .env.local
   # Add your API keys and configuration

4. Run development server

   npm run dev

5. Build for production

   npm run build

Deployment

Vercel Deployment (Recommended)

1. Connect to Vercel

   • Push code to GitHub/GitLab/Bitbucket
   • Connect repository to Vercel dashboard
   • Set environment variables in Vercel dashboard

2. Automatic Deployment

   • Videos are automatically deployed with your app
   • Served through Vercel's global CDN
   • No additional configuration needed

See VERCEL_DEPLOYMENT.md for detailed deployment instructions.

Environment Variables

NEXT_PUBLIC_CONVEX_URL=your_convex_url
NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY=your_clerk_key
CLERK_SECRET_KEY=your_clerk_secret
# ... other environment variables

Tech Stack

• Framework: Next.js 15.3.3
• Database: Convex
• Authentication: Clerk
• Styling: Tailwind CSS
• Video: HTML5 video elements
• Deployment: Vercel

Project Structure

├── app/                    # Next.js app directory
├── components/            # React components
├── convex/               # Convex database functions
├── lib/                  # Utility functions
├── public/               # Static assets
│   └── videos/          # Video files
├── VERCEL_DEPLOYMENT.md  # Deployment guide
└── VIDEO_MANAGEMENT.md   # Video management guide

Performance

• Landing Page: 12.6 kB (optimized)
• Video Loading: CDN-optimized with caching
• Build Time: ~12 seconds
• Bundle Size: Optimized for production

Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Test thoroughly
5. Submit a pull request

Support

• Check VERCEL_DEPLOYMENT.md for deployment issues
• Check VIDEO_MANAGEMENT.md for video-related questions
• Open an issue for bugs or feature requests

License

[Your License Here]