[Shopping List] Task 4 – Speech Loop (Whisper + GPT-4 Orchestration)

[akisma]

Epic

• Parent: EPIC Task 1: AI-powered Mobile Shopping List Experience for Chefs #57

Goal

Wire the end-to-end speech interaction so chefs can control shopping lists hands-free, translating voice input into backend actions through AI intent resolution.

Deliverables

• Expo-side voice capture module leveraging microphone permissions and streaming
• Backend relay endpoint calling OpenAI Whisper for speech-to-text with error retries
• Conversational intent orchestrator using GPT-4 to map utterances to CRUD/reminder actions with context awareness
• Guardrails/rate limiting to prevent abuse and handle noisy kitchen environments
• Automated tests for intent mapping, fallback dialogs, and failure states

Acceptance Criteria

• Voice commands (create list, add/remove items, send list) succeed end-to-end with latency targets documented
• Conversational context retained within active session (e.g., clarifying “add two cases of tomatoes”)
• Graceful degradation to manual UI when speech fails or is unavailable
• Telemetry logged for speech success/failure without storing raw audio beyond processing needs

Notes

• Coordinate with Task 5 for spoken confirmation contract
• Security review required for handling OpenAI credentials and audio payloads
• Consider offline/poor network guidance in UX copy

[akisma]

Task 4 Implementation Plan - Voice Integration

Prepared: November 25, 2025
Estimated Duration: 13 days (~2 weeks)
Prerequisites: ✅ Task 3 Complete (UI stubs ready, 125 tests passing)

---

🎯 Architecture Decision: Backend-Heavy Voice Processing

After completing Task 3, we've decided on a backend-centric architecture for voice processing:

📱 Mobile App
  ├─ Record Audio (expo-av)
  ├─ Detect "Hey Shoppy" (client-side pattern matching)
  └─ Upload to Backend
      ↓
🖥️ Backend API
  ├─ Receive Audio Blob
  ├─ OpenAI Whisper (Speech-to-Text)
  ├─ GPT-4 (Intent Parsing with context)
  ├─ Execute CRUD Action (existing services)
  └─ Return Result + TTS Text
      ↓
📱 Mobile App
  ├─ Update UI (React Query cache)
  └─ Speak Confirmation (expo-speech)

Why Backend-Heavy:

• ✅ OpenAI API keys secure (never exposed to client)
• ✅ Centralized rate limiting and abuse prevention
• ✅ Easier telemetry and error logging
• ✅ Leverages existing robust backend from Task 2
• ✅ Simpler to test and maintain
• ✅ Better for future multi-user scenarios

---

📋 Implementation Phases

Phase 1: Backend Voice Infrastructure (3 days)

New Backend Endpoints:

POST /api/v1/voice/command
  - Body: { audioBlob: base64, sessionId?: string }
  - Returns: { success: boolean, action: string, data: any, ttsText: string }

POST /api/v1/voice/session
  - Creates new voice session with context
  - Returns: { sessionId: string, expiresAt: timestamp }

GET /api/v1/voice/session/:id
  - Returns current session context (last 5 commands)

DELETE /api/v1/voice/session/:id
  - Ends voice session, clears context

New Services:

• VoiceService - Orchestrates Whisper + GPT-4 + existing CRUD services
• IntentParser - Uses GPT-4 to map utterances to structured actions
• SessionManager - Maintains conversation context (5-minute timeout)
• AudioProcessor - Converts audio formats, validates

OpenAI Integration:

• Whisper API for speech-to-text
• GPT-4 for intent parsing with system prompt
• Error handling with 3 retries
• Rate limiting: 30 commands/minute/user

TDD Approach:

• Mock OpenAI API responses
• Test intent mapping for all command types
• Test session context retention
• Test error handling and fallbacks

---

Phase 2: Mobile Audio Capture (2 days)

Dependencies:

npx expo install expo-av
npx expo install expo-speech

Audio Recording:

• Request microphone permissions on first use
• Record in WAV/M4A format (max 30 seconds)
• Visual feedback during recording (VoiceStatusIndicator states)
• Convert to base64 for upload

Wake Word Detection:

• Client-side pattern matching for "Hey Shoppy"
• Simple regex/phonetic matching (no ML model needed)
• Configurable sensitivity in Settings
• Visual indicator when wake word detected

TDD Approach:

• Mock expo-av Audio.Recording
• Test permission request flow
• Test recording start/stop/cancel
• Test wake word detection

---

Phase 3: Command Integration (3 days)

Wire Up Voice Components:

• Replace stub alerts with real voice flow
• Connect VoiceStatusIndicator state machine:
  • coming-soon → ready (when settings enabled)
  • ready → listening (during recording)
  • listening → processing (uploading + waiting)
  • processing → ready (on completion)
• Show VoiceActivationBanner when enabled
• Enable Settings toggles

Backend Communication:

• Upload audio to /api/v1/voice/command
• Handle response and parse action
• Update React Query cache with result
• Show errors gracefully

TDD Approach:

• Mock backend voice endpoint
• Test all state transitions
• Test network error handling
• Test response parsing

---

Phase 4: Intent → Action Mapping (2 days)

Supported Commands:

1. Create List:

   • "Hey Shoppy, create list Produce"
   • "Hey Shoppy, new list for dairy items"
   • Maps to: POST /api/v1/shopping-lists

2. Add Item:

   • "Hey Shoppy, add tomatoes"
   • "Hey Shoppy, add three cases of onions"
   • "Hey Shoppy, put olive oil on the list"
   • Maps to: POST /api/v1/shopping-lists/:id/items

3. Remove Item:

   • "Hey Shoppy, remove tomatoes"
   • "Hey Shoppy, delete the onions"
   • Maps to: DELETE /api/v1/shopping-lists/:id/items/:itemId

4. Send List:

   • "Hey Shoppy, send this list"
   • "Hey Shoppy, send the list to the manager"
   • Maps to: POST /api/v1/shopping-lists/:id/send

5. List Query:

   • "Hey Shoppy, show my lists"
   • "Hey Shoppy, what's on the produce list?"
   • Maps to: GET /api/v1/shopping-lists

Context Handling:

• Session maintains "current active list"
• "Add tomatoes" assumes current list
• "Which list?" clarification if ambiguous
• GPT-4 system prompt with context window

TDD Approach:

• Test all command variations
• Test ambiguity handling
• Test context retention across commands
• Test error responses

---

Phase 5: TTS Feedback (1 day)

Confirmation Phrases:

• Create: "Created list 'Produce' with zero items"
• Add: "Added tomatoes to the list"
• Add with quantity: "Added three cases of onions"
• Remove: "Removed tomatoes from the list"
• Send: "Sent the list to the manager"
• Error: "Sorry, I didn't understand that. Please try again."

Implementation:

• Use expo-speech for text-to-speech
• Speak confirmation immediately after action
• Queue TTS if multiple commands rapid-fire
• Settings toggle for TTS on/off

TDD Approach:

• Mock expo-speech
• Verify correct text passed to TTS
• Test TTS queue logic
• Test settings toggle

---

Phase 6: Settings & Polish (2 days)

Settings Screen Updates:

• Enable "Voice Activation" toggle (functional)
• Enable "Voice Button on Lists" toggle (functional)
• Add "Voice Feedback" toggle (TTS on/off)
• Add "Wake Word Sensitivity" slider
• Remove "Coming in Task 4" messages

Error Handling:

• Offline mode: "Voice commands require internet connection"
• Microphone denied: "Please enable microphone in Settings"
• Backend error: "Voice service unavailable, please try again"
• Unclear command: "I didn't understand. Can you repeat that?"

Performance Optimization:

• Audio compression before upload
• Session caching
• Optimistic UI updates
• Background audio processing

Final Testing:

• Full E2E with real voice
• Kitchen noise testing
• Network failure scenarios
• Different accents/speech patterns
• Rapid-fire commands

---

🧪 Testing Strategy

Backend Tests (Target: 50+ new tests):

• VoiceService unit tests (mock OpenAI)
• IntentParser tests (all command variations)
• SessionManager tests (context + timeout)
• Voice endpoint integration tests
• Rate limiting tests

Mobile Tests (Target: 40+ new tests):

• Audio recording component tests
• Wake word detection tests
• Voice state machine tests
• Backend integration tests (mock API)
• TTS confirmation tests
• Settings toggle tests

Total: ~90 new tests (target 215+ total)

---

📊 Success Criteria (from Issue #68)

• Voice commands (create/add/remove/send) succeed end-to-end
• Latency target: <3 seconds average
• Conversational context retained within session
• Graceful degradation to manual UI when speech fails
• Telemetry logged (success/failure/latency)
• Raw audio NOT stored beyond processing
• Security review: OpenAI credentials secure
• Rate limiting prevents abuse
• Noisy kitchen environment handling

---

🔐 Security Considerations

1. API Keys: Stored in backend .env only, never exposed to client
2. Rate Limiting: 30 commands/minute per user, 1000/day per account
3. Audio Storage: Deleted immediately after processing
4. Session Security: JWT-based session IDs, 5-minute timeout
5. Input Validation: Max 30-second audio, format validation
6. Error Messages: Generic errors to client (details logged server-side)

---

📈 Monitoring & Telemetry

Logged Events:

• Voice command initiated (with timestamp)
• Audio upload success/failure
• Whisper STT success/failure/latency
• GPT-4 intent success/failure/latency
• Action execution success/failure
• Total end-to-end latency
• User session duration

Metrics Dashboard:

• Command success rate (target: >85%)
• Average latency (target: <3s)
• Most common commands
• Error rates by type
• Cost per command (OpenAI API)

---

🚀 Rollout Plan

Week 1: Backend + Audio Capture (Days 1-5)

• Phase 1: Backend voice infrastructure
• Phase 2: Mobile audio recording
• Daily testing and iteration

Week 2: Integration + Polish (Days 6-13)

• Phase 3: Wire up commands
• Phase 4: Intent mapping
• Phase 5: TTS feedback
• Phase 6: Settings and final polish
• Full E2E testing

Deployment:

• Staging environment testing (Day 12)
• Security review (Day 13)
• Production deployment (Day 13 EOD)
• Monitor metrics for 48 hours

---

🎓 Learnings from Task 3 Applied

1. TDD RED-GREEN-REFACTOR-DOCUMENT - Same strict cycle
2. Component patterns - VoiceStatusIndicator states already perfect
3. React Query - Same mutation patterns for voice commands
4. Testing utilities - Reuse centralized test helpers
5. Kitchen UX - Voice feedback > visual in hands-free scenarios

---

Ready to begin! Starting with Phase 1: Backend Voice Infrastructure