feat: Real-time object detection with Meta Glasses (relay)

[ebowwa]

Feature: Real-time Object Detection with Meta Glasses

Integrate CoreMLPlayer with the Meta Glasses Relay for real-time CV on live video streams.

The Vision

Meta Glasses → iPhone → Mac (CoreMLPlayer) → Display with overlays

Use Cases

1. Hands-free AR - Wear glasses, see live detection on Mac display
2. Accessibility - Real-time scene description/object localization for blind/low-vision
3. Professional Tools - Technician/doctor inspector with live CV feedback
4. Research/Testing - Test models in real environments, collect training data

Technical Approach

1. Use MacReceiver to get frames from Meta glasses
2. Pass frames to CoreMLPlayer's detection pipeline
3. Render detection boxes/labels over live video
4. Add model selection (YOLO, SAM2, etc.)
5. Optional: recording, annotation, audio feedback

Existing Infrastructure

• ✅ Meta glasses relay (mac-relay-yolo branch)
• ✅ CoreMLPlayer detection pipeline
• ✅ Video rendering system
• ✅ Model loading/switching

Main Challenges

• Latency - Multi-hop adds delay (glasses→iPhone→Mac)
• Performance - Real-time detection needs to be fast
• UI - How to overlay detection results on live video feed

Related

• Meta glasses relay: https://github.com/ebowwa/meta-wearables-dat-ios/tree/wip/mac-relay-yolo
• SAM2 support discussion: feat: Add SAM2 (Segment Anything Model) support #3

[ebowwa]

UI Integration Design

Based on the actual codebase analysis, here's how the Meta Glasses relay would integrate:

1. Sidebar Entry (MainView.swift:24-28)

Add to the "Try the model on" section:

Section("Try the model on") {
    NavigationLink { VideoGalleryView() } label: { Label("Videos", systemImage: "video") }
    NavigationLink { ImageGalleryView() } label: { Label("Images", systemImage: "photo") }
    NavigationLink { VideoDetectionView(source: .relay) } label: { Label("Meta Glasses", systemImage: "eyeglasses") }
}

2. Extend VideoDetection.swift

Add a Source enum to the existing class:

class VideoDetection: ObservableObject {
    enum Source {
        case file(URL)
        case relay(host: String, port: Int)
    }
    
    var source: Source = .file(URL(fileURLWithPath: "")) {
        didSet {
            switch source {
            case .file(let url):
                videoURL = url  // existing AVPlayer path
            case .relay(let host, let port):
                connectToRelay(host: host, port: port)  // new network path
            }
        }
    }
}

Key Insight: No New View Needed

VideoDetectionView stays the same — it just consumes frameObjects and videoInfo regardless of where frames come from.

The DetectionView overlay doesn't care about the video source — it just needs:

• detectedObjects: [DetectedObject]
• videoSize: CGSize

Both file and relay sources publish these, so the overlay works the same way.