Skip to content

feat: Add SAM2 (Segment Anything Model) support #3

@ebowwa

Description

@ebowwa

Feature Request

Add support for SAM2 (Segment Anything Model 2) segmentation masks.

Background

CoreMLPlayer currently supports:

  • Object Detection (bounding boxes)
  • Image Classification (labels)

SAM2 outputs segmentation masks (pixel-level), which would be a new visualization type for the app.

Requirements

  1. Handle mask outputs - Parse SAM2's multi-array segmentation data
  2. Render masks - Create SwiftUI overlay views for semi-transparent masks
  3. Handle SAM2 inputs - Support point/box prompts for interactive mode
  4. Mask visualization settings - Color, opacity, blend mode controls

Existing Infrastructure to Leverage

  • ✅ Video/image pipeline
  • ✅ CoreML loading (.mlpackage support already added)
  • ✅ Coordinate transformation
  • ✅ Overlay rendering system (DetectionRect)
  • ✅ Model configuration UI

Technical Notes

  • SAM2 outputs multi-array masks
  • Need new SwiftUI view for mask overlay (similar to DetectionRect)
  • Vision framework doesn't have built-in SAM2 support, so direct CoreML usage needed
  • Mask data likely needs conversion from multi-array to visual format

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions