forked from npna/CoreMLPlayer
-
Notifications
You must be signed in to change notification settings - Fork 0
Open
Description
Feature Request
Add support for SAM2 (Segment Anything Model 2) segmentation masks.
Background
CoreMLPlayer currently supports:
- Object Detection (bounding boxes)
- Image Classification (labels)
SAM2 outputs segmentation masks (pixel-level), which would be a new visualization type for the app.
Requirements
- Handle mask outputs - Parse SAM2's multi-array segmentation data
- Render masks - Create SwiftUI overlay views for semi-transparent masks
- Handle SAM2 inputs - Support point/box prompts for interactive mode
- Mask visualization settings - Color, opacity, blend mode controls
Existing Infrastructure to Leverage
- ✅ Video/image pipeline
- ✅ CoreML loading (.mlpackage support already added)
- ✅ Coordinate transformation
- ✅ Overlay rendering system (DetectionRect)
- ✅ Model configuration UI
Technical Notes
- SAM2 outputs multi-array masks
- Need new SwiftUI view for mask overlay (similar to DetectionRect)
- Vision framework doesn't have built-in SAM2 support, so direct CoreML usage needed
- Mask data likely needs conversion from multi-array to visual format
Metadata
Metadata
Assignees
Labels
No labels