feat: implement android face detection logic using ML Kit#313
feat: implement android face detection logic using ML Kit#313mosalah12272-cyber wants to merge 7 commits intoCap-go:mainfrom
Conversation
Added ML Kit Face Detection dependency to the project.
|
Warning Rate limit exceeded
⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. 📝 WalkthroughWalkthroughAdds ML Kit face-detection dependency and integrates FaceDetector initialization and event emission into the Android CameraPreview; concurrently, most prior camera control logic and many PluginMethod implementations were removed from CameraPreview. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant App as Client (JS)
participant Plugin as CameraPreview
participant Camera as Camera Hardware
participant ML as FaceDetector
participant JS as Event Listeners
App->>Plugin: start preview / subscribe to onFaceDetected
Plugin->>Camera: request frames
Camera-->>Plugin: deliver frame/image
Plugin->>ML: process frame (FaceDetector)
ML-->>Plugin: detection results (success/failure)
Plugin->>JS: emit onFaceDetected with faces (if any)
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
android/src/main/java/app/capgo/capacitor/camera/preview/CameraPreview.java (1)
116-169:⚠️ Potential issue | 🔴 CriticalCritical regression confirmed: File is incomplete and missing all
@PluginMethodimplementations.The entire file is only 169 lines and ends abruptly mid-declaration (
private CameraXViewwith no variable name or semicolon). More critically, the complete file contains zero@PluginMethodannotations and only 7 method definitions total:load(),processFaceDetection(), and lifecycle handlers (handleOnPause(),handleOnResume(),handleOnDestroy()).This is a non-functional state. A camera plugin with no public API methods cannot work. All camera control methods (start, stop, capture, zoom, focus, flash permissions, etc.) are missing. While the face detection code is present (lines 74–114), the removal of the entire public API surface is a critical regression that would break every existing consumer of this plugin.
The file must be restored to its complete state before merging.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@android/src/main/java/app/capgo/capacitor/camera/preview/CameraPreview.java` around lines 116 - 169, The CameraPreview.java file is truncated (ends mid-declaration at "private CameraXView") and is missing all public plugin API methods annotated with `@PluginMethod` (start, stop, capture, startPreview, stopPreview, setFlashMode, setZoom, focus, requestPermissions, etc.), making the plugin non-functional; restore the complete CameraPreview class by (1) completing the CameraXView field declaration and any other truncated members, (2) re-adding all `@PluginMethod-annotated` methods that constitute the public API (use the previous commit or authoritative source to recover exact method names and signatures such as start, stopCamera/startSession, capturePhoto/capture, setFlashMode, setZoom, focusAtPoint, getSupportedResolutions, requestCameraPermissions, etc.), and (3) ensure lifecycle handlers (handleOnPause/Resume/Destroy) still interact with CameraXView and lastSessionConfig as before and that imports and faceDetector usage remain intact; if unsure, restore the file from the last known-good commit rather than hand-reconstructing to guarantee parity with existing consumers.
🧹 Nitpick comments (2)
android/src/main/java/app/capgo/capacitor/camera/preview/CameraPreview.java (1)
92-113: Replace anonymous inner classes with lambdas (Java 21).
OnSuccessListenerandOnFailureListenerare single-abstract-method (SAM) interfaces; Java 21 lambda syntax is far cleaner here. As per coding guidelines, Java 21 is the required build target for Android.♻️ Proposed refactor
- faceDetector.process(image) - .addOnSuccessListener(new OnSuccessListener<List<Face>>() { - `@Override` - public void onSuccess(List<Face> faces) { - if (faces.size() > 0) { - JSObject ret = new JSObject(); - JSArray facesArray = new JSArray(); - for (Face face : faces) { - JSObject faceObj = new JSObject(); - faceObj.put("bounds", face.getBoundingBox().flattenToString()); - facesArray.put(faceObj); - } - ret.put("faces", facesArray); - notifyListeners("onFaceDetected", ret); - } - } - }) - .addOnFailureListener(new OnFailureListener() { - `@Override` - public void onFailure(`@NonNull` Exception e) { - Log.e(TAG, "Face detection failed", e); - } - }); + faceDetector.process(image) + .addOnSuccessListener(faces -> { + if (!faces.isEmpty()) { + JSObject ret = new JSObject(); + JSArray facesArray = new JSArray(); + for (Face face : faces) { + JSObject faceObj = new JSObject(); + // emit structured bounds (see separate comment) + faceObj.put("bounds", face.getBoundingBox().flattenToString()); + facesArray.put(faceObj); + } + ret.put("faces", facesArray); + notifyListeners("onFaceDetected", ret); + } + }) + .addOnFailureListener(e -> Log.e(TAG, "Face detection failed", e));🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@android/src/main/java/app/capgo/capacitor/camera/preview/CameraPreview.java` around lines 92 - 113, Replace the anonymous inner classes for the addOnSuccessListener and addOnFailureListener calls in CameraPreview.java with Java 21 lambdas: convert the new OnSuccessListener<List<Face>>() { public void onSuccess(List<Face> faces) { ... } } into a lambda (faces -> { ... }) and convert the new OnFailureListener() { public void onFailure(Exception e) { ... } } into a lambda (e -> Log.e(TAG, "Face detection failed", e)); update the addOnSuccessListener and addOnFailureListener invocations accordingly (keeping the existing body that builds the JSObject/JSArray and calls notifyListeners("onFaceDetected", ret)), and remove the now-unneeded anonymous class boilerplate/imports if any.android/build.gradle (1)
57-57: Consider the unbundled ML Kit model to avoid a ~1.4 MB APK size increase.The bundled model (
com.google.mlkit:face-detection:16.1.7) ships the model with the app; the alternativecom.google.android.gms:play-services-mlkit-face-detection:17.1.0downloads the model via Google Play Services, significantly reducing APK size. Evaluate which is appropriate for the target distribution model.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@android/build.gradle` at line 57, Replace the bundled ML Kit dependency to use the unbundled Play Services variant to avoid the ~1.4 MB APK increase: locate the dependency line implementation 'com.google.mlkit:face-detection:16.1.7' in android/build.gradle and swap it to implementation 'com.google.android.gms:play-services-mlkit-face-detection:17.1.0' (or conditionally choose based on distribution needs); ensure you update any proguard/r8 rules or README notes and test face detection behavior on devices with and without Google Play Services to confirm expected runtime model download.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@android/build.gradle`:
- Around line 56-57: Remove the non-English/unprofessional inline comment before
the ML Kit dependency and replace it with a plain English description;
specifically, locate the line containing the dependency string
'com.google.mlkit:face-detection:16.1.7' and change the preceding comment to a
neutral English comment such as "// ML Kit Face Detection" (or remove the
comment entirely) so the gradle file remains professional and readable.
In `@android/src/main/java/app/capgo/capacitor/camera/preview/CameraPreview.java`:
- Around line 77-87: The FaceDetectorOptions in load() mixes
PERFORMANCE_MODE_FAST with expensive LANDMARK_MODE_ALL and
CLASSIFICATION_MODE_ALL which conflicts with the intended fast path; either
switch the performance mode to PERFORMANCE_MODE_ACCURATE if you need
landmarks/classifications, or keep PERFORMANCE_MODE_FAST and change
setLandmarkMode to LANDMARK_MODE_NONE (or LANDMARK_MODE_ALL -> LANDMARK_MODE_ALL
only if needed) and setClassificationMode to CLASSIFICATION_MODE_NONE to reduce
overhead; update the FaceDetectorOptions builder (the setPerformanceMode,
setLandmarkMode, setClassificationMode calls) accordingly and ensure
faceDetector = FaceDetection.getClient(options) uses the chosen trimmed
configuration.
- Line 100: The code sets faceObj.put("bounds",
face.getBoundingBox().flattenToString()) which emits a non-JSON flat string;
instead extract numeric coordinates from face.getBoundingBox() and put them as
structured properties (e.g., left, top, right, bottom and/or width/height) on
faceObj so JavaScript consumers receive parseable numeric fields; update the
CameraPreview.java code around where faceObj is built to replace the
flattenToString call with individual numeric puts using Rect's getters
(left/top/right/bottom or compute width/height) and ensure types are numbers,
not a single string.
- Around line 90-114: processFaceDetection is never invoked because there is no
ImageAnalysis use case; update CameraXView.bindCameraUseCases() to create and
attach an ImageAnalysis instance to the camera pipeline, set an Analyzer that
converts each ImageProxy into an ML Kit InputImage and calls
processFaceDetection(InputImage), and ensure the Analyzer closes the ImageProxy
(imageProxy.close()) after processing; attach the ImageAnalysis use case when
binding to the lifecycle/camera (so Face detection runs in real time) and
reference the existing processFaceDetection method from the Analyzer.
---
Outside diff comments:
In `@android/src/main/java/app/capgo/capacitor/camera/preview/CameraPreview.java`:
- Around line 116-169: The CameraPreview.java file is truncated (ends
mid-declaration at "private CameraXView") and is missing all public plugin API
methods annotated with `@PluginMethod` (start, stop, capture, startPreview,
stopPreview, setFlashMode, setZoom, focus, requestPermissions, etc.), making the
plugin non-functional; restore the complete CameraPreview class by (1)
completing the CameraXView field declaration and any other truncated members,
(2) re-adding all `@PluginMethod-annotated` methods that constitute the public API
(use the previous commit or authoritative source to recover exact method names
and signatures such as start, stopCamera/startSession, capturePhoto/capture,
setFlashMode, setZoom, focusAtPoint, getSupportedResolutions,
requestCameraPermissions, etc.), and (3) ensure lifecycle handlers
(handleOnPause/Resume/Destroy) still interact with CameraXView and
lastSessionConfig as before and that imports and faceDetector usage remain
intact; if unsure, restore the file from the last known-good commit rather than
hand-reconstructing to guarantee parity with existing consumers.
---
Nitpick comments:
In `@android/build.gradle`:
- Line 57: Replace the bundled ML Kit dependency to use the unbundled Play
Services variant to avoid the ~1.4 MB APK increase: locate the dependency line
implementation 'com.google.mlkit:face-detection:16.1.7' in android/build.gradle
and swap it to implementation
'com.google.android.gms:play-services-mlkit-face-detection:17.1.0' (or
conditionally choose based on distribution needs); ensure you update any
proguard/r8 rules or README notes and test face detection behavior on devices
with and without Google Play Services to confirm expected runtime model
download.
In `@android/src/main/java/app/capgo/capacitor/camera/preview/CameraPreview.java`:
- Around line 92-113: Replace the anonymous inner classes for the
addOnSuccessListener and addOnFailureListener calls in CameraPreview.java with
Java 21 lambdas: convert the new OnSuccessListener<List<Face>>() { public void
onSuccess(List<Face> faces) { ... } } into a lambda (faces -> { ... }) and
convert the new OnFailureListener() { public void onFailure(Exception e) { ... }
} into a lambda (e -> Log.e(TAG, "Face detection failed", e)); update the
addOnSuccessListener and addOnFailureListener invocations accordingly (keeping
the existing body that builds the JSObject/JSArray and calls
notifyListeners("onFaceDetected", ret)), and remove the now-unneeded anonymous
class boilerplate/imports if any.
android/src/main/java/app/capgo/capacitor/camera/preview/CameraPreview.java
Show resolved
Hide resolved
| private void processFaceDetection(InputImage image) { | ||
| faceDetector.process(image) | ||
| .addOnSuccessListener(new OnSuccessListener<List<Face>>() { | ||
| @Override | ||
| public void onSuccess(List<Face> faces) { | ||
| if (faces.size() > 0) { | ||
| JSObject ret = new JSObject(); | ||
| JSArray facesArray = new JSArray(); | ||
| for (Face face : faces) { | ||
| JSObject faceObj = new JSObject(); | ||
| faceObj.put("bounds", face.getBoundingBox().flattenToString()); | ||
| facesArray.put(faceObj); | ||
| } | ||
| ret.put("faces", facesArray); | ||
| notifyListeners("onFaceDetected", ret); | ||
| } | ||
| } | ||
| }) | ||
| .addOnFailureListener(new OnFailureListener() { | ||
| @Override | ||
| public void onFailure(@NonNull Exception e) { | ||
| Log.e(TAG, "Face detection failed", e); | ||
| } | ||
| }); | ||
| } |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
rg -n --type=java "processFaceDetection"Repository: Cap-go/capacitor-camera-preview
Length of output: 211
🏁 Script executed:
# Also search across all file types in case it's called from Kotlin or other files
rg -n "processFaceDetection"Repository: Cap-go/capacitor-camera-preview
Length of output: 211
🏁 Script executed:
# Find the CameraPreview.java file to examine its full context
find . -name "CameraPreview.java" -type fRepository: Cap-go/capacitor-camera-preview
Length of output: 152
🏁 Script executed:
# Look for ImageAnalysis setup in the codebase
rg -n "ImageAnalysis" --type=javaRepository: Cap-go/capacitor-camera-preview
Length of output: 57
🏁 Script executed:
wc -l ./android/src/main/java/app/capgo/capacitor/camera/preview/CameraPreview.javaRepository: Cap-go/capacitor-camera-preview
Length of output: 156
🏁 Script executed:
cat -n ./android/src/main/java/app/capgo/capacitor/camera/preview/CameraPreview.javaRepository: Cap-go/capacitor-camera-preview
Length of output: 8172
🏁 Script executed:
# Check if there are any Kotlin files in the android directory that might set up ImageAnalysis
find ./android -name "*.kt" -type f | head -20Repository: Cap-go/capacitor-camera-preview
Length of output: 57
🏁 Script executed:
# Search for ImageAnalysis in Kotlin files
rg -n "ImageAnalysis" --type=kotlinRepository: Cap-go/capacitor-camera-preview
Length of output: 223
🏁 Script executed:
find . -name "CameraXView.java" -o -name "CameraXView.kt" | head -5Repository: Cap-go/capacitor-camera-preview
Length of output: 150
🏁 Script executed:
# Search more broadly for any setup of face detection or image analysis
rg -n "setAnalyzer|ImageAnalysis|Analyzer" --type=javaRepository: Cap-go/capacitor-camera-preview
Length of output: 57
🏁 Script executed:
# Search for any callback or listener that might process frames
rg -n "onFrameAvailable|onImageAvailable|processImage" --type=javaRepository: Cap-go/capacitor-camera-preview
Length of output: 57
🏁 Script executed:
wc -l ./android/src/main/java/app/capgo/capacitor/camera/preview/CameraXView.javaRepository: Cap-go/capacitor-camera-preview
Length of output: 155
🏁 Script executed:
# Read the CameraXView file to see if there's any image analysis or face detection setup
cat -n ./android/src/main/java/app/capgo/capacitor/camera/preview/CameraXView.javaRepository: Cap-go/capacitor-camera-preview
Length of output: 50388
🏁 Script executed:
# Search for ImageAnalysis in CameraXView.java specifically
rg -n "ImageAnalysis" ./android/src/main/java/app/capgo/capacitor/camera/preview/CameraXView.javaRepository: Cap-go/capacitor-camera-preview
Length of output: 57
🏁 Script executed:
# Search for any calls to processFaceDetection in CameraXView.java
rg -n "processFaceDetection" ./android/src/main/java/app/capgo/capacitor/camera/preview/CameraXView.javaRepository: Cap-go/capacitor-camera-preview
Length of output: 57
🏁 Script executed:
# Look for frame processing, analysis, or any InputImage usage in CameraXView
grep -n "InputImage\|setAnalyzer\|Analyzer\|onImageAvailable" ./android/src/main/java/app/capgo/capacitor/camera/preview/CameraXView.javaRepository: Cap-go/capacitor-camera-preview
Length of output: 57
🏁 Script executed:
# Double-check: search entire repository for any usage of the faceDetector variable
rg -n "faceDetector\." --type=java | grep -v "faceDetector = \|faceDetector\.process\|faceDetector\.close"Repository: Cap-go/capacitor-camera-preview
Length of output: 57
processFaceDetection is dead code — it is never called anywhere.
The method is defined but there is no ImageAnalysis use case configured in the CameraX pipeline. Face detection is completely non-functional as implemented. To fix this, an ImageAnalysis use case must be added to bindCameraUseCases() in CameraXView, and its Analyzer must construct an InputImage from each ImageProxy and invoke processFaceDetection.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@android/src/main/java/app/capgo/capacitor/camera/preview/CameraPreview.java`
around lines 90 - 114, processFaceDetection is never invoked because there is no
ImageAnalysis use case; update CameraXView.bindCameraUseCases() to create and
attach an ImageAnalysis instance to the camera pipeline, set an Analyzer that
converts each ImageProxy into an ML Kit InputImage and calls
processFaceDetection(InputImage), and ensure the Analyzer closes the ImageProxy
(imageProxy.close()) after processing; attach the ImageAnalysis use case when
binding to the lifecycle/camera (so Face detection runs in real time) and
reference the existing processFaceDetection method from the Analyzer.
android/src/main/java/app/capgo/capacitor/camera/preview/CameraPreview.java
Outdated
Show resolved
Hide resolved
"Implemented real-time face detection for iOS using Apple's Vision framework. The logic processes video frames in the background to detect face rectangles and emits the data via NotificationCenter to maintain compatibility with the plugin's architecture. This completes the cross-platform requirement alongside the Android ML Kit implementation."
|
I have implemented face detection for both Android (ML Kit) and iOS (Vision Framework). Ready for review. @algora-bot |
"Refactored build.gradle to remove unprofessional comments and updated Google ML Kit Face Detection dependency to the latest stable version for better build consistency."
2 participants
This PR implements face detection for Android as requested in
#200. I used Google ML Kit for high performance.
Summary by CodeRabbit
New Features
Refactor
Breaking Changes