Feature/audio input

.gitignore

@@ -53,3 +53,4 @@ macos/Resources/litertlm-server.jar
 
 # Temporary/internal documentation
 temporary_docs/
+DESKTOP_DEBUG.md

CHANGELOG.md

@@ -1,3 +1,19 @@
+## 0.12.3
+- **Android LiteRT-LM Engine**: Added LiteRT-LM inference engine for Android
+  - Automatic engine selection based on file extension (`.litertlm` → LiteRT-LM, `.task/.bin` → MediaPipe)
+  - NPU acceleration support (Qualcomm, MediaTek, Google Tensor)
+- **Audio Input Support**: Audio input for Gemma 3n models via LiteRT-LM
+  - Platforms: Android + Desktop (macOS, Windows, Linux)
+  - WAV format (16kHz, mono, 16-bit PCM)
+  - `supportAudio` parameter in session configuration
+- **Desktop LiteRT-LM Fixes**: Fixed text chat and audio on desktop platforms
+  - Switched from Flow-based to Callback-based async API (matches Android)
+  - Audio transcription now works correctly
+- **Bug Fixes**:
+  - Fixed model deletion not removing metadata
+  - Fixed model creation failure blocking switching to another model
+  - Fixed download issues for large models
+
 ## 0.12.2
 - **Model Deletion Fix**: Fixed model deletion not removing metadata (#169)
 - **Model Switch Fix**: Fixed model creation failure blocking switching to another model (#170)

CLAUDE.md

@@ -35,6 +35,13 @@
 - ✅ **REQUIRED**: Simple, clean commit messages without AI mentions
 - ✅ **REQUIRED**: Use `--author="Sasha Denisov <denisov.shureg@gmail.com>"`
 
+## Rule 5: SEARCH ALL FILES ⛔
+- ❌ **FORBIDDEN**: Using grep/search with file extension filters unless explicitly requested
+- ✅ **REQUIRED**: When user says "search for X", search ALL files without extension filters
+- ✅ **REQUIRED**: Use `grep -rn "pattern" /path/ 2>/dev/null | grep -v node_modules | grep -v ".gradle/"` (no --include flags)
+
+**Why:** Filtering by extensions misses important files like `.podspec`, `.plist`, `.json`, etc.
+
 ---
 
 ## Project Overview
@@ -383,6 +390,23 @@ Future<void> close() async {
 | Qwen2.5 | ✅ | ❌ | ❌ | Android, iOS, Web |
 | Phi-4 | ❌ | ❌ | ❌ | Android, iOS, Web |
 
+### Platform Limitations
+
+| Platform | Vision/Multimodal | Audio | Notes |
+|----------|-------------------|-------|-------|
+| Android | ✅ Works | ✅ LiteRT-LM only | Full support |
+| iOS Device | ✅ Works | ❌ Not supported | Full vision support |
+| iOS Simulator | ❌ Broken | ❌ Not supported | MediaPipe incompatible with Apple Silicon simulator |
+| Web | ✅ Works | ❌ Not supported | MediaPipe only |
+| macOS | ⚠️ Broken (#684) | ✅ LiteRT-LM only | Vision: image sent but model hallucinates |
+| Windows | ✅ Works | ✅ LiteRT-LM only | Desktop via gRPC |
+| Linux | ✅ Works | ✅ LiteRT-LM only | Desktop via gRPC |
+
+**Known Issues:**
+- **iOS Simulator (#178)**: Vision doesn't work on Apple Silicon simulators due to MediaPipe dependency incompatibility. Use physical device.
+- **macOS Vision (#684)**: LiteRT-LM JVM SDK vision bug - image bytes sent correctly but model hallucinates response
+- **Audio**: Only supported with LiteRT-LM models (`.litertlm`), not MediaPipe (`.task`)
+
 ## Development Environment
 
 ### Required Versions
@@ -614,11 +638,38 @@ Desktop platforms (macOS, Windows, Linux) use LiteRT-LM via Kotlin/JVM with gRPC
 **Automatic Setup (recommended):**
 
 The build script automatically:
-1. Downloads Temurin JRE 21 (cached in `~/.cache/flutter_gemma/jre/`)
+1. Downloads Azul Zulu JRE 24 (cached in `~/.cache/flutter_gemma/jre/`)
 2. Copies JAR from `litertlm-server/build/libs/`
 3. Signs binaries for development
 4. Removes quarantine attributes
 
+> ⚠️ **CRITICAL: Use Azul Zulu, NOT Temurin!**
+> Temurin JRE causes Jinja template errors with LiteRT-LM native library on macOS.
+> The error manifests as: `messages[0]['content'][0]['text']` parsing failure.
+> Zulu JRE 24 works correctly with both CPU and GPU backends.
+
+**Vision/Multimodal Status (macOS):**
+
+> ⚠️ **Known Issue:** Vision is currently broken on macOS with LiteRT-LM JVM SDK.
+> - Image bytes are sent to the model (verified in logs)
+> - Model **does NOT see the image** and hallucinates a response
+> - GitHub Issue: https://github.com/google-ai-edge/LiteRT-LM/issues/684
+>
+> **Workaround:** Use text-only mode until the SDK bug is fixed.
+
+**If you encounter "Failed to create executor for subgraph" error:**
+Clear GPU cache (required after JRE changes):
+```bash
+find /var/folders -path "*/C/dev.flutterberlin.flutterGemmaExample55*" -type d 2>/dev/null | xargs rm -rf
+```
+See `DESKTOP_DEBUG.md` for full cache clearing instructions.
+
+**Technical details:**
+- Vision enablement uses same logic as Android: `visionBackend = if (maxNumImages > 0) backend else null`
+- When `supportImage: true`, client sends `maxNumImages: 1`
+- Server sets `visionBackend=GPU`, image bytes are transmitted
+- SDK internal logs show `max_num_images: 0` - this is internal default, not our code
+
 Just run:
 ```bash
 flutter run -d macos
@@ -1183,10 +1234,14 @@ implementation 'com.google.ai.edge.litertlm:litertlm-android:0.9.0-alpha01'
 ### ✅ Desktop Platform Support (v0.12.0+)
 - **macOS, Windows, Linux** support via LiteRT-LM JVM
 - **gRPC architecture** - Dart client communicates with Kotlin/JVM server
-- **Bundled JRE** - Temurin 21 automatically downloaded and bundled
+- **Bundled JRE** - Azul Zulu 24 automatically downloaded and bundled
 - **Automatic setup** - Xcode build phase handles JRE/JAR bundling
 - **Code signing** - Development signing handled automatically
 - **New models added** - Qwen3 0.6B, Gemma 3 1B LiteRT-LM format
+- **GPU acceleration** - Works on Apple Silicon (Metal backend)
+- **Vision/Multimodal** - Currently broken on macOS (SDK bug #684), image sent but model hallucinates
+
+> ⚠️ **JRE Compatibility Note:** Temurin JRE causes Jinja template errors with LiteRT-LM native library. Use Azul Zulu JRE instead.
 
 **Key Files:**
 - `lib/desktop/flutter_gemma_desktop.dart` - Dart plugin implementation

DESKTOP_SUPPORT.md

@@ -72,14 +72,14 @@ Desktop support uses a different architecture than mobile platforms:
 
 ## Supported Platforms
 
-| Platform | Architecture | GPU Acceleration | Status |
-|----------|-------------|------------------|--------|
-| macOS | arm64 (Apple Silicon) | Metal | ✅ Ready |
-| macOS | x86_64 (Intel) | - | ❌ Not Supported |
-| Windows | x86_64 | DirectX 12 | ✅ Ready |
-| Windows | arm64 | - | ❌ Not Supported |
-| Linux | x86_64 | Vulkan | ✅ Ready |
-| Linux | arm64 | Vulkan | ✅ Ready |
+| Platform | Architecture | GPU Acceleration | Vision | Audio | Status |
+|----------|-------------|------------------|--------|-------|--------|
+| macOS | arm64 (Apple Silicon) | Metal | ❌ | ✅ | ✅ Ready |
+| macOS | x86_64 (Intel) | - | - | - | ❌ Not Supported |
+| Windows | x86_64 | DirectX 12 | ❌ | ✅ | ✅ Ready |
+| Windows | arm64 | - | - | - | ❌ Not Supported |
+| Linux | x86_64 | Vulkan | ❌ | ✅ | ✅ Ready |
+| Linux | arm64 | Vulkan | ❌ | ✅ | ✅ Ready |
 
 > **⚠️ Platform Limitations**
 >
@@ -98,7 +98,7 @@ Desktop support uses a different architecture than mobile platforms:
 
 - **Flutter**: 3.24.0 or higher
 - **Dart**: 3.4.0 or higher
-- **Java Runtime**: JRE 21 (automatically downloaded if not present)
+- **Java Runtime**: JRE 24 Zulu (automatically downloaded if not present)
 - **Model Format**: LiteRT-LM `.litertlm` files only (MediaPipe `.bin`/`.task` not supported)
 
 ### macOS
@@ -229,7 +229,7 @@ This is necessary because:
 2. `macos/scripts/setup_desktop.sh` executes after app bundle is created:
    - **Builds JAR from source** if JDK 21+ is available (checks JAVA_HOME, Homebrew, system)
    - Falls back to downloading pre-built JAR from GitHub Releases
-   - Downloads Temurin JRE 21 (cached in `~/Library/Caches/flutter_gemma/jre/`)
+   - Downloads Azul Zulu JRE 24 (cached in `~/Library/Caches/flutter_gemma/jre/`)
    - Extracts native library to `Frameworks/litertlm/`
    - Signs all binaries with sandbox inheritance entitlements
 
@@ -289,7 +289,7 @@ Windows uses CMake with a PowerShell build script. No additional configuration r
 2. `windows/scripts/setup_desktop.ps1` executes:
    - **Builds JAR from source** if JDK 21+ is available (checks JAVA_HOME, common install locations)
    - Falls back to downloading pre-built JAR from GitHub Releases
-   - Downloads Temurin JRE 21 (cached in `%LOCALAPPDATA%\flutter_gemma\jre\`)
+   - Downloads Azul Zulu JRE 24 (cached in `%LOCALAPPDATA%\flutter_gemma\jre\`)
    - Extracts DLLs from JAR
 
 #### App Directory Structure