.litertlm mobile support

[Serjio42]

Hello.
I am trying to change the old preview .task model file gemma-3n-E2B-it-int4.task from here to the new stable one gemma-3n-E2B-it-int4.litertlm from here in my app code, and after the change the app stops working properly (stuck for a several minutes on the 1st chat answer attempt and then behave unexpectedly).

I wonder, whether everything is OK with these new models inference support.
I see the repeated

<end_of_turn><end_of_turn><end_of_turn><end_of_turn><end_of_turn>

in the log, looks like the model template/chat is handled incorrectly.

FYI: Google AI Edge Gallery have just moved to the .litertlm format on both Android and iOS, so it looks like the change is major and permanent.

I use FlutterGemma class under the hood.

The package version:

flutter_gemma:
    version: "0.11.13"

Thanks.

[DenisovAV]

Unfortunately, iOS support for LiteRT-LM is not publicly available yet. The Google AI Edge Gallery works on iOS via TestFlight, but the iOS source code is not published - only Android is open-sourced. According to this comment, the GPU acceleration code is not open-sourced yet and they're working on releasing it early next year.

I was planning to wait until iOS library is public and add Android + iOS support together. But if Android-only would work for you, I can add .litertlm support for Android in the next version.

Let me know!

As a workaround you can use .litertlm files with the current version by setting fileType: ModelFileType.binary - this will filter out the extra <end_of_turn> tags. However, this runs through MediaPipe, not LiteRT-LM runtime, so you won't get NPU acceleration (which is the main benefit of .litertlm). Probably not worth it.

[Serjio42]

Thanks for the answer!

Than, I will wait for the iOS support by LiteRT-LM. It is not the urgent issue.
Let's wait, and when LiteRT-LM and subsequent FlutterGemma implementations will be ready, we will be able to close this issue.

Have a good holidays, @DenisovAV!

[victox5]

Hi @DenisovAV, I've been trying different models in my app and I realized that litertlm models such as https://huggingface.co/litert-community/Qwen3-0.6B/tree/main only run on CPU, with GPU backend they crash. Is this something expected? Would it work if I take the vanilla 0.6B version and export it to .task through AI Edge Torch?

PS: Tests were on Android.

[DenisovAV]

The issue is that MediaPipe doesn't fully support .litertlm models properly.

Looks like it's time to add native LiteRT-LM support. I'll add it in the next few days.

[DenisovAV]

please try 0.12.3 Android