Supporting an Additional Input Modality (e.g. Video) in Optimum ExecuTorch

[yujiaoliang]

Background

Optimum ExecuTorch currently supports models with up to two input modalities (commonly text + image/text + audio).
While this covers many existing multimodal use cases, some newer multimodal models introduce three or more input modalities.

For example, models such as Qwen3-VL natively support:

• Text

• Image

• Video

At the moment, deploying such models on ExecuTorch requires either:

• Dropping one modality at export time, or

• Converting one modality (e.g. video) into another (e.g. sampled frames → image),

both of which reduce model completeness or expressiveness.

This raises the question:

Should Optimum ExecuTorch be extended to support an additional input modality, and if so, how should this be designed?

Discussion Points

I would like to open a discussion around the following aspects:

API / Interface Design

Should input modalities be treated as a fixed set (e.g. text/image/video)?

Or should ExecuTorch move toward a more generic “N-modality” input abstraction?

Graph Representation & Export

How should additional modality-specific preprocessing and operators be represented in the exported graph?

Are there existing constraints in ExecuTorch IR that assume two modalities?

Runtime & Scheduling

Would an additional modality require changes in scheduling, memory planning, or execution order?

Are there known limitations that would block such an extension?

Incremental Support

Would it be reasonable to start with a minimal third modality (e.g. video as a sequence of frames) and evolve from there?

Motivation

• Supporting more flexible multimodal inputs could:

• Improve compatibility with modern multimodal foundation models

• Reduce model-specific hacks or modality dropping

Call for Feedback / Contributions

I would appreciate feedback from maintainers and the community on:

• Whether this direction aligns with the project roadmap

• Potential design constraints or prior discussions

• Interest in collaborating on a prototype or RFC if this is deemed feasible

I’m happy to help with design exploration, implementation, or documentation if there is interest.