Metal (or close) GPU-enabled containers for Toolio (presumably for any MLX projects)?

[uogbuji]

For forthcoming work, it would be really useful to be able to containerize some Toolio-based deliverables. I know there are a lot of limitations and complexities to GPU-included virtualization on Apple Silicon. Indeed, the topic came up in the early days of MLX. That discussion did seem to be limited to Docker, and there are more options than Docker.

Andreas Kunar pulled together a Podman-based recipe in "GPU-Accelerated Containers for M1/M2/M3… Macs", but he's focused on llama.cpp and declared his efforts abandoned once llama.cpp's CPU inference became fast enough.

Just a couple of weeks ago a couple of Red Hatters presented a DevConf session: [GPU Accelerated Containers on Apple Silicon with libkrun and podman machine - DevConf.US 2024]
(https://www.youtube.com/watch?v=OyTJ8FtQaJ0). In effect, libkrun would support GPU passthrough for VMs, and is available for Podman as of 5.2 (and are auto-included in Podman installs). Their demo is also llama.cpp-based. Man, people gotta start putting 'spect on MLX's name 😂.

This libkrun/Podman angle is promising, and as developments continue in a variety of areas, perhaps there will come a solution for this.

[uogbuji]

This might be available now? See the macOS (Silicon) tab on this Podman doc.