Integrate onnx-go as backend to remove onnxruntime dependency

[gregfurman]

Hey! The onnx-go project is being maintained again 🥳 See https://github.com/oramasearch/onnx-go.

Would the hugot team consider adding support for it as a backend? That way the external dependency on the onnx runtime can be (optionally) removed.

[RJKeevil]

Hi Greg, thanks for the heads up. I'm a little reluctant to do the work to integrate this one, as it relies on gorgonia for its backend, which appears to be dormant (no new features for 2 years), i suspect it doesnt work with current models. Go-native backends often seem to have a problem keeping up with the state of the art, at least until the Go ML community grows. We are looking however at alternative backends and are closely working with GoMLX to natively integrate with their tooling. It is however still C based behind the scenes, but some frameworks are being developed there to make dependency management much easier in future!

[janpfeifer]

That's cool to hear. I should mention that https://github.com/oramasearch/onnx-go was one of the projects that inspired us to do onnx-gomlx, by showing the conversion was possible and a reasonable thing to do.

Greg, I'm not sure what are your interests/incentives on the https://github.com/oramasearch/onnx-go project, but I wonder if there are other ways we could collaborate on this.

One thing that comes to mind: GoMLX started with XLA as a backend (for its performance/efficiency), but it was designed with multiple-backends in mind. In particular, it's on my TODOs to use Gorgonia as a backend: while slower (a pure Go engine will never be a top-performance), being pure Go has its advantages during development (no C libraries to install), or for smaller ML models.

Anyway, just an idea. If you would like to brainstorm sometime and discuss potential other collaborating ideas, pls reach out.

Btw, I should mention I'm meeting this week folks from https://github.com/AdvancedClimateSystems/gonnx (another ONNX engine for Go), also to discuss potential collaborating ideas -- looping in @Swopper050 on this.

[gregfurman]

Greg, I'm not sure what are your interests/incentives on the https://github.com/oramasearch/onnx-go project

Thanks for the response! I'm pretty agnostic as to what framework is used, since I'm wanting integrate some more ML capabilties to Bento, a Golang stream-processing tool I help maintain.

While speed/performance is not actually a top priority, some users run Bento serverlessly (i.e in AWS Lambda) so resource usage is a consideration. A top priority though is minimising external dependencies. One of the key-selling points of Bento is that it is packaged, and runs, as a single-binary and should just work out-the-box.

I actually already have a working POC I built using Hugot that works brilliantly (see PR) but the reliance on the external C and Rust dependencies means we have to package and release the distro with Hugot seperately -- which has turned out to be a dealbreaker.

... being pure Go has its advantages during development (no C libraries to install), or for smaller ML models.

Hugot has abstracted away a lot of the scary linear algebra which makes it a lot easier to develop against. If we could help create an easy-to-use Hugot distro that runs in pure Go, that would be 🥇 !

Anyway, just an idea. If you would like to brainstorm sometime and discuss potential other collaborating ideas, pls reach out.

I would love to help out however I can! I think the Go ecosystem could really benefit from a stable library for ML inference.

[janpfeifer]

Bento looks very cool!

And I understand the desire for minimal dependencies, and in particular not wanting to manage the installation of separate Rust/C packages -- I hate that too, mainly because C++ various build systems are so complicated and just don't work together, or more specifically, they don't work together with the go tooling.

For Rust I wish the go tooling could depend on Cargo, and we could just add a dependency and everything would automatically, and statically be linked in. Maybe one day ...

Btw, from your perspective, would it help if every C dependency were linked statically ? So nothing to install in the workers ? I mean, it would still require library installation in the box building the binaries, but at least the final binary wouldn't have any dependencies ?

[gregfurman]

Bento looks very cool!

Thanks! I envision that it would plug-in really nicely to the tool and be a great value-add. Especially since it'll make ML inference a lot more accessible to our user-base.

And I understand the desire for minimal dependencies, and in particular not wanting to manage the installation of separate Rust/C packages -- I hate that too, mainly because C++ various build systems are so complicated and just don't work together, or more specifically, they don't work together with the go tooling.

Yeah exactly. I ended up not taking this approach on due to the overhead of needing to compile a libtokenizers + libonnxruntime for each GOARCH + GOOS combination we were wanting to support. Felt like I was introducing a lot of complexity to our CI and creating a complicated installation process for users, pretty much defeating the point of using Go in the first place.

Also, not to mention, that a target sys may have a different version of C installed (leading to this irritating ld: warning: object file (...) was built for newer macOS version (<ver>) than being linked (<other ver>) issue that I have cursed my local Go development environment with).

Btw, from your perspective, would it help if every C dependency were linked statically ?

I was actually looking into this during my Hugot integration! It would definitely help but, at the time, I didn't feel comfortable releasing + supporting a distro with statically linked libraries, especially given my unfamilarity with the CGO ecosystem.

With all this said, I would be very happy with a POC of a distro with statically linked libs for a single GOOS/GOARCH combination, say Linux ARM64. This way, I could see whether said libraries play nicely with Bento and what trade-offs will be made -- with larger memory usage and binary size being the obvious ones.

[riccardopinosio]

hi @gregfurman

Hugot has abstracted away a lot of the scary linear algebra which makes it a lot easier to develop against. If we could help create an easy-to-use Hugot distro that runs in pure Go, that would be 🥇 !

This is great to hear and it's the general idea of hugot, the goal was to provide the same level of abstractions as huggingface transformers/pipelines for both inference and eventually training without the low-level details and implementing the needed pipeline pre/postprocessing, then in the background hugot can use onnx, gomlx, or whatever backend indeed.

I actually already have a working POC I built using Hugot that works brilliantly (see PR) but the reliance on the external C and Rust dependencies means we have to package and release the distro with Hugot seperately -- which has turned out to be a dealbreaker.

I am currently working on enabling fine-tuning of pipelines via gomlx but given what you are discussing it might make sense to also prioritize making it easier to run hugot for your use case. I currently see two routes:

1. keeping the rust/c binding route and compile everything statically for e.g. linux as @janpfeifer suggests. Here the issue will not be tokenizers but onnxruntime, it's a bit tricky to statically link onnxruntime but it seems at least possible (see also Static library usage microsoft/onnxruntime#8555). The downside of this will be a larger binary, however the upside is that we are relying on C/rust software that is widely used an maintained. One of the issue with native go solutions at this level is that they often don't have the manpower behind them to be maintained properly.
2. Implement a pure go backend for hugot that runs inference with onnx models or potentially another format. I'm not sure what the best option would be here, whether https://github.com/oramasearch/onnx-go or https://github.com/AdvancedClimateSystems/gonnx (which @janpfeifer pointed out). Both seem gorgonia based. I would probably aim to support one way to run onnx for inference natively in hugot, so that any upstream contributions can be focused on making one lib good.

I am inclined to give a try to 2. actually because I don't estimate it will take too much work, just need to choose which library to integrate. Then I guess @gregfurman you would be able to integrate more easily hugot in bento?

[Swopper050]

Hey there! @wipsel and I are the maintainers of gonnx. I saw this issue and I just wanted to let you guys know that should you want to integrate gonnx as a backend we would be happy to help things get started. Also if operators would be missing that you guys would require we could implement them.

Feel free to contact us should you want to integrate gonnx or would want to know some more about the project!

[RJKeevil]

Hi @Swopper050 and @wipsel, i have just pushed a branch with go-onnx as an alternative backend. @riccardopinosio will do a pass to see if we can abstract the backends away further, but i think this is enough for you to take a look.

If you run the tests, first thing you will see is that all of our test models fail due to unsupported opset version. Obviously it will take time to increase the opset version, so perhaps step one is to downgrade the models opset. We tried first with distilbert and were still missing some ops, so perhaps step one is to iterate on the test models and implement the remaining ops from 13 so that we can run the test models (exported with opset 13) with gonnx.

[Swopper050]

Hi @RJKeevil, awesome.

I'll have a look which models you are using and which opsets and operators are missing and see if we can easily add this!

[RJKeevil]

As an experiment, ive also just pushed to this branch a commit that when running with "GO" as runtime, it will use https://github.com/sugarme/tokenizer instead of bindings to the rust tokenizers. If all works ok we now have a means for running pure-go Hugot.

[RJKeevil]

OK, so now we have a branch: alt-backends, if run with no build args it replicates the previous behaviour of Hugot (apart from also testing everything also with Cuda).

If run with -tags GO you can call NewGoSession to create a session with a gonnx backend, and you can run hugot_go_tests.go (which currently all fail due to opset support) @Swopper050 @wipsel @gregfurman.

If run with -tags XLA you can call NewXLASession to create a session with a goMLX backend, and you can run hugot_xla_tests.go (which currently all fail due to gcc incompatibility) @janpfeifer.

Once these unit tests pass we can hide ORT behind a similar build flag, so it ceases to be required at compile time. Then we could e.g. default to GO for ease of use, and allow users to switch to ORT or XLA for performance/compatibility/accelerators/training etc etc

Let me know if this new structure works for you, it was a pretty big change!