Build musl-based statically linked binaries

[defect]

Hi! Thanks for a cool project!

I'm testing out rotel as an alternative to the opentelemetry-collector but unfortunately we have some devices that run with a fairly old glibc (ubuntu 20.04 based), so the distributed binaries from the releases don't work on them.

I ended up building a binary that's statically linked with musl to make it self-contained and portable to systems with older glibc (or no glibc at all, i guess), and wanted to see if that is a reasonable approach? Maybe you've already tried this and come to the conclusion that it's broken :)

Some caveats, as this needs a bit of work if it's at all interesting;

• I haven't actually sent any real amount of data through the musl rotel yet. I will try to do that today or tomorrow. A basic configuration starts up with out errors at least
• I'm doing a weird link for g++ in the build step. This is a workaround i found in a github comment, and i think the issue stems from https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=988837 . It might need a bit more scrutiny to make sure it's not messing anything up
• I'm not adding statically linked musl releases for the python processor, just the rotel binary (and clickhouse-ddl)

[rjenkins]

Thanks for the PR @defect. The caveats make sense, let us know what your testing looks like. I'll do some testing on my end early next week and will share what I find. A couple additional thoughts.

1. We might also want to add a dynamically linked musl build for users who want to run on alpine. I can take a look at this next week.
2. Do you want to build with musl's default allocator. It's been a bit since I've used musl but I've read a few times the performance might not be great. re: https://nickb.dev/blog/default-musl-allocator-considered-harmful-to-performance/. We may want to try with jemalloc or MiMalloc if that's an option for you, depending on testing results.

[defect]

@rjenkins I did a bit of smoke testing on our target platform yesterday and it seems like it works for that workload (traces over OTLP only right now). However, it might be a bit more taxing on the CPU than expected as you say. I will try to do the same tests with a musl rotel vs. standard glibc rotel.

I also pushed a commit here (or would you prefer that as a separate PR?) that adds features for compiling with jemalloc or mimalloc. They both build nicely on all platforms except for jemalloc on aarch64-unknown-linux-musl, which is a little bit annoying because that was my planned target :) jemalloc needs libjemalloc-dev and librust-jemalloc-sys-dev to build on ubuntu.

I'll try to get some time this weekend to do some performance testing there.

[rjenkins]

@rjenkins I did a bit of smoke testing on our target platform yesterday and it seems like it works for that workload (traces over OTLP only right now). However, it might be a bit more taxing on the CPU than expected as you say. I will try to do the same tests with a musl rotel vs. standard glibc rotel.

I also pushed a commit here (or would you prefer that as a separate PR?) that adds features for compiling with jemalloc or mimalloc. They both build nicely on all platforms except for jemalloc on aarch64-unknown-linux-musl, which is a little bit annoying because that was my planned target :) jemalloc needs libjemalloc-dev and librust-jemalloc-sys-dev to build on ubuntu.

I'll try to get some time this weekend to do some performance testing there.

Nice, glad it's working. One PR is fine, thanks for the changes. +1 on the testing and we can see if using jemalloc or mimalloc helps any. Will take a look on Monday to help verify perf findings.

[defect]

I did a bit of not-very-scientific performance testing on our target platform (aarch64) yesterday and it does look like musl+mimalloc is promising.
Baseline rotel dynamically linked against glibc (2.31 on ubuntu 20.04):

Statically linked against musl with the default allocator:

Statically linked against musl with mimalloc:

I'll try to get the benchmark suite going with some of the {platform, libc, allocator} combinations to get some better data.

[rjenkins]

I did a bit of not-very-scientific performance testing on our target platform (aarch64) yesterday and it does look like musl+mimalloc is promising. Baseline rotel dynamically linked against glibc (2.31 on ubuntu 20.04): Statically linked against musl with the default allocator: Statically linked against musl with mimalloc:

I'll try to get the benchmark suite going with some of the {platform, libc, allocator} combinations to get some better data.

Agreed, looking promising. Nice work.

[rjenkins]

@defect How's this coming along? Have y'all made a decision on mimalloc vs. the default allocator for your project? Given the results here building with --features mimalloc for musl seems like the best option for most users.