gsw_service and app instrumentation

[AarC10]

Is your feature request related to a problem? Please describe.
It would be interesting to have metrics about ground software in Grafana as well and have an idea of the performance of our system, especially as we continue improving it now.

Describe the solution you'd like
Have Grafana dashboards that monitor GSW.

Additional context
https://grafana.com/docs/grafana-cloud/monitor-infrastructure/integrations/integration-reference/integration-golang/

[K3das]

Could integrate the Go OpenTelemetry collector (which can then export to a variety of tools and databases), not sure what performance impacts that would have.

[K3das]

It would be worth instrumenting the reader module with rate and packet loss metrics, as that would give more insight into the real-world performance of readers and help us trace bottlenecks much easier. gsw_service should be instrumented with at least packet rate metrics. Standard metrics, like go GC stats, should also be included. For both apps and gsw_service there should be a unified exporter config.

OpenTelemetry should be sufficient. Asynchronous instruments (where a callback is executed by OpenTelemetry to get a metric value) should be used to avoid OpenTelemetry directly impacting performance.