orchestrate: improve visibility on incoming connections #36
Description
It's been impossible to tell which clients and what they are logging through slog-agent.
The existing information:
- client addresses
- orchestration keys
are not sufficient in situations where the keys cannot differentiate clients (e.g. multi-instance services), and full information cannot be represented in metrics due to concerns about metric label cardinality (M inputs * N pipelines * T steps * U outputs).
A workaround is to log key fields when an incoming connection requests to dispatch logs to a previously-unused pipeline (id'ed by by orchestration keys), done in major: Use Go 1.18 generics, but turned out useless since it logs only the orchestration keys.
An option is to log the metrics keys in addition to orchestration keys, and / or dump raw headers in one of the new logs.
A better solution would be to save samples of incoming logs with information on destination pipelines assigned by orchestrator, and print them on certain signals, which may be done as part of #8