Explore options for limiting inbound connections + resources #248
Description
The receivers that rely on inbound HTTP connections (OTLP & Fluent) do not currently limit the number of concurrent connections. This could lead to resource exhaustion if too many concurrent connections are attempted at the same time. For each connection we may read + unmarshal incoming payloads into memory. It's possible that during back pressure connections build up, leading to memory/resource exhaustion.
We should identify ways to limit inbound connections and the resources associated with them.
Some examples:
- See Go's handling of MaxIdleConns, MaxIdleConnsPerHost, and MaxConnsPerHost: https://pkg.go.dev/net/http#Transport
- Tokio example of using a semaphore to limit all connections (idle + active): https://docs.rs/tokio/latest/tokio/sync/struct.Semaphore.html#limit-the-number-of-incoming-requests-being-handled-at-the-same-time
- Does the tonic gRPC lib have any limits?
Secondly, we may want to separately limit how many concurrent decoding threads are active. In the exporters we limit the number of encoders by limiting the size of the FuturesOrdered collection. We may want to do similar for receivers that spawn decoding threads. This would limit spawned threads, but we may still allocate memory for read payloads.
Finally, while limiting connections is straight-forward, it doesn't really solve the core problem of limiting memory/resources exhaustion. Given variable message sizes, it's difficult to balance fewer connections with larger payloads vs. many connections with smaller payloads. Ideally we could cap total active memory allocation size for a receiver across all connections.