HDDS-14379. Implement basic Hadoop OM client proxy provider to read from followers

[ivandika3]

What changes were proposed in this pull request?

This task is to come up with the basic implementation of follower read client proxy as a baseline before further performance improvements. The idea is to simply pick a OM node in random (which be leader or follower / listener) and use it to submit read requests. If the chosen OM node is a follower, the read requests need to keep sending to that OM node unless the OM is down which triggers failover. Write requests should be sent to the OM leader directly.

Further improvements such as followers priority or picking OM based on latency, applied index, etc will be implemented in follow up tasks. The main focus of this patch is to ensure that long-lived client can read from followers. In the previous leader proxy provider implementation, the client only read from followers until a new write request triggers OMNotLeaderException and the failover causes proxy to always be pointing to the leader.

The implementation is to introduce HadoopRpcOMFollowerReadProxyProvider which wraps
HadoopRpcOMFailoverProxyProvider. FollowerReadProxyProvider tracks a different currentOmNodeId from HadoopRpcOMFailoverProxyProvider. FollowerReadInvocationHandler will check whether the request is a read request (using OmUtils#isReadOnly) and if so forwards it to its current proxy. If it's a write request, the request if forwarded to HadoopRpcOMFailoverProxyProvider to be sent to the leader, which means that these clients do not utilize OM followers as much as we want.

So the proxy hierarchy (each with its own InvocationHandler) is

• TracingUtil's proxy (InvocationHandler: TraceAllMethod)
  • RetryProxy (InvocationHandler: RetryInvocationHandler)
    • HadoopRpcOMFollowerReadProxyProvider (InvocationHandler: FollowerReadInvocationHandler)
      • ProtocolProxy (which is created in OMFailoverProxyProviderBase#createOMProxy): ProtobufRpcEngine.Invoker

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-14379

How was this patch tested?

UT and IT.