grpcs:// URLs don't work out-of-the-box for some strict load balancers

[jasonschroeder-sfdc]

I was setting up bb-browser (which depends on bb-storage for CAS/AC) and hit a confusing error:

connection error: desc = "transport: authentication handshake failed: tls: first record does not look like a TLS handshake"

After much debugging, it seems this default is the problem:

bb-storage/pkg/util/tls.go

Line 162 in ed3a308

tlsConfig.ServerName = configuration.ServerName

If the configuration.ServerName is empty (which is the default), the LoadBalancer in front of a gRPC service may deny the connection.

As a better default, I suggest to use the server name from the URI. After configuring like this:

  blobstore: {
    actionCache: {
      grpc: {
        address: 'rbe.internal.company.com:7443',
        tls: {
            server_name: 'rbe.internal.company.com',
        },
      },
    },

Everything worked beautifully ✨

[anguslees]

+1. I lost a few days debugging an unhelpful "tls: illegal parameter" response from my non-buildbarn RBE server to bb_clientd, and found the answer just now with the help of packet dumps. It turns out it was this issue: server_name by default is the same as address, but isn't allowed to include ports.

Setting server_name to std.split(address, ":")[0] is an easy enough fix, but I think this suggests the default in code should be improved.

[EdSchouten]

It turns out it was this issue: server_name by default is the same as address, but isn't allowed to include ports.

To me that sounds like a bug in grpc-go. It should just do the right thing, even if we leave tlsConfig.ServerName unset. Could you please file an issue at grpc-go? Thanks!