Safe HTTP/2 health-check debugging for lighthouse-agent (0.20.1): GODEBUG=http2debug=2 leaks Authorization Bearer token

[leasonliu]

What would you like to be added:

We’d like a supported way (ideally upstream, no downstream fork) to observe HTTP/2 connection/health-check behavior without exposing tokens—either by:

redacting sensitive headers in the relevant debug logging path, or
providing a more targeted/safer “connection/health-check debug” mode that does not dump full requests/responses.

If there’s an existing supported knob in 0.20.1 that we missed, we’d love to use it.
If you think a downstream patch is unavoidable, any high-level guidance for maintaining a fork for admiral repo would also be appreciated.

Why is this needed:
We’re running into intermittent service-discovery sync delays and are trying to debug whether lighthouse-agent is holding on to stale/broken HTTP/2 connections to the Kubernetes API server.

Context / what triggered this

• We’re on AKS and observed periods where EndpointSlice updates appear delayed (EndpointSlice/watch processing lag on the control plane), which then delays Lighthouse’s EndpointSlice-driven updates.
• We opened a case with the AKS team. Their suspicion is that the control plane may have delays, but they also asked us to validate whether the client connection might have been broken while the agent continued to use a stale connection (i.e., connection health / keepalive / HTTP/2 behavior).
• AKS suggested we “enable HTTP/2 health check logging” to confirm whether the agent’s HTTP/2 connection is still healthy during the incident window.

Important detail

• As far as we can tell, HTTP/2 health checks are already enabled by default in the agent; our problem is observability: we can’t see useful health-check logs without enabling very verbose HTTP/2 debug output.

What we tried

• We set GODEBUG=http2debug=2 in the lighthouse-agent pod to get HTTP/2 client debug logs.
• This does show health-check activity, but it also logs request/response details broadly (and includes Authorization: Bearer …), which is not acceptable for us because logs are shipped to a centralized system.

Problem
We need a way to troubleshoot/confirm HTTP/2 connection health (especially around health checks and reconnects) without dumping sensitive headers/tokens.

[tpantelis]

Enabling GODEBUG=http2debug=2 yields output like:

2026/01/26 22:32:54 http2: Transport sending health check
2026/01/26 22:32:54 http2: Framer 0x40001f6380: wrote PING len=8 ping="Z\xc2\xcd0\xf7\bn\xa9"
2026/01/26 22:32:54 http2: Framer 0x40001f6380: read PING flags=ACK len=8 ping="Z\xc2\xcd0\xf7\bn\xa9"
2026/01/26 22:32:54 http2: Transport received PING flags=ACK len=8 ping="Z\xc2\xcd0\xf7\bn\xa9"
2026/01/26 22:32:54 http2: Transport health check success
2026/01/26 22:32:55 http2: Transport sending health check
2026/01/26 22:32:55 http2: Framer 0x40001301c0: wrote PING len=8 ping="\xae&\x85Ǫmȿ"
2026/01/26 22:32:55 http2: Framer 0x40001301c0: read PING flags=ACK len=8 ping="\xae&\x85Ǫmȿ"
2026/01/26 22:32:55 http2: Transport received PING flags=ACK len=8 ping="\xae&\x85Ǫmȿ"
2026/01/26 22:32:55 http2: Transport health check success

This is the HTTP/2 health check PING performed by Go's low-level http transport code. I don't see anything in the client-side Go http API (ie http.Transport) to control the behavior of the http2debug logging or to hook into the http2 transport code to override/alter the output.

Even if we could implement some way to provide what you're asking programmatically in the lighthouse-agent, it wouldn't be available in a release for several months. It sounds like you really need a solution now to troubleshoot your issue.

I don't know what centralized logging system you're using or how the logs are shipped to it but perhaps you could devise a way to intercept the pod log output and scrape/redact the information you want before sending it to the logging system? Or intercept the log info before it's inputted to the logging system

Or capture the network packets to/from the pod using tcpdump or Wireshark and don't enable http2debug...

[leasonliu]

1. Do you know of any performance impact / operational risk from enabling GODEBUG=http2debug=2 on lighthouse-agent? We’re worried about the volume/CPU/log-pipeline overhead and don’t want to leave it enabled for long unless you recommend it.

I've never used the flag but, as the name suggests, it's for debugging so I wouldn't leave it enabled permanently.

2. We need to validate the “stale/broken connection” hypothesis without leaking tokens. One idea we got is to instrument the client with Go’s net/http/httptrace (logging only timings/connection reuse/DNS/dial/TLS/first-byte, and logging success/failure + status code for the health-check request, but never dumping headers/bodies). Would you recommend this approach in Lighthouse/Admiral for diagnosing connection health? If so, which code path would be the right place to hook it (so it observes the same transport/pool the agent uses)?

I've never used the httptrace library but it's worth a try. We use K8s client-go library/API which internally handles the low-level HTTP interactions. However K8s does allow the underlying HTTP transport to be customized via the rest.Config. httptrace hooks in via a request’s context.Context so you could do something like this (in lighthouse agent main):

type CustomTransport struct {
    wrapped http.RoundTripper
}

func (t * CustomTransport) RoundTrip(req *http.Request) (*http.Response, error) {
    trace := &httptrace.ClientTrace{
		...
     }
	
    return t.wrapped.RoundTrip(req.WithContext(httptrace.WithClientTrace(req.Context(), trace)))
}
...

config, err := clientcmd.BuildConfigFromFlags(...)

config.Wrap(func(rt http.RoundTripper) http.RoundTripper {
    return &CustomTransport{wrapped: rt}
})

3. If upstream changes aren’t available quickly: what’s your recommended way to fork Admiral with minimal disruption? Our goal would be to avoid “heaps of changes” across Submariner references (We already forked lighthouse and operator repos and made patches to them). Any high-level guidance on the cleanest workflow would be appreciated.

Using the above approach you wouldn't need to fork Admiral and you're already forking lighthouse.

On a side note, I don't know why you're already forking lighthouse and operator but, of course, this is open source and we welcome contributions so I would encourage that if you're customizations would be generally useful.

[tpantelis]

@leasonliu Sorry I meant to reply to your comment but I accidentally edited it.

[dfarrell07]

Contributions welcome, we'd be interested in seeing your changes/patches.