[Bug] Docker Labels Race Condition

[ElioDiNino]

Describe the bug
When Docker labels are used to control socket-proxy access, if an application boots up and attempts to access the proxy right away, it will be blocked before the proxy reads the labels and applies the correct policy.

To Reproduce
Steps to reproduce the behavior:

1. Deploy a Beszel agent similar to my configuration
2. Observe the Beszel agent's attempt to access the socket and get blocked (based on forbidden IP) despite the request abiding by the regex
3. Observe the socket proxy update with the correct access policy too late

Expected behavior
Requests made at container creation/restart time should not get blocked and the access policy should be queried/fetched first.

Desktop (please complete the following information):

• OS: Debian Linux
• Version: 1.11.0

Additional context

In the logs below, you can see me doing a mix of a) applying new regex policies to the Beszel agent container and b) restarting it, which both result in the following sequence:

1. The allowlist is correctly removed for the container as it disappears
2. The request by the Beszel agent is blocked based on IP (hasn't even gotten to the allowed paths)
3. The allowlist is correctly propagated for the container

Not sure if it's relevant or not, but I run two Beszel agents on the host, and they share a Docker network, whereas the rest of my services using the socket proxy have their own network. Both are having the same issue and is blocking me from getting the Docker statistics I was getting before.

2026-01-19T21:30:17.274260651Z time=2026-01-19T21:30:17.273Z level=INFO msg="starting socket-proxy" version=1.11.0 os=linux arch=arm64 runtime=go1.25.6 URL=github.com/wollomatic/socket-proxy
2026-01-19T21:30:17.274437909Z time=2026-01-19T21:30:17.274Z level=INFO msg="configuration info" socketpath=/var/run/docker.sock listenaddress=0.0.0.0:2375 loglevel=INFO logjson=false allowfrom=127.0.0.1/32 shutdowngracetime=10
2026-01-19T21:30:17.274472702Z time=2026-01-19T21:30:17.274Z level=INFO msg="watchdog disabled"
2026-01-19T21:30:17.274484703Z time=2026-01-19T21:30:17.274Z level=INFO msg="Proxy container name provided" proxycontainername=socket-proxy
2026-01-19T21:30:17.274495411Z time=2026-01-19T21:30:17.274Z level=INFO msg="per-container allowlists enabled!"
2026-01-19T21:30:17.274505620Z time=2026-01-19T21:30:17.274Z level=INFO msg="socket proxy networks detected" socketproxynetworks="[socket-proxy-tinyauth socket-proxy-beszel-agent socket-proxy-caddy socket-proxy-diun socket-proxy-homepage]"
2026-01-19T21:30:17.274517204Z Default request allowlist:
2026-01-19T21:30:17.274527580Z    Method   Regex
2026-01-19T21:30:17.274537580Z    GET      ^(/v1\.\d{1,2})?/(_ping|version)$
2026-01-19T21:30:17.274547164Z    HEAD     ^(/v1\..{1,2})?/_ping$
2026-01-19T21:30:17.274820510Z time=2026-01-19T21:30:17.274Z level=INFO msg="socket-proxy running and listening..."
2026-01-19T21:30:17.279974205Z Request allowlist for c67060cd9dd4 (172.20.0.3):
2026-01-19T21:30:17.280028833Z    Method   Regex
2026-01-19T21:30:17.280034625Z    HEAD     ^(/v1\.\d{1,2})?/_ping$
2026-01-19T21:30:17.280039542Z    GET      ^(/v1\.\d{1,2})?/(info|containers/json(\?\S+)?|events(\?\S+)?|networks(/\w+)?)$
2026-01-19T21:30:17.280044375Z Request allowlist for ff712e71df1c (172.18.0.2):
2026-01-19T21:30:17.280048667Z    Method   Regex
2026-01-19T21:30:17.280052709Z    GET      ^(/v1\.\d{1,2})?/(version|containers/json(\?\S+)?|images/[\w./:+-]+/json)$
2026-01-19T21:30:17.280057042Z    HEAD     ^(/v1\.\d{1,2})?/_ping$
2026-01-19T21:30:17.280060876Z Request allowlist for a7fef33ed3e2 (172.30.0.2):
2026-01-19T21:30:17.280064709Z    Method   Regex
2026-01-19T21:30:17.280068209Z    HEAD     ^(/v1\.\d{1,2})?/_ping$
2026-01-19T21:30:17.280071960Z    GET      ^(/v1\.\d{1,2})?/containers/(json|[\w.-]+/json)(\?\S+)?$
2026-01-19T21:30:17.280076043Z Request allowlist for ab99a9d0a726 (172.19.0.3):
2026-01-19T21:30:17.280079793Z    Method   Regex
2026-01-19T21:30:17.280083419Z    GET      ^(/v1\.\d{1,2})?/containers/(json|[\w.-]+/(json|stats))(\?\S+)?$
2026-01-22T06:29:25.148084580Z time=2026-01-22T06:29:25.143Z level=INFO msg="removed allowlist for container" id=ab99a9d0a726 ip=172.19.0.3
2026-01-22T06:29:26.311599748Z time=2026-01-22T06:29:26.310Z level=WARN msg="blocked request" reason="forbidden IP" method=GET URL=/version client=172.19.0.3:41356 response=403
2026-01-22T06:29:26.316413183Z Request allowlist for 9ed3b64d531e (172.19.0.3):
2026-01-22T06:29:26.316513312Z    Method   Regex
2026-01-22T06:29:26.316530563Z    GET      ^(/v1\.\d{1,2})?/(info|containers/(json|[\w.-]+/(json|stats))(\?\S+)?)$
2026-01-22T06:31:53.171126074Z time=2026-01-22T06:31:53.170Z level=INFO msg="removed allowlist for container" id=9ed3b64d531e ip=172.19.0.3
2026-01-22T06:31:53.558768071Z time=2026-01-22T06:31:53.558Z level=WARN msg="blocked request" reason="forbidden IP" method=GET URL=/version client=172.19.0.3:51192 response=403
2026-01-22T06:31:53.566160567Z Request allowlist for 9ed3b64d531e (172.19.0.3):
2026-01-22T06:31:53.566238987Z    Method   Regex
2026-01-22T06:31:53.566247779Z    GET      ^(/v1\.\d{1,2})?/(info|containers/(json|[\w.-]+/(json|stats))(\?\S+)?)$
2026-01-22T06:31:53.568447657Z Request allowlist for 9ed3b64d531e (172.19.0.3):
2026-01-22T06:31:53.568467449Z    Method   Regex
2026-01-22T06:31:53.568475199Z    GET      ^(/v1\.\d{1,2})?/(info|containers/(json|[\w.-]+/(json|stats))(\?\S+)?)$
2026-01-22T06:32:52.051984892Z time=2026-01-22T06:32:52.051Z level=INFO msg="removed allowlist for container" id=9ed3b64d531e ip=172.19.0.3
2026-01-22T06:32:53.204537656Z time=2026-01-22T06:32:53.204Z level=WARN msg="blocked request" reason="forbidden IP" method=GET URL=/version client=172.19.0.3:59116 response=403
2026-01-22T06:32:53.218934970Z Request allowlist for c5f83da70ca7 (172.19.0.3):
2026-01-22T06:32:53.219005472Z    Method   Regex
2026-01-22T06:32:53.219013806Z    GET      ^(/v1\.\d{1,2})?/(version|info|containers/(json|[\w.-]+/(json|stats))(\?\S+)?)$
2026-01-22T06:33:38.873435441Z time=2026-01-22T06:33:38.873Z level=INFO msg="removed allowlist for container" id=c5f83da70ca7 ip=172.19.0.3
2026-01-22T06:33:39.263954237Z time=2026-01-22T06:33:39.263Z level=WARN msg="blocked request" reason="forbidden IP" method=GET URL=/version client=172.19.0.3:34116 response=403
2026-01-22T06:33:39.272622577Z Request allowlist for c5f83da70ca7 (172.19.0.3):
2026-01-22T06:33:39.272711622Z    Method   Regex
2026-01-22T06:33:39.272721206Z    GET      ^(/v1\.\d{1,2})?/(version|info|containers/(json|[\w.-]+/(json|stats))(\?\S+)?)$
2026-01-22T06:33:39.274679408Z Request allowlist for c5f83da70ca7 (172.19.0.3):
2026-01-22T06:33:39.274719576Z    Method   Regex
2026-01-22T06:33:39.274725243Z    GET      ^(/v1\.\d{1,2})?/(version|info|containers/(json|[\w.-]+/(json|stats))(\?\S+)?)$
2026-01-22T06:34:45.990249369Z time=2026-01-22T06:34:45.990Z level=INFO msg="removed allowlist for container" id=c5f83da70ca7 ip=172.19.0.3
2026-01-22T06:34:50.752432138Z time=2026-01-22T06:34:50.752Z level=WARN msg="blocked request" reason="forbidden IP" method=GET URL=/version client=172.19.0.3:49316 response=403
2026-01-22T06:34:50.757763682Z Request allowlist for 2ce4d13e5e89 (172.19.0.3):
2026-01-22T06:34:50.757853602Z    Method   Regex
2026-01-22T06:34:50.757869895Z    GET      ^(/v1\.\d{1,2})?/(version|info|containers/(json|[\w.-]+/(json|stats))(\?\S+)?)$

In terms of ideas for a fix, maybe before blocking a request based on IP, the socket-proxy should do a manual query of that container to see if it has the expected labels instead of relying on the Docker socket event stream to have updated the allowlist in time.

cc: @amanda-wee

[amanda-wee]

I encountered a similiar issue with Beszel's /version check last year and contributed retry logic, but it looks like that is still insufficient.

The request by the Beszel agent is blocked based on IP (hasn't even gotten to the allowed paths)

True, and because we don't see the warnings "cannot get valid IP address from request" or "cannot get valid IP address for client allowlist check", this means that it was blocked because it did not qualify based on the -allowfrom parameter or SP_ALLOWFROM environment variable.

maybe before blocking a request based on IP, the socket-proxy should do a manual query of that container to see if it has the expected labels instead of relying on the Docker socket event stream to have updated the allowlist in time

Unfortunately we only know the IP address at that point, and there isn't a Docker Engine API call to directly filter for the container corresponding to an IP address. Nonetheless, this could be possible with one Docker Engine API call to list all the containers in all the networks that the socket-proxy is in, and then iterating over the list until a match is found for the IP address, and then the labels can be found (or not). Is it a good idea to have such a potentially expensive operation when handling the requests? It wouldn't be in the happy path, and we could eliminate this performance penalty for those not using per-container allowlists by only doing this fallback if the proxy container name setting is non-empty.

That said, given that you're allowing /version for the default allowlist, there is a straightforward fix in your case: add beszel-agent to SP_ALLOWFROM. But that wouldn't work if you didn't want a default allowlist, or if some service was trying to use a far more sensitive endpoint that failed due to this race condition, hence it would be a bad idea to include it in a default allowlist.

[ElioDiNino]

I encountered a similiar issue with Beszel's /version check last year and contributed retry logic, but it looks like that is still insufficient.

It looks like the problem here is that the retry doesn't kick in if any response is received (regardless of status code), and in this case, the socket-proxy returns a valid response with status code 403. So, the Beszel agent breaks out of the retry loop on line 594 and then proceeds to attempt to decode the Docker version despite the 403 status and fails on line 608. This is supported by the error I see from Beszel running with debug logs:

2026-01-22T23:58:49.049097073Z 2026/01/22 23:58:49 DEBUG Containers err="invalid character 'F' looking for beginning of value"

Unfortunately we only know the IP address at that point, and there isn't a Docker Engine API call to directly filter for the container corresponding to an IP address. Nonetheless, this could be possible with one Docker Engine API call to list all the containers in all the networks that the socket-proxy is in, and then iterating over the list until a match is found for the IP address, and then the labels can be found (or not).

Ah, great point! I think querying all the containers that the socket proxy shares a network with is the next best solution. Some additional optimization could be done by first eliminating network possibilities based on their subnet and/or IP range found in the IPAM object of the Docker Engine network inspect endpoint.

Is it a good idea to have such a potentially expensive operation when handling the requests? It wouldn't be in the happy path, and we could eliminate this performance penalty for those not using per-container allowlists by only doing this fallback if the proxy container name setting is non-empty.

This is something I was also debating. I think/hope that with my optimization suggestion above, it would not be too much of a penalty, but I agree that it only makes sense to enable this when the proxy container name setting is non-empty.

That said, given that you're allowing /version for the default allowlist, there is a straightforward fix in your case: add beszel-agent to SP_ALLOWFROM. But that wouldn't work if you didn't want a default allowlist, or if some service was trying to use a far more sensitive endpoint that failed due to this race condition, hence it would be a bad idea to include it in a default allowlist.

I will do this for now to get my Docker statistics back, but yeah, I think this is not a generally advisable solution for someone if they are trying to fully use Docker labels for configuration and I hope a solution can be implemented :)