Support retries with Retry-After header

[kasium]

Description

Some docker/image registries have rate limiting in place, so pushing big images might lead to a 429 response. Normally, a Retry-After header is sent in addition.

Feature Request: buildx should respect this header and retry the failed operation after the given time eventually also giving up if e.g. failed 3 times in a row

[Pnkcaht]

@jsternberg Hi friend sorry to tag you, can I resolve the issues? I was going to do a cleanup like I did on the MCP gateway and cagent, so I prefer to ask first so I can make fewer PRs and some specific ones if you want.
Anyway thank you :)

[thaJeztah]

I'm not 100% sure if this could be resolved in buildx itself; pulling of images is done on the daemon side (BuildKit or Docker Engine), and may depend on code provided by containerd.

I'm also curious what the expectations are; when hitting rate limits (e.g. on Docker Hub), such rate limits would often have a long "retry after"; is the expectation for a docker buildx build to be blocked for (say) an hour, or more, for it to retry?

[thaJeztah]

FWIW, there's some tickets in containerd; at least this one (for pushing images) has some WIP pull request;

• pusher: a series of retries doesn't help with 429 TooManyRequests containerd/containerd#10447
• remote: implement backoff strategy for 429 errors containerd/containerd#12613

But there may be other related tickets / discussions.

[timja]

I'm also curious what the expectations are; when hitting rate limits (e.g. on Docker Hub), such rate limits would often have a long "retry after"; is the expectation for a docker buildx build to be blocked for (say) an hour, or more, for it to retry?

I would expect it to respect the retry after, but log what is going on.

FWIW as of a few weeks ago docker hub seems to have been rate limiting us on pushes...