Bitbucket Rate Limit Handling

[fullben]

In a project of mine, where I'm using this library, I ended up tripping the Bitbucket request rate limit configured for the user I am using for authenticating with a Bitbucket server.

This happened when calling various paged API endpoints in quick succession. As a result, the Page objects returned by the library methods would suddenly contain Error objects containing information about some JSON parsing issue.

I found determining the source of my issue rather difficult, as the Error objects provided by the Bitbucket REST library only mentioned issues with parsing the response. Only upon checking the Bitbucket server rate limiting options and enabling logging for jclouds, I was able to understand the problem, as a corresponding 429 response will be logged by jclouds whenever the parse issue occurs.

I also understand that modifying the rate limit settings for my Bitbucket user can mitigate this issue, but at the same time I feel like the library should be able to deal a bit more graceful with this scenario.

Expected Behavior

I would either expect there to be a configuration option for dealing with rate limiting (maybe I missed it?), or for the Error object to at least contain some more meaningful information.

Current Behavior

When a rate limit is tripped by the library, the Bitbucket remote responds with a 429 and a JSON object the library does not seem to understand. The corresponding Error object will only provide information about JSON parsing issues:

exceptionName: "com.google.gson.JsonSyntaxException"
message: "Failed to parse output: message=com.google.gson.stream.MalformedJsonException: Use JsonReader.setLenient(true) to accept malformed JSON at line 1 column 9 path $"

This in turn makes it impossible for me to programmatically identify this error as a rate limit issue.

Context

Calling various endpoints in quick succession for acquiring the permission information for all Bitbucket projects and their repositories on a Bitbucket server instance.

Your Environment

Bitbucket REST client lib version 3.0.2
Bitbucket server instance version 7.6.17

[cdancy]

@fullben aahhh ... yeah I've never hammered away on Bitbucket before to trigger any rate-limits. Do those exist across the board for all endpoints? I can put in a fix for this to catch the rate-limit response and return something proper instead of the default "json parse exception" that you see which is admittedly less than ideal.

[fullben]

@cdancy rate limiting in Bitbucket is user-based, as far as I can tell (see the link from my original post). So I'm assuming it's consistent behavior across all endpoints.
A better error description would be great. But tbh, I was kinda hoping jclouds or the lib itself had some rate limit error handling feature I simply just couldn't find... :)

[cdancy]

@fullben we could certainly wire something up and there might be something we can leverage ... I've just never done so. Something like "retry on http code 429" and allow for some configurability.

[fullben]

@cdancy I did some digging in the jclouds code... There is this property PROPERTY_MAX_RATE_LIMIT_WAIT (in the Constants). in the jclouds core lib, this property is only utilized in one class, the RateLimitRetryHandler, which is an abstract class. There is no implementation for this class in the core jclouds lib...
I have no clue about jclouds or how your lib uses it, but maybe this is be something that could be leveraged for dealing with Bitbucket rate limiting more gracefully?

[fullben]

@cdancy bump :)