Slow performance when merging cookies

[gyula-lakatos]

Hi everyone,

I recently started to pick up Python after 10 years of Java development. Please don't eat me for breakfast. :)

I noticed some performance issues regarding how cookies are being handled in httpx.

httpx._client.BaseClient._merge_cookies calls into the cookiejar here:

        if cookies or self.cookies:
            merged_cookies = Cookies(self.cookies)
            merged_cookies.update(cookies)
            return merged_cookies
        return cookies

As far as I see the call is caused by the if cookies or self.cookies: line, but my Python is not good enough to figure out why it wants to iterate over the whole thing. Somehow the call will end up in http.cookiejar.deepvalues and that method will use up a crapload of CPU processing power.

We do not set any cookies on 95% of the requests (only at redirects by httpx), so I see no reason why this method is even called. I mean if both client.cookies and the request cookies should be empty then why does the app call into the http.cookiejar.deepvalues? I think it would be a lot faster if the app would return self.cookies if cookies are None or cookies if self.cookies are None. If both of them contains something, then do the existing logic. It might be a good idea to look into why the existing logic calls into http.cookiejar.deepvalues as well, because that might be unnecessary too.

I dropped a performance snapshot to illustrate the problem:

Here deepvalues takes up almost 17% of the CPU time.

[zanieb]

Thanks for the detailed post!

Can you reproduce by calling bool(client.cookies)? If you put a breakpoint after the if statement is it hit? What is the value of self.cookies there?

[gyula-lakatos]

Thanks for the quick reply.

It is eventually hit, but the bool call triggers a deepvalues call. This only happens when either cookies or self.cookies is not None.

httpx._models.Cookies.__bool__:

    def __bool__(self) -> bool:
        for _ in self.jar:
            return True
        return False

This might happen because http.cookiejar.CookieJar has it's __iter__ implemented like this:

    def __iter__(self):
        return deepvalues(self._cookies)

This is just an "educated" guess on my part though.

[zanieb]

Wow that implementation in the standard library is kind of insane. Your diagnosis looks correct without me investigating further. I'd suggest a bug report upstream.

I imagine we can avoid calling the __bool__ and __iter__ methods in the meantime by checking the length explicitly or something.

Do you have a simple MRE that we can use to check for a performance improvement?

[gyula-lakatos]

I created an issue here #2875. Yes, it is kinda an "unusual" behaviour let's just say that.

Unfortunately, I don't have a small example, but just by setting 5-10 cookies on either the request or the client could work. I used a get request, but I think all of them are affected. Calling a local endpoint should be sufficient as well. If you still fail to reproduce the problem feel free to ping me. I'm working on a quite complex distributed crawler so getting a small example is not possible just with some copy+paste. If it is needed I can create a small script that reproduces the problem.

[zanieb]

Ah I meant I think we should consider reporting this as a performance bug in Python.

A small script that shows bool(client.cookies) with and without the deepvalues in the standard library would be ideal — another maintainer may get to it to but the more you can help the sooner it'll be addressed.

[gyula-lakatos]

Ouch, I see. Do you know how quickly things are being fixed in Python? Also I think this could cause backwards incompatibility issues? I'm happy to report it anyways.

Do you know any workarounds that could fix this in the meanwhile? The issue gets worse and worse while the crawler is running, until a certain point where half of the CPU usage goes into calculating the deepvalues function. Is there a way in httpx to do not accept cookies while following redirects?