ssl.SSLWantReadError / httpx.ReadTimeout appearing randomly #2941
Replies: 4 comments 21 replies
-
|
Ah correction, it doesn't need to be batched. This still reproduces the issue: async def main() -> None:
try:
while True:
for _ in range(20):
await make_request()
print(f"total reqs {total}")
finally:
print(f"total reqs {total}") |
Beta Was this translation helpful? Give feedback.
-
|
Looks like redis-py had a similar issue and fixed it by ignoring the ssl.SSLWantReadError (I think) redis/redis-py@e897c17 Related report here: #2615 (comment) Found a reference to this error in another library: dpallot/simple-websocket-server#100, the solution in that PR was to retry... |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
Another test run. The 2263rd request timed out, and this time it was definitely an issue with the server not sending any response. The randomness of the problem makes debugging so hard since it takes an enormous amount of time to trigger it. |
Beta Was this translation helpful? Give feedback.
-
|
Another run, this time it timed out at the 88th request already. I had increased the timeout to 30 seconds, and the wireshark output clearly shows that there was no response from the server in that time. The server did promptly respond to the connection close notification (the |
Beta Was this translation helpful? Give feedback.
-
|
Did one more run with |
Beta Was this translation helpful? Give feedback.
-
|
I now believe some of my findings were in error, as it seems to alternate between the two API ip addresses and I was only listening to traffic to one of them. |
Beta Was this translation helpful? Give feedback.
-
|
So, to anyone else reading these, my conclusion from this effort was that I couldn't find anything wrong with the client side, and all signs pointed to a random failure on the back-end, sometimes manifesting as timeouts, sometimes a 502 HTTP error. To check whether aiohttp fared any better than httpx, I created two scripts: aiohttpimport asyncio
import os
from itertools import count
import aiohttp
from aiohttp import ClientResponseError
api_key = os.environ.get("OPENAI_API_KEY")
if not api_key:
raise ValueError("Expected OPENAI_API_KEY environment variable to be present")
async def make_request(client) -> None:
async with client.post(
"/v1/chat/completions",
json=dict(
messages=[
{
"role": "user",
"content": "Say this is a test",
}
],
model="gpt-3.5-turbo",
),
) as response:
response.raise_for_status()
await response.read()
async def main() -> None:
timeouts = []
for round in count(1):
async with aiohttp.ClientSession(
base_url="https://api.openai.com",
headers={"Authorization": f"Bearer {api_key}"},
timeout=aiohttp.ClientTimeout(total=30, connect=5),
) as client:
for iteration in count(1):
print(f"\rRound {round}, iteration {iteration}...", end="", flush=True)
try:
await make_request(client)
except ClientResponseError as exc:
print("response error:", exc)
except TimeoutError:
print(f"timeout")
timeouts.append(iteration)
print(
f"Average iterations until timeout: {sum(timeouts) / (round)}"
)
break
else:
print("ok", end=" ", flush=True)
asyncio.run(main())httpximport os
import asyncio
from itertools import count
import httpx
from aiohttp import ClientResponseError
from httpx import ReadTimeout
api_key = os.environ.get("OPENAI_API_KEY")
if not api_key:
raise ValueError("Expected OPENAI_API_KEY environment variable to be present")
async def make_request(client) -> None:
response = await client.post(
"/chat/completions",
json=dict(
messages=[
{
"role": "user",
"content": "Say this is a test",
}
],
model="gpt-3.5-turbo",
),
)
response.raise_for_status()
async def main() -> None:
timeouts = []
for round in count(1):
async with httpx.AsyncClient(
base_url="https://api.openai.com/v1",
headers={"Authorization": f"Bearer {api_key}"},
timeout=httpx.Timeout(
timeout=30.0,
connect=5.0,
),
) as client:
for iteration in count(1):
print(f"\rRound {round}, iteration {iteration}...", end="", flush=True)
try:
await make_request(client)
except ClientResponseError as exc:
print("response error:", exc)
except ReadTimeout:
print(f"timeout")
timeouts.append(iteration)
print(
f"Average iterations until timeout: {sum(timeouts) / (round)}"
)
break
else:
print("ok", end=" ", flush=True)
asyncio.run(main()) |
Beta Was this translation helpful? Give feedback.
-
|
So, what is the solution here? Should we add retry to take care of this? |
Beta Was this translation helpful? Give feedback.
-
|
Interesting question. If we've got a server failure at the point of establishing the SSL connection, then I think it is safe for us to retry. Tho... perhaps a good starting point would be to make sure we're able to replicate locally / demo the existing behavior with a test case or replicable code sample. It might be easiest to replicate this type of failure case using raw socket code, eg... #1927 (comment) |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
When continually making requests to the OpenAI API, it will eventually fail with a
httpx.ReadTimeouterror that is caused by anssl.SSLWantReadErrorerror.Here's an example script:
This eventually fails for me from anywhere between 1 and 1.3k requests.
Stack trace
For context, I'm trying to reproduce an issue reported by users of the
openai-pythonlibrary that useshttpxunder the hood. openai/openai-python#769This also happens when:
with httpx.Client(limits=httpx.Limits(max_connections=1))v0.23.0orv0.18.0verify=FalseI haven't verified that this independently of the OpenAI API yet so it could be an API issue but thats very unlikely as users have reported downgrading to an older version that doesn't use httpx which fixes the issue. I have also repeatedly made requests using
node-fetchwhich succeeded.I have attempted to reproduce this without using the OpenAI API under the following conditions, all of which did not successfully reproduce the issue:
respxto mock the responseprismmock server under http (not https)Any ideas?
Beta Was this translation helpful? Give feedback.
All reactions