Potential SSL errors when accessing urls with underscores in

[teodoryantcheff]

A Docker container having 2 DNS names

'-' and '_' in the name is the only difference.

Started like this

docker run -d --restart unless-stopped                             \
        --env-file $(pwd)/.env.prod                                \
        --name test-svc                                            \
        --net nginx-proxy                                          \
        -e VIRTUAL_HOST=test-svc.gosgt.io,test_svc.gosgt.io        \
        registry.digitalocean.com/xxxxxxx/test-svc

and running a very simple fastspi app

from fastapi import FastAPI
from fastapi.responses import Response

app = FastAPI()

@app.get("/")
async def root():
    return Response(status_code=200, content='ok')

When accessed like this

import httpx

r = httpx.get('https://test-svc.gosgt.io/')
print(r.status_code)

results in

200

and when GET'd with this

import httpx

r = httpx.get('https://test_svc.gosgt.io/')  # <<< underscore in the url
print(r.status_code)

results in this

Traceback (most recent call last):
  File "<...>\venv\Lib\site-packages\httpcore\_exceptions.py", line 10, in map_exceptions
    yield
  File "<...>\venv\Lib\site-packages\httpcore\_backends\sync.py", line 174, in start_tls
    raise exc
  File "<...>\venv\Lib\site-packages\httpcore\_backends\sync.py", line 169, in start_tls
    sock = ssl_context.wrap_socket(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\ssl.py", line 517, in wrap_socket
    return self.sslsocket_class._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\ssl.py", line 1108, in _create
    self.do_handshake()
  File "C:\Python311\Lib\ssl.py", line 1379, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'test_svc.gosgt.io'. (_ssl.c:1006)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<...>\venv\Lib\site-packages\httpx\_transports\default.py", line 66, in map_httpcore_exceptions
    yield
  File "<...>\venv\Lib\site-packages\httpx\_transports\default.py", line 228, in handle_request
    resp = self._pool.handle_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<...>\venv\Lib\site-packages\httpcore\_sync\connection_pool.py", line 262, in handle_request
    raise exc
  File "<...>\venv\Lib\site-packages\httpcore\_sync\connection_pool.py", line 245, in handle_request
    response = connection.handle_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<...>\venv\Lib\site-packages\httpcore\_sync\connection.py", line 99, in handle_request
    raise exc
  File "<...>\venv\Lib\site-packages\httpcore\_sync\connection.py", line 76, in handle_request
    stream = self._connect(request)
             ^^^^^^^^^^^^^^^^^^^^^^
  File "<...>\venv\Lib\site-packages\httpcore\_sync\connection.py", line 156, in _connect
    stream = stream.start_tls(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<...>\venv\Lib\site-packages\httpcore\_backends\sync.py", line 158, in start_tls
    with map_exceptions(exc_map):
  File "C:\Python311\Lib\contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "<...>\venv\Lib\site-packages\httpcore\_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ConnectError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'test_svc.gosgt.io'. (_ssl.c:1006)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<...>\src\test_certs.py", line 6, in <module>
    r = httpx.get('https://test_svc.gosgt.io/')
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<...>\venv\Lib\site-packages\httpx\_api.py", line 189, in get
    return request(
           ^^^^^^^^
  File "<...>\venv\Lib\site-packages\httpx\_api.py", line 100, in request
    return client.request(
           ^^^^^^^^^^^^^^^
  File "<...>\venv\Lib\site-packages\httpx\_client.py", line 814, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<...>\venv\Lib\site-packages\httpx\_client.py", line 901, in send
    response = self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<...>\venv\Lib\site-packages\httpx\_client.py", line 929, in _send_handling_auth
    response = self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<...>\venv\Lib\site-packages\httpx\_client.py", line 966, in _send_handling_redirects
    response = self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<...>\venv\Lib\site-packages\httpx\_client.py", line 1002, in _send_single_request
    response = transport.handle_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<...>\venv\Lib\site-packages\httpx\_transports\default.py", line 227, in handle_request
    with map_httpcore_exceptions():
  File "C:\Python311\Lib\contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "<...>\venv\Lib\site-packages\httpx\_transports\default.py", line 83, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'test_svc.gosgt.io'. (_ssl.c:1006)

Tested doing the same GET requests with curl on a console, aiohttp, and requests -- all work.
httpx spits that SSL error in my face.
Oh, and ran the httpx part from both Windows, wsl2, and Linux. All of them complain about SSL when calling https://test_svc.gosgt.io.

fastapi==0.103.2
gunicorn==21.2.0
h11==0.14.0
httpcore==0.18.0
httpx==0.25.0
idna==3.4
requests==2.31.0
starlette==0.27.0

Python 3.11.5

What am I missing?

[teodoryantcheff]

Both test-svc.gosgt.io and test_svc.gosgt.io are public.
Please help me understand what's wrong.

[karpetrosyan]

Hi! Sorry for the delayed answer.
I can guess the problem is that browsers somehow normalize their sni_hostname in the ssl context. Anyway, you can set the sni_hostname manually.

Try this

import httpx

url = "https://test_svc.gosgt.io/"

client = httpx.Client()

request = client.build_request("GET", url, extensions={"sni_hostname": "test-svc.gosgt.io"})

response = client.send(request)

print(response.text)  # ok

I don't know why the browser automatically changes the sni-hostname; I will double-check our implementation.

[teodoryantcheff]

@karpetrosyan Thank you for the reply!

Yes, your code snippet does work correctly -- printing a ok.

Here's the full set of combinations between url and sni_hostname (note dashes and underscores):

url       sni_hostname   result 
test_svc  test_svc       ...certificate is not valid for 'test_svc.gosgt.io'. (_ssl.c:1006)
test_svc  test-svc       ok
test-svc  test_svc       ...certificate is not valid for 'test_svc.gosgt.io'. (_ssl.c:1006)
test-svc  test-svc       ok

[teodoryantcheff]

Should it really be throwing exceptions though?
This "service" is covered by a wildcard certificate -- *.gosgt.io, so at least as far as my understanding goes, using it under both subdomains (test_svc and test-svc) should not result in errors and exceptions.

And as pointed out in the original post, only httpx seems to not like doing httpx.get('https://test_svc.gosgt.io/').
requests, aiohttp, and console curl do work.

Am I wrong in my thinking?