Object dispatch not working

[thorwhalen]

I may be doing something wrong, but can't get the object dispatching to work
(see Mapping python collections builtin methods to REST API patterns).

See the code in py2http/examples/object_dispatch.py:

backend_store = {
    'some_key': b'some_value',
    'some_other_key': b'some_other_value',
}

handlers = [
    dict(
        endpoint=backend_store,
        name='backend_store',
        attr_names=['__iter__', '__getitem__', '__setitem__'],
    ),
]

from py2http import mk_app

app = mk_app(handlers, publish_openapi=True, publish_swagger=True)


if __name__ == '__main__':
    app.run()

Running this:

python py2http/examples/object_dispatch.py
Bottle v0.12.25 server starting up (using WSGIRefServer())...
Listening on http://127.0.0.1:8080/
Hit Ctrl-C to quit.

And trying it with

import requests

requests.post(
    'http://localhost:3030/backend_store', 
    json={
        "_attr_name": "__iter__",
        "kwargs": {}
    }
)

I get:

ConnectionError: HTTPConnectionPool(host='localhost', port=3030): Max retries exceeded with url: [/backend_store](https://file+.vscode-resource.vscode-cdn.net/backend_store) (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x109025ed0>: Failed to establish a new connection: [Errno 61] Connection refused'))

Or:

curl -X 'POST' \
  'http://localhost:3030/backend_store' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "_attr_name": "__iter__",
  "kwargs": {}
}'

I get

curl: (7) Failed to connect to localhost port 3030 after 4 ms: Couldn't connect to server

Yet the server seems to be responding:

python py2http/examples/object_dispatch.py
Bottle v0.12.25 server starting up (using WSGIRefServer())...
Listening on http://127.0.0.1:8080/
Hit Ctrl-C to quit.

127.0.0.1 - - [22/Feb/2024 08:27:40] "GET /swagger HTTP/1.1" 200 1173
127.0.0.1 - - [22/Feb/2024 08:27:40] "GET /swagger/swagger.json HTTP/1.1" 200 553
127.0.0.1 - - [22/Feb/2024 08:38:48] "GET /swagger HTTP/1.1" 200 1173
127.0.0.1 - - [22/Feb/2024 08:38:48] "GET /swagger/swagger.json HTTP/1.1" 200 553

Swagger's message is:

Failed to fetch.
Possible Reasons:

CORS
Network Failure
URL scheme must be "http" or "https" for CORS request.

[thorwhalen]

The problem may not be with object dispatch but something else, since web services that worked before don't any more.

For example, getting this:

Bottle v0.12.25 server starting up (using GunicornServer(certfile=None, keyfile=None))...
Listening on http://localhost:3030/
Hit Ctrl-C to quit.

[2024-02-22 08:50:37 +0100] [29952] [INFO] Starting gunicorn 21.2.0
[2024-02-22 08:50:37 +0100] [29952] [INFO] Listening at: http://127.0.0.1:3030 (29952)
[2024-02-22 08:50:37 +0100] [29952] [INFO] Using worker: sync
[2024-02-22 08:50:37 +0100] [30000] [INFO] Booting worker with pid: 30000
[2024-02-22 08:51:24 +0100] [29952] [CRITICAL] WORKER TIMEOUT (pid:30000)
[2024-02-22 08:51:24 +0100] [30000] [INFO] Worker exiting (pid: 30000)
[2024-02-22 08:51:25 +0100] [29952] [ERROR] Worker (pid:30000) exited with code 1
[2024-02-22 08:51:25 +0100] [29952] [ERROR] Worker (pid:30000) exited with code 1.
[2024-02-22 08:51:25 +0100] [30003] [INFO] Booting worker with pid: 30003

with another one.

[thorwhalen]

Okay, so there's layers of things happening here, some that I don't understand.

What I don't understand

unpredictable backend server choice

Sometimes py2http chooses to use WSGIRefServer instead of GunicornServer -- unpredictably, for me.

Bottle v0.12.25 server starting up (using WSGIRefServer())...
Listening on http://127.0.0.1:8080/
Hit Ctrl-C to quit.

Swagger works... sometimes

Swagger works... and the doesn't anymore (I start getting those "CORS or Network failure or URL scheme" errors, and then they stick, even after relaunching the server.

Now, even while swagger doesn't work, I still get the curl and requests to work.

This is not always the case, but that's because Swagger's message is catchall. The errors I was getting before, I was getting them simply because there was a network error (see later).

What I DO understand

Swagger curl pointing to wrong url

So, when (for reasons I do not know) py2http chooses to use WSGIRefServer, it listens on 8080.
To go to swagger, I also use 8080 -- but when I execute, swagger displays a curl that uses 3030.
So of course, it fails with the catchall (unhelpful) error message.

Argument type

I used:

  "_attr_name": "__iter__",
  "kwargs": {}

because it's the example I found, and I assumed that one was supposed to specify all arguments in a kwargs dict,
which is empty here since __iter__ has no arguments. This is not how it works.
Instead, I should be specifying the keyword arguments at the same level as the "_attr_name" field.
So I should be doing:

url = 'http://localhost:3030/backend_store'
add_args = {
  "_attr_name": '__iter__',
}
requests.post(url, json=add_args).json()

or

curl -X 'POST' \
  'http://localhost:3030/backend_store' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "_attr_name": "__iter__"
}'

Argument type

But we still have a problem.
We'd like to do

url = 'http://localhost:3030/backend_store'
add_args = {
  "_attr_name": '__getitem__',
  "key": 'some_key'
}
requests.post(url, json=add_args).json()

but actually dict.__getitem__ has only one, positional-only, argument.

So: Handling position-only arguments issue.

Output types

There were two more problems with py2http/examples/object_dispatch.py code.
One is that the values of the backend_store were bytes: This causes problems when doing a __getitem__ since it's not JSON serializable.
The second is that the __iter__ return type is a generator, also not serializable.

Both of these things can be handled at the egress level (output_trans), but since we also have the problem of the positional-only, we can also just wrap the dict.
So we can change our previous code to this new code:

from typing import MutableMapping

class BackendStore(MutableMapping):
    """MutableMapping wrapper that assumes that values are given as strings,
    but stored as bytes"""

    def __init__(self, store):
        self.store = store

    def __iter__(self):
        return list(self.store)

    def __getitem__(self, key: str) -> str:
        return self.store[key].decode()

    def __setitem__(self, key: str, value: str):
        self.store[key] = value.encode()

    def __len__(self) -> int:
        return len(self.store)

    def __delitem__(self, key: str):
        del self.store[key]


backend_store = BackendStore(
    {
        'some_key': b'some_value',
        'some_other_key': b'some_other_value',
    }
)

handlers = [
    dict(
        endpoint=backend_store,
        name='backend_store',
        attr_names=['__iter__', '__getitem__', '__setitem__'],
    ),
]

from py2http import mk_app

app = mk_app(handlers, publish_openapi=True, publish_swagger=True)

if __name__ == '__main__':
    app.run()

Of course, if not explicitly specified, we still can't predict what the root url will be (depends on the unpredictable server that will be chosen), but here's some code to test the service above:

def test_service(rooturl='http://localhost:8080'):
    import requests

    url = f'{rooturl}/backend_store'

    add_args = {
        "_attr_name": '__iter__',
    }
    assert requests.post(url, json=add_args).json() == ['some_key', 'some_other_key']

    add_args = {"_attr_name": '__getitem__', "key": 'some_key'}
    assert requests.post(url, json=add_args).json() == 'some_value'