Enable http2py to handle "$ref" in it's openAPI specs

[thorwhalen]

I tried to make a python binder from this openAI openAPI specs but it failed:

import graze
import yaml

# You can use `requests.get` instead of `graze.graze` here:
specs = yaml.safe_load(graze.graze('https://raw.githubusercontent.com/openai/openai-openapi/master/openapi.yaml'))

api = HttpClient(openapi_spec=api_specs)  # fails

@valentin-feron said it was because of the $ref field.

This field needs to be expanded. There are tools for that. Off the top of my head, jsonschema (more function full) and jsonref (lighter -- no deps).

We should make sure http2py can handle the $ref field.

In fact, in general: http2py should be able to make a python binder for any valid OpenAPI spec

From Valentin

I tried to use http2py to consume the OpenAI API's spec at https://raw.githubusercontent.com/openai/openai-openapi/master/openapi.yaml
It is not working because http2py does not support $ref keys in the spec. I tried to use a 3rd party library called openapi_parser

from openapi_parser import parse

path = '/Users/vferon/otosense/sources/notebooks/tools/py2http2py/openai_spec.yml'
openai_api_specs = parse(path)

but I got an error:

ParserError: Schema does not contain 'type' property

Apparently the solution does not support $ref keys either. In order to make it work, we would need to improve our own parser in http2py by replacing any $ref with the subdict it reffers to in the openapi spec object.

[thorwhalen]

Expanded the $ref fields, but it still doesn't seem to work.

import graze
import yaml

# You can use `requests.get` instead of `graze.graze` here:
specs = yaml.safe_load(graze.graze('https://raw.githubusercontent.com/openai/openai-openapi/master/openapi.yaml'))


from functools import partial
import re
from dol.paths import _path_get as path_get  # intending on switching both!

from dol.recipes import search_paths

def string_matching(pattern, p, k, v):
    matches_string = re.compile(pattern).search
    return (
        isinstance(k, str) and matches_string(k)
        or
        isinstance(v, str) and matches_string(v)
    )

it = search_paths(specs, pkv_filt=partial(string_matching, '\$ref'))
t = list(it)
print(f'{len(t)}')
# 53

import jsonref
expanded_specs = jsonref.JsonRef.replace_refs(specs)
print(list(search_paths(expanded_specs, pkv_filt=partial(string_matching, '\$ref'))))
# []


print(t[0])
# ('paths', '/engines', 'get', 'responses', '200', 'content', 'application/json', 'schema', '$ref')


# See NOT expanded:
print(path_get(specs, t[0][:-1]))
# {'$ref': '#/components/schemas/ListEnginesResponse'}

# See expanded
path_get(expanded_specs, t[0][:-1])
# {'type': 'object',
#  'properties': {'object': {'type': 'string'},
#   'data': {'type': 'array',
#    'items': {'title': 'Engine',
#     'properties': {'id': {'type': 'string'},
#      'object': {'type': 'string'},
#      'created': {'type': 'integer', 'nullable': True},
#      'ready': {'type': 'boolean'}},
#     'required': ['id', 'object', 'created', 'ready']}}},
#  'required': ['object', 'data']}

Then, running HttpClient gave me the error:

--> argtype = details['type']
KeyError: 'type'

In this commit
I replaced this getitem (and another related one) with:

            argtype = details.get("type", None)
            pytype = pytype_for_oatype.get(argtype, None)

api = HttpClient(openapi_spec=expanded_specs)

then runs through, but:

from http2py import HttpClient
api = HttpClient(openapi_spec=expanded_specs)
list(x for x in dir(api) if not x.startswith('_'))
# ['auth_type',
#  'base_url',
#  'ensure_login',
#  'handle_request',
#  'init_security',
#  'is_request_method',
#  'login_input_keys',
#  'login_url',
#  'openapi_spec',
#  'receive_login',
#  'refresh_input_keys',
#  'refresh_inputs',
#  'refresh_login',
#  'refresh_url',
#  'register_method',
#  'request_func',
#  'session',
#  'set_header',
#  'set_profile',
#  'title',
#  'verify_cert',
#  'version']