[Enhancement/Bug Fix] Tiingo Backend Uses JSON Which Conflicts With Python Boolean Calls

[GenusGeoff]

• Tiingo Python version: 0.13
• Python version: 3.8.8
• Operating System: Windows/Ubuntu 18.04

Description

Made a call to IEX Intraday data enabling the "&afterHours=True" parameter. The problem is with the Pythonic need for True and False to be capitalized which conflicts with JSONs need to use "true" and "false" lowercase.

I expected the API call to return fruitful but instead returned None.

What I Did

I suggest that we utilize a JSON.dump(url_parameters) going forward to be better able to adapt to changes in the Tiingo API. A simple function could be created that is called before returning the URL string to the requests library.

Please see an excerpt from an email exchange from @tiingo:

Interesting little 'bug' or 'feature': When getting intraday IEX quotes and choosing afterHours=true or afterHours=false the boolean MUST BE lowercase which is an interesting little incompatibility with Python that likes Booleans to be either True or False. So, I can workaround with using an encapsulated 'true' or 'false' but as you continue making your API v2, this might be something worth examining.

@tiingo response:

This behavior is intentional

REST APIs typically take data in JSON format - the lowercase true/false is a function of following this spec (JSON booleans are lowercase).

In the backend, the json is parsed into Python using a standard json parsing library.

This is a function of following standard web practices - if you want to send a request to the API, you could use a standard json encoding library (ujson, simplejson for example). then do json.dumps(pythonObj) to get the JSON representation

I believe it's only a matter of time, as in my attempted enhancement, before the V2 API or later functionality exposes this "bug" in the code.

[GenusGeoff]

@hydrosquall I'm not sure if you saw this note or not.

[hydrosquall]

Hi @GenusGeoff , thanks for the ping, I missed this previously. I don't have time to look into this right now, but I can give a pointer to anyone who might want to look into this:

1. How the params keyword argument works for requests.requests

https://docs.python-requests.org/en/master/user/quickstart/#passing-parameters-in-urls

2. This is the common REST client used by all the endpoints.

tiingo-python/tiingo/restclient.py

Lines 47 to 50 in b4b42b9

	resp = self._session.request(method,
	'{}/{}'.format(self._base_url, url),
	headers=self._headers,
	**kwargs)

One could take the params object in the kwargs, and if any of the values of the key:value pairs the a boolean True or False convert them to the strings true or false .

The JSON.dumps solution may not work for us directly since params accepts a Python object rather than a string, but it's good to know this is an option for other scenarios.

[GenusGeoff]

Hi @GenusGeoff , thanks for the ping, I missed this previously. I don't have time to look into this right now, but I can give a pointer to anyone who might want to look into this:

1. How the params keyword argument works for requests.requests

https://docs.python-requests.org/en/master/user/quickstart/#passing-parameters-in-urls

I did a test with Requests:

Python 3.8.10 (default, May 19 2021, 13:12:57) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.

import requests
requests.version
'2.25.1'
payload = {'myBoolean': True, 'myOtherBoolean': False, 'regularKey': 'AAPL'}
r = requests.get('https://httpbin.org/get', params=payload)
print(r.url)
https://httpbin.org/get?myBoolean=True&myOtherBoolean=False&regularKey=AAPL

At least with Version 2.25.1 of the Requests module, the keys and pairs are entered directly. I'll look to see if there's a way to modify such behavior internal to Requests as that would be the best answer. If that isn't possible then we'll need a hacky workaround to search if any of the values are Python Boolean and convert them to JSON Boolean.

[GenusGeoff]

I'm unsure if this is an efficient answer, but a simple answer is to loop through the params dictionary keys and values and make a simple substitution, e.g.

import requests
payload = {'key1': 'value1', 'key2': 'value2', 'boolean': True}
for key in payload:
  if payload[key] is True:
    payload[key] = 'true'
  if payload[key] is False:
    payload[key] = 'false'

Clearly, this can be made into a lambda expression or written in any of many ways to be far more efficient, but I wrote it in a very verbose format to clearly illustrate the pseudo-code.

This is the output from printing a loop of the keys and values:

key1 value1
key2 value2
boolean true

@hydrosquall

[hydrosquall]

@GenusGeoff , this looks pretty good, thanks for looking into this and finding an initial solution. I think if you want to try introducing it, it could be done through the JSON.dump method, which will ensure we handle JSON serializable data well in general.

tiingo-python/tiingo/restclient.py

Lines 47 to 50 in 11b8e45

	resp = self._session.request(method,
	'{}/{}'.format(self._base_url, url),
	headers=self._headers,
	**kwargs)

import json

def format_params(params):
     new_params = {}
     for key in params:
         new_params[key] = json.dump(params[key])

     return new_params
    

# meanwhile, in restclient.py

formatted_params = format_params(params)
requests.request(# previous code,
                              params=formatted_params,
                            
)