Intraday code gets confused by daylight savings timezone cutover

[thirtythreeforty]

• Tiingo Python version: 0.5.0
• Python version: 3.9
• Operating System: Debian 11

Description

Was trying to download IEX intraday by walking backward through dates.

What I Did

In [111]: q = c.get_dataframe('AAPL', endDate='2021-04-09T12:16:00.000000000', startDate='2021-02-08 12:16:00', fmt='csv', frequency='1Min')
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-111-384784fbb5a0> in <module>
----> 1 q = c.get_dataframe('AAPL', endDate='2021-04-09T12:16:00.000000000', startDate='2021-02-08 12:16:00', fmt='csv', frequency='1Min')

~/Downloads/ib-historical-data/.venv/lib/python3.9/site-packages/tiingo/api.py in get_dataframe(self, tickers, startDate, endDate, metric_name, frequency, fmt)
    292         if pandas_is_installed:
    293             if type(tickers) is str:
--> 294                 prices = self._request_pandas(
    295                     ticker=tickers, params=params, metric_name=metric_name)
    296             else:

~/Downloads/ib-historical-data/.venv/lib/python3.9/site-packages/tiingo/api.py in _request_pandas(self, ticker, metric_name, params)
    203         # Localize to UTC to ensure equivalence between data returned in json format and
    204         # csv format. Tiingo daily data requested in csv format does not include a timezone.
--> 205         if prices.index.tz is None:
    206             prices.index = prices.index.tz_localize('UTC')
    207 

AttributeError: 'Index' object has no attribute 'tz'

This happens because when the code tries to convert the index to_datetime, pandas returns a plain Index:

tiingo-python/tiingo/api.py

Lines 212 to 217 in 6b032a0

	prices.index = pd.to_datetime(prices.index)
	# Localize to UTC to ensure equivalence between data returned in json format and
	# csv format. Tiingo daily data requested in csv format does not include a timezone.
	if prices.index.tz is None:
	prices.index = prices.index.tz_localize("UTC")

...presumably because it can't work out the correct timezone of the data, since it has the DST cutover, sample:

2021-03-12 15:55:00-05:00,120.78,121.085,120.78,120.94,22801.0
2021-03-12 16:00:00-05:00,120.94,120.94,120.94,120.94,0.0
2021-03-15 09:30:00-04:00,121.4,121.49,120.425,120.9,39581.0
2021-03-15 09:35:00-04:00,120.885,121.33,120.83,120.96,23986.0

[hydrosquall]

Hi @thirtythreeforty , thanks for the detailed report.

In the short-term, I would advise using the non-dataframe oriented API to download the raw data and parse them into datetimes using your own preferred logic, since that API is simpler, and doesn't attempt to do any type related data coercion.

client.get_ticker_price('AAPL', ...)

[cavart28]

Possibly related is the following interesting behavior, unexpected to me. By querying an interval of 2 minutes, I would expect 2 (or 3) minutes but for some reason I get a dataframe with 10 rows, starting with the correct date/time but ending with a time of 15:59

start = '2023-03-15 15:50:00'
end = '2023-03-15 15:52:00'

df =  tiingo_client.get_dataframe(
        tickers='QQQ', frequency='1min', startDate=start, endDate=end
    )

df