AttributeError on prescribed use of memento_info

[shawnmjones]

This was reported by @jayvdb in https://phabricator.wikimedia.org/T112250:

---

When there is a gateway error in the timegate server, Memento client response doesnt include what we expect, and doesnt raise an exception.

When we were using memento_info.get('mementos').get('closest').get('uri')[0], it failed with an AttributeError:

======================================================================
ERROR: test_newest (tests.weblinkchecker_tests.TestMementoWebCite)
Test WebCite for newest https://google.com.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/jayvdb/pywikibot-core/tests/weblinkchecker_tests.py", line 73, in test_newest
    archivedversion = self._get_archive_url('https://google.com')
  File "/home/travis/build/jayvdb/pywikibot-core/tests/weblinkchecker_tests.py", line 45, in _get_archive_url
    self.timegate_uri)
  File "/home/travis/build/jayvdb/pywikibot-core/scripts/weblinkchecker.py", line 181, in _get_closest_memento_url
    return memento_info.get('mementos').get('closest').get('uri')[0]
AttributeError: 'NoneType' object has no attribute 'get'

https://travis-ci.org/jayvdb/pywikibot-core/jobs/79818375#L3326

A patch to weblinkchecker.py has been merged to raise a more informative Exception. weblinkchecker.py doesn't have serious problem here, as it catches any error and falls back from Internet Archive -> WebCite -> user error.

---

Ideas for remediation:

• @hariharshankar has suggested filling the data structure out so that this does not happen.
• raise an exception if the TimeGate comes back with nothing
• create a convenience method for this use case that would handle this error internally

[jayvdb]

Sorry I didnt work out how to reproduce this

If this library cant recover from the gateway error, such as trying a different gateway, I'd prefer a detailed exception, and possibly even the underlying requests exception being re-raised.

[shawnmjones]

It is possible that we didn't fully understand the issue.

This problem usually occurs because there are no mementos, not due to a gateway error.

In [8]: dt = datetime.datetime(2010, 4, 24, 19, 0)
In [9]: memento_info = mc.get_memento_info("http://www.glorb.glob", dt)
In [10]: memento_info.get("mementos").get("closest").get("uri")[0]
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-10-dc666403313c> in <module>()
----> 1 memento_info.get("mementos").get("closest").get("uri")[0]

AttributeError: 'NoneType' object has no attribute 'get'

Under normal operation, get_memento_info returns a full data structure.

In [11]: memento_info = mc.get_memento_info("http://www.cnn.com", dt)

In [12]: memento_info
Out[12]:
{'mementos': {'closest': {'datetime': datetime.datetime(2010, 4, 24, 10, 15, 14),
   'http_status_code': 200,
   'uri': [u'http://wayback.archive-it.org/all/20100424101514/http://www.cnn.com/']},
  'first': {'datetime': datetime.datetime(1996, 10, 13, 23, 7, 10),
   'uri': ['http://arquivo.pt/wayback/19961013230710/http://www.cnn.com/index.html']},
  'last': {'datetime': datetime.datetime(2016, 8, 19, 6, 28, 20),
   'uri': ['http://web.archive.org/web/20160819062820/http://www.cnn.com/']}},
 'original_uri': 'http://www.cnn.com',
 'timegate_uri': 'http://timetravel.mementoweb.org/timegate/http://www.cnn.com'}

@hariharshankar suggested that we always fill in the data structure returned from get_memento_info so that the get methods will not throw the AttributeError. The final .get("closest").get("uri") will just return an empty list.

We were wondering if this solution is satisfactory.

[jayvdb]

This was part of our automated test suite. There are mementos the day before and the day after. Something else went wrong.

I raised the bug on the same day as the Travis build, and as I said "When there is a gateway error in the timegate server," in the Phabricator task, I am confident that I interrogated the timegate service myself manually and experienced a gateway error.

[shawnmjones]

Okay, I understand better now.

I think I have somewhat reproduced the problem. The httpstat.us site is supposed to return the status code requested, so I just inserted their 503 page as the TimeGate, and viola! nothing.

In [18]: mc = MementoClient(timegate_uri='http://httpstat.us/503/', check_native_timegate=False)

In [19]: memento_info = mc.get_memento_info("https://google.com", dt)

In [20]: memento_info
Out[20]:
{'original_uri': 'https://google.com',
 'timegate_uri': 'http://httpstat.us/503/https://google.com'}

The httpstat.us site did not quite work as advertised, returning a 500 status code for the TimeGate URI, but nonetheless something different should have happened here.

I agree. An exception should have been raised. Thanks for bringing this to our attention.