Requirement
- The get_page requirement is currently set to static manually set timeout duration. It would great to have an adaptive method get_page , which will not timeout because of few network stability issues but will also not wait indefinitely.
Where to look
- Scrapper.py inside commons
Update: An implementation is provided for exponential wait time increase in utility package as AdaptiveRequest class. The issue is not resolved, as in case of fault network the system may end up raising its wait time indefinite, i.e leading back to the old problem of having no wait time for request.