Selenium Using Selenium and BeautifulSoup is another option. See scrapy.py to get started. References https://www.zenrows.com/blog/selenium-python-web-scraping#headless-chrome https://medium.com/ymedialabs-innovation/web-scraping-using-beautiful-soup-and-selenium-for-dynamic-page-2f8ad15efe25