High Memory Usage with ScrapydWeb #236
Description
High Memory Usage with ScrapydWeb
I've observed an issue where ScrapydWeb causes excessive memory usage when running alongside Scrapyd. On an EC2 instance with 8GB of RAM and 2 vCPUs, Scrapyd alone uses around 2GB of memory, which is expected. However, when ScrapydWeb is started, the memory usage spikes to 100%, leading to the instance becoming unresponsive and eventually stopping.
Expected behavior
ScrapydWeb should operate without causing such a drastic increase in memory usage, allowing the system to remain stable.
Actual behaviour
When ScrapydWeb is running, the memory usage increases significantly, leading to the system becoming unresponsive.
Environment:
Instance: AWS EC2, 8GB RAM, 2 vCPUs
Scrapy Version: 2.11.2
Scrapyd Version: 1.4.3
ScrapydWeb Version: 1.5.0
Operating System: Linux Arm
Additional Information:
This issue occurs consistently with ScrapydWeb.
The instance is running minimal other processes, so the memory spike is attributable to ScrapydWeb.
*Request for Assistance:
Any guidance on resolving this memory usage issue or optimizing ScrapydWeb would be greatly appreciated.