Home

Welcome to the NScrapy wiki!

NScrapy is a distributed cross platform spider framework, there are 3 parts in NScrapy currently

1.Spider that analysis the webpage

2.Scheduler that handles request sending from Spider and distribute the request to any idle Downloader

3.Downloader that downloads page by getting url from Request, and then send response back to RedisMQ

Downloader has a Shell named DownloaderShell, which can be launched separatedly, all the DownloaderShell is actually a work process that can pick request from RedisMQ individually

NScrapy is written in .net core so that you can distribute NScrapy to whatever plat form that you want.

Tested Platform:

Raspberry 3B+ debian

Aliyun-Centos

wintel-win10