Spider Utility

Utilities for spider.

Features

• Database connector
• Local file path generator
• Network session
• Data structure
• Log
• Exception

Installation

pip install spiderutil

Documentation

Still working...

Examples

Still working...

How to contribute

Open a new issue when you have questions, advices, or you encounter a bug.

Recommend packages

Parsers

• beautifulsoup4: parse the webpage like a tree.

pip install beautifulsoup4

• scrapy/parsel: Parse webpage with XPath or CSS selectors.

pip install parsel

Encoding and Decoding

• google/brotli: decode brotli encoding.

pip install brotli

Database

• SQLAlchemy: Process the SQL statement like object

pip install SQLAlchemy

• PyMySQL: Connect to MySQL

pip install pymysql

• mkleehammer/pyodbc: Connect to Microsoft SQL Server

pip install pyodbc

In addition to: level12/sqlalchemy_pyodbc_mssql.