RFC: Session Templates for Common Crawling Patterns

[mudassaralichouhan]

Summary

Propose a reusable template system for common crawling patterns (e.g., news, e-commerce) to improve developer experience, reduce repetitive configuration, and standardize crawler behavior across similar site types.

Motivation

• Reduce repetitive configuration for common site types
• Standardize crawling patterns across the project
• Improve developer productivity

Detailed Design

Template Structure

{
  "name": "news-site",
  "description": "Template for news websites",
  "config": {
    "maxPages": 500,
    "maxDepth": 3,
    "spaRenderingEnabled": true,
    "extractTextContent": true,
    "politenessDelay": 1000
  },
  "patterns": {
    "articleSelectors": ["article", ".post", ".story"],
    "titleSelectors": ["h1", ".headline", ".title"],
    "contentSelectors": [".content", ".body", ".article-body"]
  }
}

API Endpoints

• GET /api/templates - List available templates
• POST /api/crawl/add-site-with-template - Use template for crawling
• POST /api/templates - Create new template

Implementation Plan

1. Phase 1: Core template system
2. Phase 2: Template API endpoints
3. Phase 3: Pre-built templates
4. Phase 4: Template validation and testing

Testing Strategy

• Unit tests for template loading/application
• Integration tests with real websites
• Performance benchmarks

Migration Strategy

• Backward compatible with existing API
• Gradual migration path

[Hatef-Rostamkhani]

Thanks a lot for drafting this RFC 🙏
The idea of reusable templates for common crawling patterns makes total sense and aligns well with our session-based crawler.

To move forward:

Please work on this in your own fork of the repository.

When ready, open a pull request against our master branch.

We may also create a dedicated feature/templates branch on our side to collect related work before merging into master.

RFC documents themselves will live under /rfcs/ so we can keep proposal history clear.

Labeling this issue as type:rfc + status:proposed for now. Once reviewed and accepted, we’ll switch it to status:accepted and proceed with implementation.

Really appreciate your initiative and looking forward to the first PR 🎉

[mudassaralichouhan]

Thanks, Hatef!

I’ll begin work on the feature in my fork and submit a pull request targeting master when ready. Also noted on placing RFC documents under /rfcs/—I’ll make sure to follow that structure.