Add site-specific download schemas/recipes

[MALathon]

Summary

Create a registry of site-specific "recipes" or schemas that encapsulate the best way to download PDFs from different publishers/websites. This allows fetcharoo to intelligently handle different site structures without users needing to figure out the right parameters each time.

Motivation

Different sites have different PDF structures:

• Springer: Chapters named 978-3-xxx_N.pdf, need numeric sorting
• arXiv: Usually single PDF, straightforward
• IEEE/ACM: May require authentication, have supplements
• University sites: Wildly varying structures

Currently users must manually specify sort_by, include/exclude patterns, etc. A schema registry would provide sensible defaults that "just work."

Proposed Design

1. SiteSchema Base Class

from dataclasses import dataclass
from typing import Optional, List, Callable
import re

@dataclass
class SiteSchema:
    """Base class for site-specific download configurations."""
    name: str
    url_pattern: str  # Regex to match URLs
    
    # PDF discovery
    include_patterns: List[str] = None
    exclude_patterns: List[str] = None
    pdf_selector: str = None  # CSS selector for PDF links
    
    # Ordering
    sort_by: str = None  # 'numeric', 'alpha', 'alpha_desc', 'none'
    sort_key: Callable = None
    
    # Output
    default_output_name: str = None  # e.g., "{title}.pdf"
    
    # Behavior
    recommended_depth: int = 1
    request_delay: float = 1.0
    
    def matches(self, url: str) -> bool:
        return bool(re.match(self.url_pattern, url))
    
    def extract_metadata(self, url: str, html: str) -> dict:
        """Override to extract title, author, etc. from page."""
        return {}

2. Built-in Schemas

# fetcharoo/schemas/springer.py
class SpringerBook(SiteSchema):
    name = "springer_book"
    url_pattern = r"https?://link\.springer\.com/book/.*"
    
    include_patterns = ["*_*.pdf"]  # Chapter PDFs have underscore
    exclude_patterns = ["*bbm*", "*fm*"]  # Exclude front/back matter
    sort_by = "numeric"
    recommended_depth = 1
    request_delay = 1.0
    
    def sort_key(self, url):
        # Extract chapter number from 978-3-xxx_N.pdf
        match = re.search(r'_(\d+)\.pdf$', url)
        return int(match.group(1)) if match else float('inf')
    
    def extract_metadata(self, url, html):
        # Extract book title from page
        soup = BeautifulSoup(html, 'html.parser')
        title = soup.select_one('h1.c-article-title')
        return {'title': title.text if title else 'book'}

class SpringerChaptersOnly(SpringerBook):
    """Download only individual chapters, not full book PDF."""
    name = "springer_chapters"
    exclude_patterns = ["*bbm*", "*fm*", "*978-3-*-*-?.pdf"]  # Exclude full book

# fetcharoo/schemas/arxiv.py
class ArxivPaper(SiteSchema):
    name = "arxiv"
    url_pattern = r"https?://arxiv\.org/(abs|pdf)/.*"
    sort_by = "none"
    recommended_depth = 0
    
    def extract_metadata(self, url, html):
        # Extract paper ID and title
        ...

3. Schema Registry

# fetcharoo/schemas/registry.py
_SCHEMAS: Dict[str, SiteSchema] = {}

def register_schema(schema: SiteSchema):
    """Register a schema in the global registry."""
    _SCHEMAS[schema.name] = schema

def get_schema(name: str) -> Optional[SiteSchema]:
    """Get schema by name."""
    return _SCHEMAS.get(name)

def detect_schema(url: str) -> Optional[SiteSchema]:
    """Auto-detect schema from URL."""
    for schema in _SCHEMAS.values():
        if schema.matches(url):
            return schema
    return None

def list_schemas() -> List[str]:
    """List all registered schema names."""
    return list(_SCHEMAS.keys())

# Decorator for easy registration
def schema(cls):
    register_schema(cls())
    return cls

4. Integration with Main API

from fetcharoo import download_pdfs_from_webpage

# Auto-detect schema from URL
result = download_pdfs_from_webpage(
    url='https://link.springer.com/book/10.1007/978-3-031-41026-0',
    schema='auto'  # Detects SpringerBook
)

# Explicitly use a schema
result = download_pdfs_from_webpage(
    url='https://link.springer.com/book/...',
    schema='springer_chapters'
)

# Use schema instance with overrides
from fetcharoo.schemas import SpringerBook
result = download_pdfs_from_webpage(
    url='...',
    schema=SpringerBook(request_delay=2.0)
)

# Schema parameters are defaults; explicit params override
result = download_pdfs_from_webpage(
    url='...',
    schema='springer_book',
    sort_by='alpha'  # Overrides schema's 'numeric'
)

5. CLI Integration

# Auto-detect
fetcharoo https://link.springer.com/book/... --schema auto

# Explicit schema
fetcharoo https://link.springer.com/book/... --schema springer_chapters

# List available schemas
fetcharoo --list-schemas

6. User-Defined Schemas

Users can add custom schemas in a config file:

# ~/.config/fetcharoo/schemas.yaml
my_university:
  url_pattern: "https?://library\\.myuni\\.edu/.*"
  include_patterns: ["*.pdf"]
  exclude_patterns: ["*thumbnail*", "*preview*"]
  sort_by: alpha
  request_delay: 2.0

Or programmatically:

from fetcharoo.schemas import SiteSchema, register_schema

@register_schema
class MyUniversityLibrary(SiteSchema):
    name = "myuni"
    url_pattern = r"https://library\.myuni\.edu/.*"
    sort_by = "alpha"

Implementation Plan

1. Create SiteSchema base dataclass
2. Implement schema registry with auto-detection
3. Add built-in schemas: Springer, arXiv, generic
4. Integrate schema parameter into download_pdfs_from_webpage
5. Add CLI --schema flag and --list-schemas
6. Support YAML config for user schemas
7. Add tests for schema matching and behavior
8. Document schema system in README

Open Questions

1. Should schemas be able to define custom PDF extraction logic (beyond CSS selectors)?
2. How to handle authentication requirements (just document, or provide hooks)?
3. Should we ship schemas for sites that may have legal restrictions?
4. Version schemas separately to update without library release?

Related

• Builds on Add sort_key parameter for merge mode ordering #3 (sort ordering), Deduplicate PDF URLs by default #4 (deduplication), Add custom output filename for merge mode #5 (output naming)

[MALathon]

Additional: Schema Validation & Health Monitoring

Per feedback - schemas need periodic validation since sites change their structure.

Schema Health System

@dataclass
class SchemaHealth:
    schema_name: str
    last_validated: datetime
    status: Literal['healthy', 'degraded', 'broken']
    sample_url: str  # Known-good URL for testing
    expected_pdf_count: int  # Expected minimum PDFs
    last_error: Optional[str] = None

class SiteSchema:
    # ... existing fields ...
    
    # Validation
    test_url: str = None  # Sample URL for validation
    expected_min_pdfs: int = 1  # Minimum PDFs to find
    
    def validate(self) -> SchemaHealth:
        """Test if schema still works against sample URL."""
        try:
            pdfs = find_pdfs_from_webpage(
                self.test_url, 
                recursion_depth=self.recommended_depth,
                # Use schema settings...
            )
            status = 'healthy' if len(pdfs) >= self.expected_min_pdfs else 'degraded'
            return SchemaHealth(
                schema_name=self.name,
                last_validated=datetime.now(),
                status=status,
                sample_url=self.test_url,
                expected_pdf_count=self.expected_min_pdfs
            )
        except Exception as e:
            return SchemaHealth(
                schema_name=self.name,
                last_validated=datetime.now(),
                status='broken',
                sample_url=self.test_url,
                expected_pdf_count=self.expected_min_pdfs,
                last_error=str(e)
            )

Validation CLI

# Validate all schemas
fetcharoo --validate-schemas

# Validate specific schema
fetcharoo --validate-schema springer_book

# Output:
# ✓ springer_book: healthy (found 12 PDFs, expected ≥1)
# ⚠ arxiv: degraded (found 0 PDFs, expected ≥1)  
# ✗ ieee_paper: broken (ConnectionError: 403 Forbidden)

CI/Scheduled Validation

# .github/workflows/validate-schemas.yml
name: Validate Schemas
on:
  schedule:
    - cron: '0 0 * * 0'  # Weekly
  workflow_dispatch:  # Manual trigger

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install fetcharoo
      - run: fetcharoo --validate-schemas --output json > results.json
      - name: Create issue if broken
        if: failure()
        uses: actions/github-script@v7
        with:
          script: |
            // Parse results.json and create/update issue for broken schemas

Schema Versioning & Updates

class SiteSchema:
    version: str = "1.0.0"  # Schema version
    last_verified: str = "2025-01-15"  # Last known working date
    maintainer: str = None  # Who to ping if broken

# Registry tracks health
_SCHEMA_HEALTH: Dict[str, SchemaHealth] = {}

def get_schema_with_warning(name: str) -> SiteSchema:
    """Get schema, warn if degraded/broken."""
    schema = get_schema(name)
    health = _SCHEMA_HEALTH.get(name)
    if health and health.status != 'healthy':
        logger.warning(f"Schema '{name}' is {health.status}: {health.last_error}")
    return schema

Remote Schema Registry (Future)

Could fetch updated schemas from a central registry:

# Check for schema updates
fetcharoo --update-schemas

# Fetches from https://github.com/MALathon/fetcharoo-schemas
# or a JSON endpoint with latest schema definitions

This allows schema fixes without requiring a full library update.

Implementation additions to plan:

• Add test_url and expected_min_pdfs to SiteSchema
• Implement validate() method
• Add --validate-schemas CLI command
• Create GitHub Action for weekly validation
• Auto-create issues for broken schemas
• Consider remote schema registry for decoupled updates

[MALathon]

Implementation Breakdown

This parent issue has been broken down into the following child issues:

Core Infrastructure

1. Create SiteSchema base dataclass #11 - Create SiteSchema base dataclass
2. Implement schema registry with auto-detection #12 - Implement schema registry with auto-detection

Built-in Content

3. Add built-in schemas for common sites (Springer, arXiv) #13 - Add built-in schemas for common sites (Springer, arXiv)

Integration

4. Integrate schema parameter into download functions #14 - Integrate schema parameter into download functions
5. Add CLI support for schemas (--schema, --list-schemas) #15 - Add CLI support for schemas (--schema, --list-schemas)

Validation & Health

6. Add schema validation and health checking #16 - Add schema validation and health checking
7. Add CLI commands for schema validation #17 - Add CLI commands for schema validation
8. Add GitHub Action for automated schema validation #18 - Add GitHub Action for automated schema validation

Extensibility

9. Support user-defined schemas via YAML config file #19 - Support user-defined schemas via YAML config file
10. Add remote schema registry for community schemas #20 - Add remote schema registry for community schemas

Recommended Implementation Order

#11 → #12 → #13 → #14 → #15 → #16 → #17 → #18 → #19 → #20
 │      │      │      │      │      │      │      │      │
 └──────┴──────┴──────┴──────┴──────┴──────┴──────┴──────┘
   Core foundation    Integration    Validation    Extensibility

Each issue can be merged as a separate PR, building incrementally on the previous work.