Skip to content

JWT Authentication & Rate Limiting System#43

Open
RichardCMX wants to merge 41 commits intomainfrom
feature/auth-rate-limits
Open

JWT Authentication & Rate Limiting System#43
RichardCMX wants to merge 41 commits intomainfrom
feature/auth-rate-limits

Conversation

@RichardCMX
Copy link
Collaborator

Overview

Implements a comprehensive JWT-based authentication system and tiered rate limiting protection for all API endpoints, enhancing security and preventing abuse.

Related Issues

Closes #30

Changes Made

🔑 JWT Authentication System

  • User Registration (POST /api/auth/register/)
    • Password confirmation validation
    • Automatic JWT token generation on registration
    • User profile data in response
  • User Login (POST /api/auth/login/)
    • Access token (1-hour lifetime) and refresh token (7-day lifetime)
    • Token rotation with blacklisting for security
    • User data included in response
  • Token Refresh (POST /api/auth/refresh/)
    • Seamless token renewal without re-authentication
    • Rotating refresh tokens with blacklisting
  • User Profile (GET /api/auth/profile/)
    • JWT-authenticated access to user information
    • Proper 401 responses for unauthorized access

🛡️ Rate Limiting Protection

Tiered rate limiting strategy across all API endpoints:

  • Public Light endpoints (health, ready): 100 requests/minute
  • Public Medium endpoints (arrivals, schedule): 60 requests/minute
  • Public Heavy endpoints (search): 30 requests/minute
  • Auth Sensitive operations (login): 5 requests/minute
  • Auth Registration: 3 requests/minute
  • Auth General operations (profile): 20 requests/minute

Features:

  • IP-based rate limiting using django-ratelimit
  • Detailed 429 error responses with retry information
  • Configurable via environment variables
  • Can be disabled for development/testing (RATELIMIT_ENABLE)
  • Redis-backed for distributed deployments

📦 Dependencies Added

  • djangorestframework-simplejwt==5.3.0 - JWT authentication
  • django-ratelimit==4.1.0 - Rate limiting middleware

🧪 Testing

20 comprehensive tests added (100% passing):

  • 10 JWT authentication tests (api/tests/test_jwt_auth.py)
    • Registration, login, token refresh, profile access
    • Error handling and validation scenarios
    • Protected vs public endpoint access control
  • 10 rate limiting tests (api/tests/test_rate_limiting.py)
    • Rate limit enforcement across all tiers
    • 429 error response validation
    • Configuration and toggle testing
    • Authenticated vs unauthenticated limits

📝 Documentation

  • Updated README.md with authentication and rate limiting sections
  • Updated CHANGELOG.md with comprehensive feature documentation
  • Updated api/tests/README.md with new test documentation
  • Added configuration examples and usage guides

Security Enhancements

  • ✅ Token rotation prevents replay attacks
  • ✅ Blacklisting prevents use of compromised tokens
  • ✅ Rate limiting protects against DoS and brute force attacks
  • ✅ Intelligent tiered limits based on endpoint sensitivity
  • ✅ IP-based tracking prevents circumvention
  • ✅ Environment-based configuration prevents secrets in code

Backward Compatibility

  • ✅ All existing API endpoints work unchanged
  • ✅ Public endpoints remain accessible without authentication
  • ✅ Optional authentication can be adopted incrementally
  • ✅ Rate limiting can be disabled for development

Testing Results

Found 65 test(s).
Creating test database for alias 'default'...
✓ PostgreSQL extensions installed in test database (postgis, pg_trgm, unaccent)
System check identified no issues (0 silenced).
.................................................................
----------------------------------------------------------------------
Ran 65 tests in 13.239s

OK

Configuration

.env additions:

# JWT Authentication
SECRET_KEY=your-super-secure-secret-key-here

# Rate Limiting
RATELIMIT_ENABLE=true

Usage Examples

# Register new user
curl -X POST "http://localhost:8000/api/auth/register/" \
  -H "Content-Type: application/json" \
  -d '{"username": "newuser", "email": "user@example.com", "password": "secure123", "password_confirm": "secure123"}'

# Login and get tokens
curl -X POST "http://localhost:8000/api/auth/login/" \
  -H "Content-Type: application/json" \
  -d '{"username": "newuser", "password": "secure123"}'

# Access protected endpoint
curl "http://localhost:8000/api/auth/profile/" \
  -H "Authorization: Bearer YOUR_ACCESS_TOKEN_HERE"

# Refresh token
curl -X POST "http://localhost:8000/api/auth/refresh/" \
  -H "Content-Type: application/json" \
  -d '{"refresh": "YOUR_REFRESH_TOKEN_HERE"}'

Files Modified

  • README.md - Authentication and rate limiting documentation
  • CHANGELOG.md - Feature documentation
  • datahub/settings.py - JWT and rate limiting configuration
  • api/auth_views.py - JWT authentication views (new)
  • api/rate_limiting.py - Rate limiting utilities (new)
  • api/views.py - Rate limiting integration
  • api/urls.py - Authentication endpoints
  • api/tests/test_jwt_auth.py - JWT tests (new)
  • api/tests/test_rate_limiting.py - Rate limiting tests (new)
  • api/tests/README.md - Test documentation
  • pyproject.toml - Dependencies

Checklist

  • Code follows project style guidelines
  • Tests added and passing (20 new tests, 65 total)
  • Documentation updated (README, CHANGELOG, test docs)
  • No breaking changes
  • Security best practices followed
  • Backward compatible
  • Environment configuration documented

Next Steps

After merging, this branch serves as the base for:

  • feature/client-management - Client application management
  • feature/security-performance - Additional security and performance feature

RichardCMX and others added 30 commits September 28, 2025 17:22
…as HH:MM:SS; validate stop_id; add Fuseki flags and DAL docs; add /api/schedule/departures/ endpoint
…tures endpoint; tests(api): add tests for DAL-backed schedule departures
…add Fuseki dev guide and update README/architecture; fix duplicate FUSEKI_ENDPOINT
…tion (LimitOffsetPagination, page size 50)\n- Add /api/alerts route (ServiceAlertViewSet)\n- Add /api/arrivals endpoint integrating with ETAS_API_URL (Project 4)\n- Add /api/status endpoint reporting DB/Redis/Fuseki health\n- Update OpenAPI (datahub.yml) for pagination + new endpoints
…Message, StopTimeUpdate to real model fields
…, stop-time-updates), pagination, OpenAPI docs, ETAs config, and testing instructions
…rules, feed-messages, stop-time-updates; polish examples for core endpoints
- Implement /api/search/ endpoint with ranking for stops and routes
  - Support for fuzzy text matching with relevance scoring
  - Configurable search types (stops, routes, all)
  - Limit and pagination support
  - Feed-specific search capability

- Add /api/health/ endpoint for basic health checks
  - Simple status check returning 200 OK
  - Minimal response for lightweight monitoring

- Add /api/ready/ endpoint for readiness checks
  - Database connectivity verification
  - Current feed availability check
  - Returns 503 when not ready to serve requests

- Comprehensive test coverage for all new endpoints
  - Search functionality tests with various scenarios
  - Health endpoint validation tests
  - Edge cases and error handling tests

- Full OpenAPI documentation integration
- Proper error handling and validation
- Follows existing code patterns and conventions
- Fix missing FloatField import in views.py for search functionality
- Add comprehensive edge case tests for search:
  - Case insensitivity testing
  - Special characters and Unicode handling
  - Numbers and symbols in queries
  - Very long query handling
- Add additional health endpoint test for multiple current feeds
- Ensure robust error handling and graceful degradation
- Add custom API root view to include search, health, and ready endpoints
- Update /api/ to show all endpoints including new search and health services
- Add comprehensive README documentation for Issue #28:
  - Search API with intelligent ranking and fuzzy matching
  - Health monitoring endpoints for load balancers and Kubernetes
  - Complete usage examples with curl commands
  - Response format documentation
  - Integration examples (Docker health checks, K8s probes)

The API root now shows:
- search: http://localhost:8000/api/search/
- health: http://localhost:8000/api/health/
- ready: http://localhost:8000/api/ready/
✨ Features:
- Add djangorestframework-simplejwt dependency for JWT support
- Implement user registration endpoint with JWT token generation
- Add custom JWT login/refresh views with user information
- Create user profile endpoint requiring authentication

🔐 Security:
- Configure JWT with 1-hour access tokens and 7-day refresh tokens
- Apply authentication requirements to all GTFS data CRUD endpoints
- Keep public endpoints (health, search, arrivals) accessible without auth
- Add proper 401 error responses with detailed messages

🧪 Testing:
- Add comprehensive JWT authentication test suite
- Test user registration, login, token refresh, and profile access
- Verify endpoint protection and public endpoint accessibility
- All 10 tests passing successfully

📝 API Endpoints:
- POST /api/auth/register/ - Register new user
- POST /api/auth/login/ - Login and get JWT tokens
- POST /api/auth/refresh/ - Refresh access token
- GET /api/auth/profile/ - Get authenticated user profile

Addresses issue #30 acceptance criteria: 'JWT for registered clients'
✨ JWT Authentication Features:
- User registration with validation (username, email, password confirmation)
- JWT token-based login with access/refresh token pairs
- Token refresh functionality for seamless auth renewal
- Protected user profile endpoint with authentication
- Custom JWT views with enhanced error handling and user data
- Comprehensive test suite (10 tests) covering all auth scenarios

🛡️ Rate Limiting Implementation:
- Comprehensive rate limiting across all API endpoints
- Tiered rate limiting strategy:
  • Light endpoints (health/ready): 100 req/min
  • Medium endpoints (arrivals/schedule): 60 req/min
  • Heavy endpoints (search): 30 req/min
  • Auth endpoints: 5-20 req/hour for sensitive operations
- IP-based rate limiting with configurable limits
- Proper 429 error responses with retry information
- Rate limiting can be disabled via RATELIMIT_ENABLE setting
- Unified rate_limiting.py module with both simple and decorator approaches

🧪 Testing & Quality:
- 20 comprehensive tests (10 JWT auth + 10 rate limiting)
- All tests passing with 100% success rate
- Test coverage for edge cases and error scenarios
- Organized test structure in api/tests/ directory

⚙️ Configuration:
- Added django-ratelimit dependency (v4.1.0)
- JWT settings configured in Django settings
- Environment-based rate limiting configuration
- Backward compatible implementation

🔧 Technical Implementation:
- RESTful API design with proper HTTP status codes
- DRF integration with custom authentication classes
- Clean separation of concerns between auth and rate limiting
- Documentation-ready with OpenAPI schema integration
- Production-ready error handling and validation

This implementation provides robust authentication and API protection
suitable for production deployment with comprehensive test coverage.
…ation

📚 Documentation Updates:
- Enhanced README.md with detailed authentication and rate limiting sections
- Added step-by-step JWT authentication workflow with cURL examples
- Documented rate limiting tiers and configuration options
- Updated security checklist for production deployment
- Added environment variable documentation for new features

📝 CHANGELOG.md Creation:
- Comprehensive changelog following Keep a Changelog format
- Detailed documentation of JWT authentication implementation
- Complete rate limiting feature documentation with technical details
- Test coverage documentation (20 tests, 100% passing)
- Security enhancements and configuration examples
- Performance impact analysis and compatibility information

🔧 Features Documented:
- JWT user registration, login, token refresh, and profile endpoints
- Tiered rate limiting across all API endpoints (14 endpoints protected)
- Security configuration with environment-based settings
- Production deployment considerations and security checklist

This documentation update provides complete guidance for using the new
authentication and rate limiting features introduced in the previous commit.
- Remove Fuseki Docker service from docker-compose.yml
- Remove fuseki_data volume
- Delete storage/fuseki_schedule.py implementation
- Delete api/tests/test_fuseki_schedule.py integration tests
- Remove docker/fuseki/ configuration directory
- Remove docs/dev/fuseki.md documentation
- Update storage/factory.py to use only PostgreSQL repository
- Remove FUSEKI_ENABLED and FUSEKI_ENDPOINT from settings.py
- Remove Fuseki environment variables from .env.local.example
- Update README.md and docs/architecture.md to remove Fuseki references

PostgreSQL with Redis caching is now the sole storage backend.
- Document Data Access Layer implementation
- Document new /api/schedule/departures/ endpoint
- Document Redis caching configuration
- Document Fuseki removal
- Follow Keep a Changelog format
- Add class-level docstring explaining DAL testing
- Document setUp method for test data preparation
- Add docstrings for test_returns_404_when_stop_missing
- Add docstrings for test_returns_departures_with_expected_shape
- Improve test readability and maintainability
- Document test structure and organization
- Explain test coverage for schedule departures endpoint
- Provide examples for running tests
- Document test data setup approach
- Add guidelines for adding new tests
RichardCMX and others added 11 commits November 13, 2025 12:25
- Document /api/arrivals/ endpoint with ETA service integration
- Document /api/status/ health check endpoint
- Document /api/alerts/, /api/feed-messages/, /api/stop-time-updates/
- Document global pagination implementation
- Document ETAS_API_URL configuration
- Document comprehensive test suite for arrivals endpoint
- Add class-level docstring explaining ETA service integration testing
- Document test_arrivals_returns_expected_shape
- Document test_arrivals_propagates_upstream_error
- Document test_arrivals_requires_stop_id
- Document test_arrivals_accepts_wrapped_results_object
- Document test_arrivals_handles_unexpected_upstream_structure_as_empty_list
- Document limit validation tests
- Document test_arrivals_returns_501_if_not_configured
- Add test_arrivals.py documentation
- Document all 9 test cases for arrivals endpoint
- Add examples for running arrivals tests
- Document mocked HTTP request testing approach
- Update coverage section with new test areas
- Add unittest.mock to dependencies
- Update search queries to use __unaccent lookup for accent-insensitive matching
- Support multilingual searches (Spanish, Portuguese, etc.)
- Searches like 'San José' now match 'San Jose' and vice versa
- Trigram similarity now operates on unaccented text for better fuzzy matching

This improves search experience for Costa Rican transit data with accented characters.
BUG DISCOVERED:
Issue #28 (search/autocomplete endpoints) implemented TrigramSimilarity
for fuzzy text matching but never created the required PostgreSQL pg_trgm
extension. The code silently fell back to basic string matching (icontains)
via try/except blocks in api/views.py lines 1064-1104 and 1125-1179.

This bug went undetected because:
- Original tests validated API response structure, not trigram functionality
- Exception handling masked the missing extension
- Fallback logic allowed endpoints to return results

IMPACT:
- Search accuracy degraded (no fuzzy matching)
- Search performance reduced (no trigram indexing)
- Feature deployed incomplete

FIX:
Add PostgreSQL extension setup for both main and test databases:

1. docker/db/init.sql
   - Creates pg_trgm extension in dev/prod database on first container run
   - Mounted via docker-compose.yml at /docker-entrypoint-initdb.d/
   - Enables TrigramSimilarity queries in search endpoints

2. datahub/test_runner.py
   - Custom Django test runner (InfobusTestRunner)
   - Creates pg_trgm extension in isolated test database
   - Required because Django doesn't copy extensions to test DB

3. datahub/settings.py
   - Configure TEST_RUNNER to use InfobusTestRunner
   - Ensures extensions available during test execution

4. docker-compose.yml
   - Mount init.sql to PostgreSQL initialization directory
   - Extension created automatically on database first start

VERIFICATION:
Comprehensive integration tests now verify actual trigram functionality
instead of just API response structure, catching this missing setup.

Resolves incomplete implementation from commit ea877e2 (Issue #28).
- Add SpectacularSwaggerView to api/urls.py
- Available at /api/docs/swagger/
- Provides interactive forms for testing all API endpoints
- Complements existing ReDoc documentation at /api/docs/
- Document /api/search/ with fuzzy matching and unaccent support
- Document /api/health/ and /api/ready/ endpoints
- Document PostgreSQL extensions (pg_trgm, unaccent)
- Document Swagger UI and ReDoc integration
- Document comprehensive test suites
- Add multilingual search documentation with unaccent extension
- Document accent-insensitive search (San Jose matches San José)
- Add Interactive API Documentation section
- Document Swagger UI at /api/docs/swagger/
- Document ReDoc and DRF Browsable API
- Improve search feature descriptions
Resolved conflicts in CHANGELOG.md and datahub/settings.py by keeping both sets of configurations:
- Kept JWT and rate limiting configs from auth-rate-limits
- Kept TEST_RUNNER and security settings from search-health-endpoints
- Combined both branches' features
- Document test_jwt_auth.py with 10 test cases
- Document test_rate_limiting.py with 10 test cases
- Update test dependencies with JWT and Redis requirements
- Update test coverage section with security features
- Add test running examples for new test files
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Authentication (JWT) and public rate limits

2 participants