Skip to content

Refactor: Worker-based appraisal with Redis caching#8

Merged
soumyaray merged 18 commits intomainfrom
refactor-microservice
Dec 23, 2025
Merged

Refactor: Worker-based appraisal with Redis caching#8
soumyaray merged 18 commits intomainfrom
refactor-microservice

Conversation

@soumyaray
Copy link
Contributor

@soumyaray soumyaray commented Dec 23, 2025

Summary

  • Move appraisal logic from API to background worker
  • Use Redis as results cache instead of Rack::Cache reverse-proxy
  • Worker clones repo, runs git blame, caches JSON in Redis
  • API becomes lightweight cache reader + request dispatcher
  • Environment isolation via Redis database indexes (dev=/0, test=/1)

Key Changes

  • New Worker::AppraiseProject service with Dry::Transaction pipeline
  • New Service::FetchOrRequestAppraisal (replaces AppraiseProject)
  • New Value::Appraisal domain object for appraisal results
  • Removed Rack::Cache configuration
  • Renamed env vars: CLONE_* → WORKER_*, REDISCLOUD_URL → REDIS_URL
  • Worker DDD restructure with parallel layer structure

Test plan

  • All unit tests pass
  • Integration tests pass with Redis service
  • Acceptance tests pass with worker running

🤖 Generated with Claude Code

soumyaray and others added 18 commits December 7, 2025 23:32
@soumyaray @claude
Add CLAUDE.local.md to gitignore
9249e09 
Local planning file for refactoring discussions that should not be tracked.

Co-Authored-By: Claude <noreply@anthropic.com>
@soumyaray @claude
refactor: separate worker output from test output in acceptance tests
6dd6a1b 
Worker output now redirected to spec/logs/worker.log (gitignored) to
provide cleaner test output. A summary of worker activity is shown
after tests complete.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@soumyaray @claude
refactor: add Value::Appraisal domain object for worker-based caching
8a689b8 
Introduces Value::Appraisal as an immutable Dry::Struct representing
the result of appraising a project folder. Supports both success and
error states with factory methods.

- Attributes: status, project, folder_path, folder, error_type, error_message
- Methods: success?, error?, cache_key, ttl
- TTL constants: SUCCESS_TTL (1 day), ERROR_TTL (10 seconds)

Part of Phase 1 of the worker-based appraisal refactoring.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@soumyaray @claude
refactor: add Representer::Appraisal for JSON serialization
7d54665 
Add presentation layer representer for Value::Appraisal:
- Serializes status symbol as string for JSON output
- Conditionally includes folder contributions on success
- Conditionally includes error_type and message on error
- Reuses existing Project and FolderContributions representers

Part of Phase 1: Domain + Infrastructure Foundation

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@soumyaray @claude
refactor: add Redis cache methods and Phase 1 unit tests
ab41b13 
Cache::Remote:
- Add get(key), set(key, value, ttl:), exists?(key) methods
- Uses setex for atomic set with expiration

Test infrastructure:
- Add fakeredis gem for isolated Redis testing in unit tests
- Add CacheHelper for creating fake cache instances
- Add unit tests for Value::Appraisal and Representer::Appraisal
- Add unit tests for Cache::Remote new methods

All tests passing (68 runs, 191 assertions, 96% coverage)

Part of Phase 1: Domain + Infrastructure Foundation

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@soumyaray @claude
refactor: add worker appraisal service with Redis caching
18aec78 
Worker service (Worker::AppraiseProject):
- Dry::Transaction with steps: prepare_inputs → clone_repo → appraise_contributions → cache_result
- Converts OpenStruct project to Entity::Project
- Creates Value::Appraisal (success or error), serializes with Representer
- Stores JSON in Redis with TTL

Worker modifications:
- git_clone_worker.rb: routes AppraisalRequest to new flow, CloneRequest to legacy
- job_reporter.rb: parses both request formats, exposes folder_path and progress_callback
- clone_monitor.rb: adds AppraisalMonitor with new progress phases (clone 0-50%, appraise 50-90%, cache 90-100%)

Presentation layer:
- Response::AppraisalRequest struct (project, folder_path, id)
- Representer::AppraisalRequest for JSON serialization

Tests: 78 runs, 212 assertions, 94.6% coverage

Part of Phase 2: Modify Worker to Appraise

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@soumyaray @claude
refactor: complete Phase 3 - simplify API with Redis cache pass-through
31bb391 
Phase 3 Implementation:
- Add FetchOrRequestAppraisal service (replaces AppraiseProject)
- API checks Redis cache first, returns cached JSON directly
- Cache miss sends AppraisalRequest to worker, returns 202
- Worker stores serialized JSON in Redis with TTL

Redis Infrastructure:
- Use key prefix approach (test: prefix) instead of DB numbers
- Cloud Redis doesn't support multiple databases
- Local Redis for dev/test (redis://localhost:6379)
- Production uses cloud Redis

Developer Experience:
- rake redis:start/stop/remove/ensure tasks for Docker container
- rake spec automatically ensures Redis is running
- GitHub Actions: Redis service (Linux) + Homebrew (macOS)
- Updated README and CLAUDE.md with setup docs

Test Updates:
- Updated acceptance tests for new response format
- Integration tests for FetchOrRequestAppraisal
- Removed fakeredis (using real Redis with test prefix)

82 tests pass, 93.66% coverage

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@soumyaray @claude
refactor: Phase 4 cleanup - remove old service and unused representers
04909ef 
Remove deprecated code replaced by worker-based appraisal architecture:
- Remove AppraiseProject service (replaced by FetchOrRequestAppraisal)
- Remove ProjectFolderContributions response and representer (replaced by Value::Appraisal)
- Remove Rack::Cache headers from appraisal endpoint (Redis is source of truth)
- Remove old AppraiseProject integration test

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@soumyaray @claude
refactor: move contributions domain to worker-only location
b392629 
Move git blame analysis domain objects to workers/domain/ since they're
only used by the worker, not the API:

- Create require_worker.rb parallel to require_app.rb for worker code loading
- Move entities: FolderContributions, FileContributions, LineContribution, Contributor
- Move values: FilePath, CodeLanguage, CreditShare, Contributors
- Move lib: ContributionsCalculator, Types
- Update Value::Appraisal to use Nominal(Object) type for folder attribute
- Update spec_helper to load worker domain for tests

The API only needs Value::Appraisal (for cache key format).
All heavy contribution entities are now worker-only concerns.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@soumyaray @claude
refactor: move git infrastructure to worker-only location
d170ac4 
Move git-related infrastructure to workers/infrastructure/ since it's
only used by the worker for clone and blame operations:

- Move gateway: git_command.rb (git CLI wrapper)
- Move repositories: git_repo.rb, local_repo.rb, remote_repo.rb,
  blame_reporter.rb, repo_file.rb, repo_store.rb
- Move mappers: contributions_mapper.rb, folder_contributions_mapper.rb,
  file_contributions_mapper.rb, blame_contributor.rb, porcelain_parser.rb
- Update require_worker.rb to load infrastructure folder by default
- Update spec_helper to load worker infrastructure for tests

API infrastructure now contains only: cache, database, github, messaging

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@soumyaray @claude
refactor: restructure worker with DDD layers
0258352 
Reorganize worker files into DDD-style layers parallel to the API:

- application/controllers/worker.rb - Shoryuken entry point
- application/services/appraise_project.rb - Business workflow
- application/requests/job_reporter.rb - Request parsing + progress
- infrastructure/messaging/progress_publisher.rb - Faye HTTP gateway
- presentation/values/progress_monitor.rb - Progress calculation

Update require_worker.rb to load: domain, infrastructure, presentation, application
Update Rakefile and Procfile with new worker path

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@soumyaray @claude
docs: add architecture section to README
f56b47b 
Add brief architecture overview explaining API/Worker separation
and parallel DDD layer structures.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@soumyaray
chore: renamed worker to Appraiser
d9f51cb
@soumyaray @claude
refactor: remove Rack::Cache reverse-proxy caching
a5ea58d 
- Remove rack-cache and redis-rack-cache gems
- Delete local_cache.rb file-based cache (was for Rack::Cache)
- Remove Rack::Cache middleware configuration from environment.rb
- Simplify cache rake tasks (remove Rack::Cache specific tasks)
- Update redis_cache.rb and cache_helper.rb comments
- Remove LOCAL_CACHE config from secrets_example

Redis is now used directly for appraisal caching via Cache::Remote,
eliminating the need for HTTP reverse-proxy caching layer.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@soumyaray @claude
refactor: use Redis database indexes and rename queue env vars
c4cecc2 
Cache infrastructure:
- Use Redis database indexes for environment isolation (dev=/0, test=/1)
- Rename REDISCLOUD_URL → REDIS_URL for consistency
- Remove key prefix logic from Cache::Remote (simpler implementation)
- Consolidate rake tasks under 'cache' namespace (technology-agnostic)
  - cache:status, cache:ensure, cache:list, cache:wipe
  - cache:redis:start, cache:redis:stop, cache:redis:remove

Queue infrastructure:
- Rename CLONE_QUEUE → WORKER_QUEUE (reflects broader responsibility)
- Rename CLONE_QUEUE_URL → WORKER_QUEUE_URL
- Update shoryuken config files with new queue names
- Fix queues:create task to work before queue exists

Testing:
- Use `bash spec/acceptance_tests` for full test suite with worker
- 75 runs, 201 assertions, 0 failures, 93.21% coverage

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@soumyaray
chore: rename worker service namespace and remove legacy code
0552c3d 
Worker service:
- Rename Worker::AppraiseProject → Appraiser::Service::AppraiseProject
- Move service into Appraiser::Service namespace for consistency
- Remove legacy perform_clone_only method (no longer needed)
- Remove appraisal_request? check (all requests are appraisal requests)

Controller simplification:
- Remove redundant comments and conditional branches
- Simplify cache hit response handling

Code style:
- Use Ruby 3.4 'it' parameter instead of '_1' in method chains
@soumyaray
chore: use REDIS_URL secret in GitHub Actions workflow
5993c73 
Keep environment variable configuration consistent across local,
CI, and production environments.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
@soumyaray
ci: simplify workflow to Linux-only with latest actions
0a32fb0 
- Remove macOS matrix (Linux matches Heroku production environment)
- Update actions/checkout from v4 to v6
- Keep ruby/setup-ruby@v1 (recommended semantic versioning)

🤖 Generated with [Claude Code](https://claude.com/claude-code)
@soumyaray soumyaray merged commit 356e237 into main Dec 23, 2025
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@soumyaray