feat: Stock Docker Compose Integration via harbor.yaml #204
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Use temp file approach instead of in-place editing to avoid compatibility issues between GNU sed (-i without space) and BSD sed (-i with space for backup extension). This ensures harbor.sh works correctly regardless of which sed implementation is installed (common issue on macOS with Homebrew's GNU sed).
- Add OS detection to use appropriate grep syntax - Use basic regex (-oE) for macOS/Darwin instead of Perl regex (-oP) - Maintains compatibility across GNU and BSD grep implementations - Fixes tunnel URL extraction on macOS systems 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
…l-url fix: improve grep portability in extract_tunnel_url function
Harbor routines run in containerized Deno environment but couldn't access JSR registry for downloading dependencies like @std/yaml. This caused 'harbor ls' and other commands to fail on Linux systems. The fix adds --network=host to the Docker run command in run_routine(), allowing containers to access external registries while maintaining the same security model (short-lived utility containers with existing filesystem access). Fixes network connectivity issues when downloading JSR packages on Linux systems where Docker's default bridge networking is more restrictive. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
- Add routines/upstream.ts: Core transformation module for stock compose files - Modify routines/mergeComposeFiles.ts: Integrate upstream loading into merge flow - Add dify2/ test service with stock dify compose file - Add harbor.upstream.yaml schema for declaring transformations - Update AGENTS.md with design documentation - Add Progress.md for development tracking Transformation features: - Service name prefixing (api -> dify2-api) - Container name with HARBOR_CONTAINER_PREFIX - depends_on reference updates - network_mode: service:X reference updates - Named volume prefixing - harbor-network injection - env_file injection for Harbor integration - Optional init container support for config preparation
…ection - Rename config file to harbor.yaml for future extensibility - Add HarborConfig interface with upstream, metadata, configs sections - Add loadHarborConfig() function for loading harbor.yaml - loadUpstreamConfig() now extracts upstream: section from harbor.yaml - Add hasHarborConfig() and findHarborConfigServices() functions - Update AGENTS.md and Progress.md documentation This allows harbor.yaml to serve as a unified service configuration file that can hold upstream transformation config, service metadata for the Harbor App, and config merging declarations.
…sform
- Add VolumeEntry type to handle both string and object volume formats
- Prefix service network references (ssrf_proxy_network -> dify2-ssrf_proxy_network)
- Add implicit {prefix}-default network for services referencing 'default'
- Update AGENTS.md with implementation details
- Update Progress.md with test results
- Add upstream .env.example injection to transformation - Download ssrf_proxy config files (docker-entrypoint.sh, squid.conf.template) - Create sandbox config directory with config.yaml - Add service hostname overrides to global .env (DB_HOST, REDIS_HOST, etc.) - Fix port conflict (PLUGIN_DEBUGGING_PORT 5003->5013) - Add db/redis volume directories to .gitignore - Update Progress.md with Session 4c details All dify2 services now running: - web, api, worker, worker_beat, plugin_daemon - redis (healthy), db_postgres (with --profile postgresql) - sandbox (healthy), ssrf_proxy
Major refactor of upstream compose transformation:
- Replace service name prefixing with internal network isolation
- Services keep original names (api, web, redis, db)
- Create {namespace}-internal network for service isolation
- Exposed services get {namespace}-{service} alias on harbor-network
- No env rewrites needed - internal services use original names
- Remove transformDependsOn - depends_on stays unchanged
Schema changes:
- 'prefix' -> 'namespace'
- 'include/exclude' -> 'services.include/exclude'
- Add 'expose' list for services visible on harbor-network
Benefits:
- Upstream compose works as-is (no env var rewriting)
- No conflicts with other Harbor services
- Clean separation - no service-specific vars in global .env
- Backward compatible with existing Harbor services
Updated files:
- routines/upstream.ts - New transformation logic
- dify2/harbor.yaml - New schema with namespace/expose
- compose.dify2.yml - Use original service names
- AGENTS.md, Progress.md, DISCUSSION_DRAFT.md - Documentation
- harbor.yaml is now single source of truth for upstream services
- overrides section replaces separate compose.{service}.yml files
- Service names are prefixed (dify2-api) to avoid collision
- Original names available as aliases on internal network
- Updated AGENTS.md with new architecture
- expose can be string (default alias) or object {service: alias}
- Example: expose: [api, {web: dify-web}]
- Updated AGENTS.md and harbor.yaml documentation
- Default alias is now original service name (matches upstream)
- Custom alias syntax for conflict avoidance: {service: alias}
- Updated documentation in AGENTS.md and harbor.yaml
- overlays under upstream with target service nesting - Structure: overlays.<other_service>.<target_service>.<compose_props> - Moved ollama config from overrides to overlays (conditional) - Updated AGENTS.md and DISCUSSION_DRAFT.md with final syntax
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR implements the feature proposed in Discussion #202 - enabling Harbor to use stock Docker Compose files from upstream projects with zero modifications.
Built with AI-assisted coding (Claude).
Key Innovation: Namespace Isolation via Internal Networks
Instead of rewriting service names and environment variables, we use Docker Compose networks for conflict prevention:
How it works:
dify2-internalapi,db,redis(original names)harbor-networkdify2-api,dify2-web(aliased)Benefits:
Implementation
Files Added/Modified
routines/upstream.ts(~400 lines) - Core transformation logicroutines/mergeComposeFiles.ts- Integration with existing merge flowdify2/- Working example with Dify's stock compose (10+ services)AGENTS.md- Updated documentation with schema and transformation rulesProgress.md- Development history and design decisionsDISCUSSION_DRAFT.md- RFC documentationTransformation Rules
apiapi(unchanged)container_name: X${HARBOR_CONTAINER_PREFIX}.{namespace}-{original}depends_on: [redis]depends_on: [redis](unchanged - internal network)mydata{namespace}-mydata{namespace}-internal+harbor-network(with alias for exposed)Testing
Tested with Dify2 integration (10+ services) running for 2+ weeks:
Verified working:
harbor up dify2- Services start correctlyharbor logs dify2- Logs workharbor down dify2- Cleanup works--profile postgresql)compose.x.*.ymlpattern)Note: I haven't tested the entire
harbor.shCLI - feedback on compatibility with other commands would be appreciated.Relation to Discussion #202
This addresses the need for richer orchestration capabilities. The
harbor.yamlacts as a transformation layer that:If the TypeScript SDK direction is confirmed, this could become the first "backend" that the SDK targets, or remain as a simpler YAML alternative for basic cases.
Future Directions (not in this PR)
overlays:- Declarative cross-service confighooks:- Lifecycle scripts (pre_up, post_up)secrets:- SOPS/Vault integration