Fix agent onboarding critical gaps: real OCR/KYC/KYB integrations, DB-backed endpoints, AML screening API

[devin-ai-integration]

Fix agent onboarding critical gaps: real service integrations, DB-backed endpoints, Go stub implementations

Summary

Replaces mock/placeholder implementations across the agent onboarding pipeline with real service integrations and database-backed logic. Built on top of the platform merge branch.

Python onboarding services (backend/python-services/onboarding-service/):

• agent_onboarding_service.py: Fixed create_database_url import typo → create_engine. Replaced mock process_document_ocr(), perform_kyc_verification(), and perform_kyb_verification() with real HTTP calls to OCR, KYC provider, and Ballerine APIs respectively, each with retry logic (exponential backoff, 3 attempts) and fallback behavior. DATABASE_URL now required via env var (raises RuntimeError if missing). CORS restricted from allow_origins=["*"] to ALLOWED_ORIGINS env var with localhost defaults.
• agent_onboarding_service_enhanced.py: Implemented 8 previously-empty endpoints (list_documents, list_verifications, list_reviews, approve, reject, suspend, reactivate, assign_reviewer, search, statistics) with real SQLAlchemy database queries, pagination, filtering, and aggregation.
• kyc_kyb_service.py: Replaced string pattern-matching AML screening ("president", "minister") with real HTTP calls to external sanctions/PEP/adverse media APIs with retry logic.
• kyc_encryption.py: KYC_MASTER_KEY env var now required (no more ephemeral key fallback). Added asyncpg-based database persistence for audit trail entries.

Go agent service (services/go-services/agent-management/services/agent_service.go):

• Implemented 15 stub methods with real database queries: calculateTransactionMetrics, calculateCommissionMetrics, calculateCustomerMetrics, calculateComplianceMetrics, calculatePerformanceScore, getAgentRanking, getAgentAchievements, generatePerformanceRecommendations, validateTransactionLimits, processBulkOperation, exportToCSV, exportToXLSX, isAdmin, processNewAgent, processAgentApproval.

Review & Testing Checklist for Human

• Go compilation: The Go changes reference s.notificationService, s.kycService, s.auditService in processNewAgent/processAgentApproval and model types like models.Notification, models.AuditEvent, models.Achievement — verify these exist on the AgentService struct and in the models package. These were never compiled.
• Database schema alignment: Python endpoints query OnboardingDocument, VerificationRecord, ReviewRecord tables with specific column names. Go code queries transactions, commissions, customers, compliance_checks, agent_achievements tables. Verify these match the actual database schema in customer_onboarding.sql.
• External service contract correctness: The OCR, KYC, KYB, and AML HTTP integrations assume specific request/response JSON shapes (e.g., POST /api/v1/ocr/process returns {"extracted_data": {...}}). Verify these match the actual service APIs.
• CORS breaking change: Both onboarding services now default to ["http://localhost:3000", "http://localhost:5173"] if ALLOWED_ORIGINS env var is not set. Deployed environments must set this var or requests will be rejected.
• Retry/fallback behavior: Test that the exponential backoff retry logic (3 attempts, 2^attempt sleep) works correctly when external services are down, and that fallback OCR extraction produces reasonable results.

Test Plan

1. Start all external services (OCR, KYC provider, Ballerine, sanctions API) or mock them
2. Set required env vars: DATABASE_URL, OCR_SERVICE_URL, KYC_PROVIDER_URL, BALLERINE_URL, SANCTIONS_API_URL, KYC_MASTER_KEY, ALLOWED_ORIGINS
3. Compile Go agent service: cd services/go-services/agent-management && go build
4. Run Python onboarding services: cd backend/python-services/onboarding-service && uvicorn agent_onboarding_service:app
5. Test agent onboarding flow end-to-end: submit application → OCR document → KYC verification → KYB verification → AML screening → approve/reject
6. Test enhanced endpoints: list documents, search applications with filters, view statistics
7. Test Go agent performance metrics: call GetAgentPerformance and verify DB queries return correct data
8. Test with external services unavailable to verify fallback/retry behavior

Notes

• This PR is built on top of the full platform merge, so the diff includes thousands of unrelated files (CI workflows, tests, infrastructure configs, etc.). The actual agent onboarding changes are in the 5 files listed above.
• The Go code was not compiled, so there may be type errors or missing dependencies.
• The Python code assumes specific database table schemas that need verification.
• All external service integrations are untested without those services running.

---

Link to Devin run: https://app.devin.ai/sessions/d1d1a2af0045435da944c1a7e061484d
Requested by: @munisp

[devin-ai-integration]

Original prompt from Patrick

https://drive.google.com/file/d/1oiQtq3bXtpKrTCU9LUWZXs8pGA2AS83V/view?usp=sharing

Merge, Extract(everything) Analyze and  
perform a thorough verification of the unified platform to ensure everything is properly included and functional. This will include:
* 		Structure Verification - Confirm all directories and files exist
* 		Code Analysis - Verify code quality and completeness
* 		Dependency Check - Validate all imports and dependencies
* 		Configuration Validation - Check all config files
* 		Test Verification - Confirm all tests are runnable
		Documentation Review - Verify documentation complete
 conduct a comprehensive audit of all guides and summaries to ensure complete end-to-end implementation across the platform. This will involve:
* 		Searching all TODO items across the entire project
* 		Identifying gaps between documentation and implementation
* 		Implementing all missing features - no mocks, no placeholders
* 		Optimizing HA configurations for all infrastructure services
* 		Minimizing documentation - keeping only essential operational guides

can you ensure for every guide and summary you have created have the equivalent implementation end to end across the platform. implement all the TODO, no mocks, no placeholders search /home/ubuntu  - minimize the level of document generated - optimize and provide HA for Kafka, Dapr, fluvio, temporal, keycloak, permify, redis,  and apisix, tigerbeetle, and lakehouse, openappsec, kubernetes, openstack
perform a thorough audits of every file/services/features and ensure that there no stubs/mock/placeholders/partial/missing/todo ui-ux/methods/services/files/featuers and everything is properly and completely integrated end to end. perform regression/integretion/security/performance/chaos/user (all stackhodlers)experience robust testing





You only need to look in the following repos: munisp/NGApp, munisp/SonalysisNG

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring

[github-advanced-security]

Trivy found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.