Architecture Overview

Architecture Overview

Netra v2 follows an Applied ML Systems architecture, designed for high throughput and workload isolation.

High-Level Design

The system is composed of loose microservices communicating via Redis Streams.

graph TD
    User([User]) -->|HTTP| UI[React UI]
    User -->|HTTP| API[FastAPI Gateway]
    
    API -->|Push Event| Redis{Redis Stream: netra:events}
    
    subgraph "Workload Isolation"
        WorkerIO["Worker Ingest (I/O Bound)"]
        WorkerML["Worker ML (CPU Bound)"]
    end
    
    Redis -->|Consume| WorkerIO
    WorkerIO -->|Raw Data| RedisRaw{Redis Stream: netra:raw}
    RedisRaw -->|Consume| WorkerML
    
    WorkerIO -->|Legacy Scans| Ruby[Ruby Bridge]
    
    WorkerML -->|Update| Graph[(Neo4j Graph)]
    WorkerML -->|Store Artifacts| S3[(MinIO Object Store)]

Loading

Component Breakdown

1. Ingestion Worker (netra-ingest)

• Role: High-speed data gathering.
• Nature: I/O Bound (Network heavy).
• Tasks:
  • DNS Resolution (AsyncIO).
  • Port Scanning.
  • Running legacy Ruby scripts via RubyBridge.
• Input: netra:events:ingest
• Output: netra:data:raw

2. ML Analysis Worker (netra-ml)

• Role: Data processing and Inference.
• Nature: CPU Bound.
• Tasks:
  • Parsing raw scan data.
  • Running ML models for False Positive reduction.
  • Updating the Knowledge Graph.
• Input: netra:data:raw

3. The "Bus" (Redis Streams)

We use Redis Streams to ensure Payload Persistence. If a worker crashes, the event remains in the stream until explicitly acknowledged (XACK).

4. Knowledge Graph (Neo4j)

Netra maps the world state as a graph:

• Nodes: Domain, IPAddress, Block, Vulnerability.
• Edges: RESOLVES_TO, HOSTED_ON, AFFECTS.

Data Flow Example

1. User submits example.com via API.
2. API pushes event to netra:events:ingest.
3. Ingest Worker picks up the job, runs dos_check.rb and resolves IPs.
4. Ingest Worker pushes JSON result to netra:data:raw.
5. ML Worker consumes the raw JSON, creates (d:Domain {name: "example.com"}) in Neo4j.