Resource Metric Streamer

A lightweight, efficient Python agent that collects system CPU and RAM metrics and streams them to an Apache Kafka topic in real-time.

Features

• Real-time Metrics: Captures CPU and RAM usage percentages.
• Kafka Streaming: Streams metrics to a Kafka topic as structured JSON messages.
• Configurable: Easily adjust polling intervals and Kafka settings via environment variables.
• Lightweight: Minimal resource footprint on the host system.
• Built with: uv, psutil, confluent-kafka, and pydantic.

Prerequisites

• Python: 3.9 or higher.
• Kafka: A reachable Kafka broker.
• uv: For fast Python package and environment management.

Installation

1. Clone the repository:

   git clone <repository-url>
   cd cpu_usage

2. Start Local Kafka (Optional - requires Podman):

   # Using Podman Compose
   podman-compose up -d

3. Install dependencies:

   uv sync

Configuration

The agent is configured using environment variables:

Variable	Description	Default
KAFKA_BOOTSTRAP_SERVERS	List of Kafka brokers (comma-separated)	localhost:9092
KAFKA_TOPIC	Target Kafka topic for metrics	resource_metrics
METRIC_POLLING_INTERVAL	Polling frequency in seconds	5.0

Usage

Run the agent using uv:

# Using default configuration
uv run python -m cpu_usage.main

# With custom configuration
KAFKA_BOOTSTRAP_SERVERS="broker:9092" KAFKA_TOPIC="system_stats" uv run python -m cpu_usage.main

Data Schema

Metrics are sent as JSON objects with the following structure:

{
  "hostname": "my-server-01",
  "ip_address": "192.168.1.100",
  "cpu_usage_percent": 12.5,
  "ram_usage_percent": 45.2,
  "timestamp": "2026-02-16T15:30:00.123456"
}

Development

Running Tests

Execute the test suite with coverage reporting:

uv run pytest --cov=src/cpu_usage --cov-report=term-missing

Linting and Formatting

Check and format the code according to the project style:

uv run ruff check .
uv run ruff format .