Skip to content

Add Prometheus metrics exporter #47

@willibrandon

Description

@willibrandon

Description:
Export log event metrics to Prometheus for monitoring and alerting. This enables teams to monitor logging health and set up alerts based on log patterns.

Package: github.com/willibrandon/mtlog/adapters/prometheus

Proposed API:

import mtlogprom "github.com/willibrandon/mtlog/adapters/prometheus"

// Create metrics exporter
exporter := mtlogprom.NewExporter(logger,
    mtlogprom.WithCounters("log_events_total", "level", "source"),
    mtlogprom.WithHistogram("log_event_size_bytes"),
    mtlogprom.WithSinkMetrics(), // Track sink performance
)

// Expose at /metrics endpoint
http.Handle("/metrics", exporter.Handler())

// Or use as an enricher to add metrics
logger := mtlog.New(
    mtlog.WithConsole(),
    mtlog.WithEnricher(exporter),
)

Metrics to Export:

Event Metrics:

  • log_events_total{level="error",source="api"} - Count by level and source
  • log_event_size_bytes - Histogram of event sizes
  • log_properties_per_event - Histogram of property counts

Performance Metrics:

  • log_sink_latency_seconds{sink="seq"} - Sink write latency
  • log_sink_errors_total{sink="elasticsearch",error="timeout"} - Sink errors
  • log_sink_events_dropped_total{sink="file",reason="full"} - Dropped events

Template Cache Metrics:

  • log_template_cache_hits_total - Cache hit count
  • log_template_cache_misses_total - Cache miss count
  • log_template_cache_size - Current cache size
  • log_template_parse_duration_seconds - Template parsing time

Example Prometheus Queries:

# Alert on high error rate
rate(log_events_total{level="error"}[5m]) > 10

# Monitor sink performance
histogram_quantile(0.99, log_sink_latency_seconds)

# Track template cache efficiency
rate(log_template_cache_hits_total[5m]) / 
  (rate(log_template_cache_hits_total[5m]) + rate(log_template_cache_misses_total[5m]))

Implementation Notes:

  • Use prometheus/client_golang
  • Metrics should be zero-cost when not enabled
  • Consider cardinality limits for labels
  • Support custom metric namespaces
  • Provide pre-built Grafana dashboard JSON

Metadata

Metadata

Assignees

Labels

enhancementNew feature or requestobservabilityEnhancements that improve production observability and debugging capabilities

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions