Lethe: Super High-Performance Log Rotation for Go

an AGILira fragment

Lethe is a lock-free log rotation library for Go, designed for maximum performance, seamless file management, and production-grade reliability — featuring intelligent auto-scaling between synchronous and asynchronous modes.

Features

• Zero-Lock Architecture: Atomic operations and CAS-based coordination eliminate mutex overhead
• Auto-Scaling Performance: Intelligent switching between sync and MPSC modes based on contention
• Zero-Allocation Hot Paths: Pre-allocated buffers and time caching reduce GC pressure
• Flexible Configuration: JSON, environment variables, programmatic setup (zero deps)
• Universal Compatibility: Direct io.Writer implementation works with any logging framework
• Native Iris Integration: Specifically designed for Iris ultra-high performance logging
• Built to Scale - handle millions of log entries with minimal latency

Installation

go get github.com/agilira/lethe

Quick Start

import "github.com/agilira/lethe"

// Create logger with sensible defaults
logger, err := lethe.NewWithDefaults("app.log")
if err != nil {
    log.Fatal(err)
}
defer logger.Close()

// Use as io.Writer - works with any logging framework
logger.Write([]byte("Hello, Lethe!\n"))

Magic API for Iris Integration

Automatic zero-configuration integration with Iris

QuickStart (Recommended)

import (
    "github.com/agilira/iris"
    "github.com/agilira/lethe"
)

// One line - production ready with rotation!
writer := lethe.QuickStart("app.log")
defer writer.Close()

// Direct Iris integration with automatic optimization
logger, err := iris.New(iris.Config{Output: writer})
defer logger.Close()
logger.Start()

// Use normally - automatic optimization is applied
logger.Info("Magic API provides seamless integration")

Custom Configuration

// Advanced Magic API with custom settings
writer := lethe.NewIrisWriter("app.log", &lethe.Logger{
    MaxSizeStr: "200MB",
    MaxBackups: 10,
    Compress:   true,
    Async:      true,
})

// Automatic runtime optimization automatically detected
logger, err := iris.New(iris.Config{Output: writer})

Magic Features:

• Zero Configuration: QuickStart() provides instant production-ready logging
• Runtime Integration: Automatic WriteOwned() zero-copy optimization
• Auto-Detection: Runtime capability discovery and performance tuning
• Graceful Fallback: Works with any logger, optimizes specifically with Iris

Complete Magic API Guide: docs/IRIS_INTEGRATION.md | Examples: examples/iris-integration/

Configuration

Lethe supports flexible configuration from multiple sources with zero external dependencies:

JSON Configuration

// Load from JSON
jsonConfig := `{
  "filename": "app.log",
  "max_size_str": "100MB",
  "max_backups": 10,
  "compress": true,
  "async": true
}`

config, err := lethe.LoadFromJSON([]byte(jsonConfig))
logger, err := lethe.NewWithConfig(config)

Environment Variables

export LETHE_FILENAME="app.log"
export LETHE_MAX_SIZE="100MB"
export LETHE_COMPRESS="true"
export LETHE_ASYNC="true"

// Load from environment
config, err := lethe.LoadFromEnv("LETHE")
logger, err := lethe.NewWithConfig(config)

Combined Configuration

// Combine sources with precedence: Defaults → JSON → Environment
source := lethe.ConfigSource{
    Defaults:   &lethe.LoggerConfig{Filename: "app.log", MaxSizeStr: "10MB"},
    JSONFile:   "config.json",
    EnvPrefix:  "LETHE",
}

config, err := lethe.LoadFromSources(source)
logger, err := lethe.NewWithConfig(config)

Perfect for Docker, Kubernetes, and CI/CD deployments!

📖 Complete Configuration Guide

Performance

Lethe is engineered for ultra-high performance logging. The following benchmarks demonstrate sustained throughput with minimal overhead and intelligent auto-scaling.

Write Performance (Sync Mode):            ~3.3 μs/op     (zero-lock operations)
Write Performance (MPSC Mode):            ~3.3 μs/op     (multi-producer scaling)
High Contention (Sync):                   ~109 ns/op     (atomic coordination)
High Contention (MPSC):                   ~105 ns/op     (lock-free scaling)
Zero-Allocation Hot Paths:                0 B/op         (pre-allocated buffers)
Throughput Scaling:                       1-1000+ goroutines (adaptive buffering)

Architecture

Lethe's architecture intelligently auto-scales between sync and async modes based on load, ensuring optimal performance.

graph LR
    App[Application] --> Lethe[Lethe Logger]
    Lethe --> Sync[Sync Mode]
    Lethe --> MPSC[MPSC Mode]
    Sync --> File[(Log File)]
    MPSC --> Buffer[Ring Buffer]
    Buffer --> Consumer[Consumer]
    Consumer --> File
    File --> Rotate[Rotation]

    %% Styling
    classDef app fill:#e1f5fe,stroke:#01579b,stroke-width:2px
    classDef lethe fill:#f3e5f5,stroke:#4a148c,stroke-width:2px
    classDef mode fill:#e8f5e8,stroke:#1b5e20,stroke-width:2px
    classDef buffer fill:#fff3e0,stroke:#e65100,stroke-width:2px
    classDef file fill:#fce4ec,stroke:#880e4f,stroke-width:2px

    class App app
    class Lethe lethe
    class Sync,MPSC,Consumer mode
    class Buffer buffer
    class File,Rotate file

Loading

Key Features:

• Zero-lock operations with atomic coordination
• Auto-scaling between sync/async modes
• Background compression and integrity checks

Native Integration:

• Iris Magic API - Automatic runtime integration with lethe.QuickStart() and lethe.NewIrisWriter()
• Zero-Copy Optimization - Automatic WriteOwned() detection for maximum performance
• Standard Library - Direct io.Writer implementation
• Universal Compatibility - Works with any logging framework (Zap, Zerolog, Logrus etc..)

For Maximum Performance & Simplicity: Use Iris Magic API integration with lethe.QuickStart("app.log") for instant production-ready logging with automatic optimization. Zero configuration required.

📖 See docs/ARCHITECTURE.md for detailed technical architecture

Use Cases

• Iris Magic API Integration: Zero-configuration logging with automatic runtime optimization
• Ultra-High Performance Logging: Zero-copy transfers with WriteOwned() capability detection
• Microservices Logging: Automatic rotation with compression and cleanup
• High-Throughput Applications: Auto-scaling between sync/async modes
• Production Systems: Crash-safe operations with integrity checks
• Container Environments: Automatic file management and rotation

The Philosophy Behind Lethe

In Greek mythology, Lethe was one of the Oceanids, the daughters of Oceanus and Tethys. As the personification of forgetfulness, Lethe possessed the power to grant oblivion and renewal—the ability to cleanse the past and provide fresh beginnings without the burden of accumulated history.

This embodies Lethe's design philosophy: intelligent log management that gracefully handles the past (old log files) while ensuring the present (current logging) operates at maximum efficiency. Like the Oceanid who could wash away memories, Lethe automatically manages log rotation and cleanup, allowing applications to focus on their core functionality without being weighed down by log file management.

Lethe doesn't just rotate logs—it intelligently manages the entire logging lifecycle, adapting to your application's performance needs while maintaining the reliability and efficiency that production systems demand, just as the Oceanid Lethe provided renewal and fresh starts in the ancient myths.

File System Security

Lethe includes production-grade file system security with path validation and permission management:

// Default security configuration
config := &lethe.LoggerConfig{
    Filename:           "app.log",
    MaxSizeStr:         "100MB",
    MaxBackups:         10,
    Compress:           true,
    Checksum:           true, // SHA-256 integrity checks
    LocalTime:          true,
    ErrorCallback: func(op string, err error) {
        // Custom error handling for security events
        log.Printf("Lethe security event [%s]: %v", op, err)
    },
}

Security Notes: Lethe automatically validates file paths and prevents directory traversal attacks. For production deployments, ensure proper file permissions and consider using dedicated log directories. Enable checksums for log integrity verification in security-critical environments.

Documentation

Quick Links:

• Quick Start Guide - Get running in 2 minutes
• Iris Magic API Guide - Automatic runtime integration
• Configuration Guide - JSON, environment variables, and advanced setup
• API Reference - Complete API documentation
• Architecture Guide - Deep dive into zero-lock log rotation design
• Examples - Production-ready integration patterns

License

Lethe is licensed under the Mozilla Public License 2.0.

---

Lethe • an AGILira fragment