Skip to content

Logging System Comparison

Jump to bottom Edit New page
jarretraim edited this page Feb 22, 2013 · 1 revision

As part of the logging as a service effort, many existing logging systems were investigated. Many features, design decisions and terminology was taken from these systems. However, as a key goal of logging is to be OpenStackable, it was decided that rather than using existing systems, we would develop our own logging components using the OpenStack commons packages in Python.

The following are the notes from our investigations into existing projects, both from a technical and product perspective.

Open Source Log Systems

  • Scribe
  • Flume
  • logstash
    • JRuby
    • Self-Contained
    • Pushes data to elastic search
    • Provides simple web ui
    • inputs -> filters -> outputs
    • dozens of options
  • Chukwa
  • fluentd
  • kafka
  • Graylog2

As a Service

Loggly

  • Price based on MB/day + retention
  • Archive logs to S3
  • Secure logging (HTTPS/TLS)
  • JSON enabled logs can be parsed and queried
  • Alerting is a separate app hosted on google app engine using PagerDuty
  • Supports OAuth for app integration
  • Saved searches
  • Unlimited clients - based on MB/day
    • First 30 days is free - allows customer to determine log volume
    • System will stop indexing logs if the user exceeds their maximum
    • Unindexed logs are held until the next day - not lost
  • Max retention time is 90 days, then archive to S3
  • Log sources
    • Linux / OSX Syslog TCP / UDP
    • Windows Powershell (HTTP API)
    • Windows Snare (open source agent)
    • Windows .NET library
    • Windows NTSyslog
    • Any HTTPS ReSTish endpoint

Papertrail

  • Price based on GB / Month - 1 year archive included, 1-2 weeks search included
  • Long tern archive on S3
  • Group inputs into related event streams - e.g. Rackspace v Amazon, DB v Web, etc.
  • Alerting is built in
  • Save searches is built in
  • Encryption supported with syslog
  • Supports read-only users
  • Supports PagerDuty
  • Sources
    • Syslog
    • SyslogAgent, NTSyslog - 3rd party agent
    • remote_syslog - papertrail provided agent

Logentries

  • Interactive graphing in web-ui
  • Supports encrypted logs
  • Built in alerts - only seems to support email and POST notification
  • Pricing - Metered
  • Free tier of 1 GB / month with 1 week online retention
    • .99 / GB indexed
    • .99 / GB storage up to 100GB, .49 after 100GB
    • The GB is total logs consumed so retention means you are paying for more logs longer
    • Minimum charge of / mo
  • Sources
    • syslog
    • logentries agent
    • HTTP PUT
    • Snare 3rd party agent
    • Android

AlertLogic

  • Security & Compliance tool
  • PCI 10.2, 10.3, 10.5, 10.6 and 10.7
  • Daily reports of PCI events
  • Log review is an enhancement - not required
  • Sources
    • AlertLogic agent
    • Syslog
  • Integrations
    • Events - email, smartphone and ticketing systems
  • Workflow for incident notification
  • Included number of parsers for common log formats
  • Physical or virtual appliance
    • VMWare only
  • GIAC-certified security analysis and researchers
  • 24x7 SOC
  • No pricing - enterprise

Splunk

  • Available aaS (Storm) or in datacenter
  • Pricing is total data stored
    • 1GB free, scaling down up to $3k for 1TB
  • Sources
    • File upload
    • syslog
    • ReST API
    • splunk agents
  • Export data to CSV & JSON
Add a custom footer
Add a custom sidebar

Clone this wiki locally